Title: Neural Isometries: Taming Transformations for Equivariant ML
Venue: NeurIPS 2024
Accepted: yes
Keywords: Equivariance

Geometric Deep Learning

Representation Learning
Abstract: Real-world geometry and 3D vision tasks are replete with challenging symmetries that defy tractable analytical expression. In this paper, we introduce Neural Isometries, an autoencoder framework which learns to map the observation space to a general-purpose latent space wherein encodings are related by isometries whenever their corresponding observations are geometrically related in world space. Specifically, we regularize the latent space such that maps between encodings preserve a learned inner product and commute with a learned functional operator, in the same manner as rigid-body transformations commute with the Laplacian. This approach forms an effective backbone for self-supervised representation learning, and we demonstrate that a simple off-the-shelf equivariant network operating in the pre-trained latent space can achieve results on par with meticulously-engineered, handcrafted networks designed to handle complex, nonlinear symmetries. Furthermore, isometric maps capture information about the respective transformations in world space, and we show that this allows us to regress camera poses directly from the coefficients of the maps between encodings of adjacent views of a scene.

Neural Isometries:

Abstract

Real-world geometry and 3D vision tasks are replete with challenging symmetries that defy tractable analytical expression. In this paper, we introduce Neural Isometries, an autoencoder framework which learns to map the observation space to a general-purpose latent space wherein encodings are related by isometries whenever their corresponding observations are geometrically related in world space. Specifically, we regularize the latent space such that maps between encodings preserve a learned inner product and commute with a learned functional operator, in the same manner as rigid-body transformations commute with the Laplacian. This approach forms an effective backbone for self-supervised representation learning, and we demonstrate that a simple off-the-shelf equivariant network operating in the pre-trained latent space can achieve results on par with meticulously-engineered, handcrafted networks designed to handle complex, nonlinear symmetries. Furthermore, isometric maps capture information about the respective transformations in world space, and we show that this allows us to regress camera poses directly from the coefficients of the maps between encodings of adjacent views of a scene.

1 Introduction

We constantly capture lossy observations of our world - images, for instance, are 2D projections of the 3D world. Observations captured consecutively in time are often related by transformations which are easily described in world space, but are intractable in the space of observations. For instance, video frames captured by a camera moving through a static scene are fully described by a combination of the 3D scene geometry and SE (3) camera poses. In contrast, the image space transformations between these frames can only be characterized by optical flow, a highdimensional vector field that does not itself have any easily tractable low-dimensional representation.

Geometric deep learning seeks to build neural network architectures that are provably robust to transformations acting on their inputs, such as rotations [1-3], dilations [4, 5], and projective transformations [6, 7]. However, such approaches are only tractable for transformations that have group structure, and, even in those cases, still require meticulously handcrafted and complex architectures. Yet, many real-world transformations of interest, for instance in vision and geometry processing, altogether lack identifiable group structure, such as the effect of camera motion in image space-see Fig. 1 to the right. Even when group-structured, they are often non-linear and non-compact, such as is the case of image

Encoder

Isometry

Input to generic equivariant NN

山

()3

E(Ty)

T

homographies and non-rigid shape deformations where existing approaches can be prohibitively expensive.

In this paper, we take a first step towards a class of models that learns to be equivariant to unknown and difficult geometric transformations in world space. We propose Neural Isometries ( NIso ), an autoencoder framework which learns to map the observation space to a latent space in which encodings are related by tractable, highly-structured linear maps whenever their corresponding observations are geometrically related in world space.

Specifically, observations are encoded into latents preserving their spatial dimensions. Images, for instance, can be encoded into latent functions defined over a lower resolution grid or the patch tokens of a ViT. For observations sharing some potentially unknown relationship in world space, we enforce their encodings be related by a functional map τ - a linear transformation on the space of latent functions. In particular, we require that τ is an isometry such that it preserves a learned inner product and commutes with a learned functional operator, in the sense that rigid body transformations commute with the Laplace operator in Euclidean space.

Neural Isometries exhibit unique properties that make them a promising step towards an architectureagnostic regime for self-supervised equivariant representation learning. We experimentally validate two principle claims regarding the efficacy and applicability of our approach:

Neural Isometries recover a general-purpose latent space in which challenging symmetries in the observation space can be reduced to compact, tractable maps in the latent space. We show that this can be exploited by simple isometry-equivariant networks to achieve results on par with leading hand-crafted equivariant networks in tasks with complex non-linear symmetries.

The latent space constructed is geometrically informative in that it encodes information about the transformations in world space. We demonstrate that robust camera poses can be regressed directly from the isometric functional maps between encodings of adjacent views of a scene.

2 Related Work

Geometric Deep Learning. Geometric deep learning is generally concerned with hand-crafting architectures that are equivariant to ( i.e. that commute with) known transformations acting on the data [1-4, 6-17]. To this end, many successful architectures exploit group representations by using established mappings, such as the Fourier or spherical harmonic transforms, to map features onto domains where the group actions manifest equivalently as linear transformations [1-3, 7]. In most cases, the representations considered are finite-dimensional and irreducible which, loosely speaking, means that the group action in observation space can be expressed exactly by frequency-preserving block-diagonal matrices acting on the transform coefficients. While finite-dimensional irreducible representations ( IRs ) are attractive building blocks for equivariance due to their computationally exploitable structure, they often don't exist for non-compact groups, precluding generalizations to most non-linear symmetries, let alone those ill-modeled by groups. We instead avoid a heuristic choice of symmetry model and seek an approach that enables robustness to arbitrary transformations that may not even be group-structured.

Learned Spectral Operator

Solve for Isometry

Image

Image

Latent

Latent

Recon.

Recon.

Un-

project

Un-

project

Functional

Eigenbasis

Projection

Eigenbasis

Projection

Spec.

dropout

Spec.

dropout

Equiv. Loss

Recon. Loss

Multip. Loss

from nagging theoretical constraints, FMs have displayed remarkable representational capacity, well modeling a variety of highly-complex non-rigid deformations [24] including those without group structure, such as partial [25] and inter-genus correspondence [26]. Please see [27] for an outstanding introduction to functional maps.

Discovering Latent Symmetries. A number of recent approaches have proposed autoencoder frameworks wherein given symmetries in the base space manifest as simple operations in the latent space [28-35]. Perhaps most similar to our approach is recent work on a Neural Fourier Transform (NFT) that has sought to manifest group actions in observation space as IRs in the the latent space [36, 37]. These methods offer impressive theoretical guarantees under the conditions that the observed transformations are either known or are a group action, although these assumptions may not hold for real-world data with complex symmetries. In contrast, our method is wholly unsupervised, and assumes no knowledge of the transformations in observation space nor that they even form a group.

3 Method Overview

Neural Isometries are an architecture-agnostic autoencoder framework which learns to map pairs of observations related in world space to latents which are related by an approximately isometric FM τ -see Fig. 2. We first formulate encoding and decoding, and define a FM in the latent space. Next, we show how a functional operator Ω and mass matrix M can be learned in the latent space to regularize τ by requiring it to be an isometry. Specifically, for observations sharing some potentially unknown relationship in world space, we enforce there exist a FM τ between their encodings satisfying two key properties: 1) That τ preserves the functional inner product in the latent space determined by M ; and 2). τ commutes with the functional operator Ω . Subsequently, we show that such maps can be recovered analytically through a differentiable, closed-form least squares solve in the operator eigenbasis. Last, we formulate NIso as an optimization problem incorporating both the strictness of the isometric correspondence between latents and the eigenvalue multiplicity of the operator, the latter of which controls the structure of the maps.

In experiments, we demonstrate that NIso are capable of both discovering approximations of known operators and constructing latent spaces where complex, non-linear symmetries in the observation space manifest equivalently as isometries. We show how the latter property can be exploited by demonstrating that a simple vector neuron MLP [38] acting in our pre-trained latent space can achieve results on par with state-of-the-art handcrafted equivariant networks operating in observation space. Subsequently, we consider the task of pose estimation, and demonstrate that robust SE (3) camera

poses can be extracted from latent transformations, serving as evidence that NIso encourages models to encode information about transformations in world space.

4 Neural Isometries

We consider an observation space O ⊂ L 2 ( M, R n ) consisting of functions defined over some domain M ( e.g. with M the plane and n = 3 for RGB images). Here, elements of O are in fact captures from some world space W with σ : W → O representing the mechanism from which O is formed from W . Furthermore, we assume there is a potentially unknown collection of phenomena { T } acting on the world space that relates observations. That is, for some w ∈ W , ψ = σ ( w ) ∈ O , and denoting Tψ ≡ σ ( Tw ) , we assume that Tψ is also in O and that we are able to associate it with ψ .

Additionally, we consider an autoencoder consisting of an encoder and decoder

mapping between observation space and a space of latent functions over some domain N . In practice, we operate over discretizations of M and N , with ψ ∈ O and E ( ψ ) ∈ L 2 ( N, R d ) represented as tensors ψ ∈ R | M |× n and E ( ψ ) ∈ R | N |× d . For example, if M consists of the pixel indices of an image, then N could be grid or token indices if E is a ConvNet or ViT, respectively.

Goal: Equivariance of Latent Functions. Our aim is to train the autoencoder such that for any T acting in the world space and corresponding observations ψ, Tψ ∈ O , there exists a linear map τ : L 2 ( N, R d ) → L 2 ( N, R d ) such that

In other words, we desire our latent space to be equivariant under world space transformations. As our problem is discrete, τ is a functional map - an | N | × | N | matrix representation of maps on L 2 ( N, R d ) . In the case of latents with | N | = H × W pixels, τ is a matrix whose rows express each pixel in E ( Tψ ) as a linear combination of pixels in E ( ψ ) , similar to the weight matrix one might obtain from a cross-attention operation.

4.1 Regularization Through Isometries

We will find τ by solving a least-squares problem of the form τ = min π ∥E ( Tψ ) -π E ( ψ ) ∥ . Unfortunately, as we will show in experiments, a direct solve without additional regularization leads to uninformative maps that capture little information about the actual world-space transformations T . To add structure, we might ask that τ be orthogonal with τ ⊤ τ = I | N | , generating gradients promoting latent codes having the property ∥E ( ψ ) ∥≈∥E ( Tψ ) ∥ .

However, we can obtain more structure yet. We propose to learn a representation of the latent geometry by jointly regressing a diagonal mass matrix M and positive semi-definite ( PSD ) operator Ω ∈ R | N |×| N | such that τ manifests as an isometry . That is, τ preserves the functional inner product defined by M -⟨ f, g ⟩ M = f ⊤ M g for f, g ∈ L 2 ( N, R d ) - and is Ω -commutative with

Together, the conditions in Equation (3) form a strong regularizer, the effects of which are best seen in the expression of τ in the eigenspectrum of Ω . As Ω is a PSD matrix with respect to the inner product defined by M , it can be expressed in terms of its spectral decomposition as

where Λ = diag ( { λ i } 1 ≤ i ≤| N | ) is the diagonal matrix of (non-negative) eigenvalues and Φ ∈ R | N |×| N | is the matrix whose columns are the M -orthogonal eigenfunctions of Ω . Denoting

as the projection of τ into the eigenbasis, it can be shown that the conditions in Eq. (3) reduce to τ Ω being orthogonal and Λ -commutative [27]. This is equivalent to asking that τ Ω be both sparse and condensed in that it forms an orthogonal, block-diagonal matrix , with the size of each block determined by the multiplicity of the eigenvalues in Λ .

4.2 Estimating τ and End-to-End Optimization

Fig. 2 visualizes Neural Isometries's training loop. First, τ is estimated between pairs of encoded T -related observations such that it approximately satisfies the conditions for an Ω -isometry as in Eq. (3). Second, the weights of the autoencoder, M , and Ω are jointly updated with respect to a combined loss term, promoting: a) latent equivariance as in Eq. (2), b) the ability of the decoder to reconstruct observations, and c) distinct eigenvalues Λ which encourage a diagonal-as-possible τ Ω .

Recovering τ Between Latents. Instead of estimating τ directly, we equivalently estimate τ Ω in the eigenbasis of Ω , motivated by the corresponding simplification of the conditions in Eq. (3). Let

be the map given by encoding followed by projection into the eigenbasis of Ω . Then, given observations ψ and Tψ , we define τ Ω to be the solution to the least squares problem

While Eq. (7) has an exact analytical solution [39], we instead approximate τ Ω with a fuzzy analogue which we find better facilitates backwards gradient flow to the parameters of Ω .

Specifically, letting κ : R | N |×| N | → O ( | N | ) denote the Procrustes projection to the nearest orthogonal matrix ( e.g. through the SVD), we recover τ Ω via the approximation

where [ P Λ ] ij = exp( -| λ i -λ j | ) is a smooth multiplicity mask over the eigenvalues Λ applied element-wise. See the supplement for details.

To facilitate the recovery of τ Ω , we parameterize M directly by its diagonal elements and Ω in terms of its spectral decomposition, learning a M -orthogonal matrix of eigenfunctions Φ and nonnegative eigenvalues Λ . This has the added benefit of enabling a low-rank approximation of Ω by parameterizing only the first k eigenvalues and eigenfunctions, i.e. Φ ∈ R | N |× k and Λ = diag ( { λ i } 1 ≤ i ≤ k ) with k ≤ | N | , mirroring similar approaches in SoTA FM pipelines [19, 40, 41]. This reduces the complexity of the orthogonal projection in Eq. (8) from | N | × | N | to k × k .

Optimization. During training, the autoencoder is given pairs of T -related observations ( ψ, Tψ ) , which are mapped to the latent space and τ Ω is estimated as in Eq. (8) giving τ = Φ τ Ω Φ ⊤ M . First, an equivariance loss is formed between the eigenspace projections of the encodings,

We note that for full rank Ω , this loss is equivalent to measuring the degree to which the equivariance condition in Eq. (2) holds due to the orthogonality of Φ . Next we compute a reconstruction loss ,

with τ -1 = Φ τ Ω ⊤ Φ ⊤ M , forcing the decoder to map the transformed latents to the corresponding T -related observations. Last, we formulate a multiplicity loss which promotes distinct eigenvalues and ensures Ω is 'interesting' by preventing it from regressing to the identity. We observe that the eigenvalue mask P Λ can be viewed as a graph in which the number of connected components ( i.e. the number of distinct eigenvalues) is equivalent to the dimension of the nullspace of the graph Laplacian ∆ P Λ formed from the mask [42]. As a measure of the nullspace dimension, we use the norm of the eigenvalues of ∆ P Λ , given by L M = ∥ ∆ P Λ ∥ . The NFT [36, 37] takes a similar approach, wherein a diagonalization loss is imposed on estimated transformations themselves, though our experiments show it be far less effective in enforcing structure. The total loss is the sum of aforementioned terms

with α, β ≥ 0 weighting the contributions of the equivariance and multiplicity losses.

In experiments, we also consider a similar triplet regime as proposed in [36, 37], where the autoencoder is given triples of T -related observations ( ψ, Tψ, T 2 ψ ) (assuming T is composable) and the estimated map τ between the encodings of ( ψ, Tψ ) is used to form equivariance and reconstruction losses between ( Tψ,T 2 ψ ) and vice-versa. This works to prevent τ from 'cheating' by encoding

Learned Eigenfunctions

Data

Operator

GT    : Laplacian

Latent Map

NIso Outputs

Reference

Toric

Spherical

privileged information about the relationship between pairs beyond T , a property we show to be critical in enforcing a useful notion of latent equivariance. However, triples of T -related observations are rare in practical settings, and we show in experiments that a major benefit of our isometric regularization is that our multiplicity loss (promoting a diagonal-as-possible and thus sparse and condensed τ Ω ) can serve as an effective substitute for access to triples.

Spectral Dropout. While the composite loss in Equation (11) promotes latent equivariance and distinct eigenvalues, it does not, however, promote a physically meaningful ordering of the eigenvalues -i.e. that small eigenvalues correspond to smooth, low-frequency eigenfunctions and larger eigenvalues to eigenfunctions with sharper, high-frequency details. Though such an ordering amounts to a permutation in the spectral dimension and is not a necessary condition for the existence of diagonalized isometric maps, it could be potentially useful in downstream applications where a classical notion of eigenvalues as frequencies is desired.

To this end we propose an optional spectral dropout layer applied after computing the equivariance loss L E and before the basis unprojection. The layer is implemented as follows: During training, there is a 50% chance that dropout will be applied to given example in a batch. Then, for each such example, a spectral index 1 < i ≤ k is randomly chosen and all features with index j ≥ i are masked out. Intuitively, this forces the decoder to produce the best possible reconstructions with the lowest frequency eigenfunctions, which appear most often and thus must capture large scale features, while reserving higher frequencies for filling in fine details.

4.3 A Simple Example: Approximating the Toric and Spherical Laplacians

We demonstrate that NIso are able to learn a compact representation of isometries that reflect the dynamics of transformations in world space. To do so, we perform two experiments in which we consider pairs of observations formed by 16 × 16 images from ImageNet [43] viewed functions on toric and spherical grids, and respectively transformed by random circular shifts and SO(3) rotations to form pairs. Thus pairs are related by the isometries of the torus and sphere which commute with the Laplacian on each domain. Taking the encoder and decoder to be the identity map (making the equivariance and reconstruction losses equivalent) we optimize for M , Ω ∈ R 256 × 256 via the pairwise training procedure described in sec. 4.2, including the use of spectral dropout. For the toric experiments, we learn a full-rank parameterization of Ω with k = 256 ; for the sphere, we learn a

low-rank approximation with k = 64 . As seen in Fig 3, NIso regresses operators with significant structural similarities to the Laplacian matrices formed by the standard 3 × 3 stencil on the torus and the low rank approximation using the first 64 spherical harmonics. In both cases, NIso recovers an eigenspace that diagonalizes τ Ω between shifted images, with eigenfunctions ordered by their energy. As our approach is data-driven and our estimated maps are only approximately isometric, our learned operator and its eigenspectrum do not perfectly correspond to the ground-truth Laplacian. Instead, we are able to characterize similar, non-trivial spatial relationships that are preserved under shifts.

4.4 Representation Learning with NIso

Viewed in terms of representation learning, NIso can be seen as a recipe for the self-supervised pre-training of a network backbone E satisfying the equivariance condition in Eq. (2) such that transformations T in the world space manifest as isometries τ in the sense of Eq. (3).

Exploiting Equivariance in Latent Space As we demonstrate in experiments, a simple off-theshelf isometry-equivariant head can be appended to the pre-trained backbone and fine-tuned to achieve competitive results in tasks with challenging symmetries. We employ a simple strategy wherein a low rank k approximation of Ω is learned during the pre-training stage. Thus, the eigenspace projections of the encodings of T -related observations are k × d tensors that are nearly equivalent up to an orthogonal transformation τ Ω . As such, we pass the projected encodings to a head consisting of an O ( k ) -equivariant vector neuron ( VN ) MLP [38]. We note that for large-scale tasks NIso is potentially well-suited to pair with DiffusionNet [44], which can make use of the learned eigenbasis to perform accelerated operations in the latent space, though we do not consider this regime here.

Pose Extraction from Latent Isometries. We propose that the recovered functional maps τ encode information about world-space transformations T . To test this, we consider a simple pose estimation paradigm consisting of a pre-training phase and fine-tuning phase. In the first phase, a NIso autoencoder is trained using T -related pairs of observations consisting of adjacent frames from video sequences. Subsequently, the decoder is discarded and the same pairs of observations are considered during fine-tuning. In the second phase, isometries τ Ω are estimated between the eigenspace projections of the encoded observations, vectorized, and passed directly to an MLP which predicts the parameters of the SE (3) transformation corresponding to the relative camera motion in world space between the adjacent frames. In our experiments, the weights of the NIso backbone are frozen during fine-tuning to better evaluate the information about world space transformations encoded during the unsupervised pre-training phase. At evaluation, trajectories are recovered by composing estimated frame-to-frame poses over the length of the sequence.

5 Experiments

In this section, we provide empirical evidence through experiments that NIso 1) recovers a generalpurpose latent space that can be exploited by isometry-equivariant networks to handle challenging symmetries (5.1, 5.2); and 2) NIso encodes information about transformations in world space through the construction of isometric maps in the latent space from which geometric quantities such as camera poses can be directly regressed (5.3). Here we pre-train NIso without spectral dropout, as the ordering of the learned spectrum is irrelevant in our target applications and we find it slightly decreases accuracy of the prediction head. We provide reproducibility details in the supplement in addition to experiments quantitatively evaluating the degree of learned equivariance. We note that a consistent theme in our experiments are comparisons against the unsupervised variant of the NFT [37] (the semisupervised variants cannot be applied because the symmetries we consider have no finite-dimensional IRs). Like our approach, the NFT seeks to relate latents via linear transformations though, it differs fundamentally in that maps are guaranteed additional structure only if the world space transformations are a compact group action. While not originally proposed by the authors, we evaluate it in place of our approach in the same self-supervised representation learning regimes discussed in sec. 4.4. Thus the role of these comparisons is to show that our proposed isometric regularization better and more consistently provides a tractable and informative latent space.

5.1 Homography-Perturbed MNIST

In our first set of experiments, we consider classification on the homNIST dataset [6] consisting of homography-perturbed MNIST digits. Following the procedure outlined in sec. 4.4, the classification

network consists of a pre-trained NIso encoder backbone followed by a VN-MLP. Specifically, pre-training is performed by randomly sampling homographies from the distribution proposed in [6] which are applied to the elements of the standard MNIST training set to create pairs of observations. Here the weights of the encoder backbone are frozen and the equivariant head is trained only on the original (unperturbed) MNIST training set and evaluated on the perturbed test set. Thus the aim of these experiments is to directly quantify the degree to which pre-trained latent space is both equivariant and distinguishable.

Acc.

NIso

92.52 (

±

0.91)

w/ triplet

97.38 (

±

0.23)

w/o

L

E

77.30 (

±

2.56)

w/o

L

M

45.27 (

±

1.20)

NFT [37]

41.93 (

±

0.84)

w/ triplet

67.15 (

±

1.10)

AE w/ aug.

80.96 (

±

1.95)

homConv [6]

95.71 (

±

0.09)

LieDecomp [45]

98.30 (

±

0.10)

0

4

3

9

3

9

With this in mind, we perform three ablations. In the first, we train the NIso autoencoder in a triplet regime (4.2) made possible by the synthetic parameterization of T as homographies. In the second and third, we train the autoencoder in the standard pairwise regime without considering the equivariance loss L E and multiplicity loss L M , respectively. Additionally we compare the efficacy of our approach versus an NFT backbone pre-trained in the same manner. Last, we pre-train and evaluate a baseline backbone which considers only a reconstruction loss without respect to T -related pairs, but trains with data-augmentation during the fine-tuning phase by applying randomly sampled homographies.

Results are shown in Tab. 1, averaged over five randomly initialized pre-training and fine-tuning runs with standard errors. Also included are those reported by homConv [6] and LieDecomp [45], top-performing homography-equivariant networks which serve as a handcrafted baseline. While NIso pre-trained with the triplet regime produces results on par with the handcrafted baselines, pre-training with the pairwise regime-which reflects a real-world scenario-achieves a classification accuracy above 90 % , signifi- cantly better than all but the three aforementioned approaches. Critically, performance drops when the multiplicity loss L M is omitted, which corresponds to a regime where τ must only preserve the inner product and Ω converges to a multiple of the identity operator. This suggests that sparsifying τ Ω by filtering it through a low-multiplicity eigenspace ( i.e. enforcing that it is an 'interesting' isometry) is fundamental in forcing the network to disentangle the structure of the observed transformations from the content of the observations themselves. In the same vein, the equivariance loss L M is also clearly instrumental, as the reconstruction loss alone does not explicitly enforce that E ( ψ ) is in fact mapped to E ( Tψ ) under the estimated τ . Furthermore, while the authors report that latent maps tend to converge to orthogonal maps for compact group actions in world space [36, 37], both NFT regimes preform poorly, implying that the learned maps do not replicate the properties of finite-dimensional IRs when the group is non-compact.

5.2 Conformal Shape Classification

Acc.

NIso

90.26 (

±

1.27)

NFT [37]

83.24 (

±

2.03)

AE w/ aug.

69.36 (

±

2.81)

MC[7]

86.5

T

Next, we apply NIso to classify conformally-related 3D shapes from the augmented SHREC '11 dataset [7, 46]. We follow [7] by mapping each mesh to the sphere and subsequently rasterizing to a grid. During pre-training, T -related pairs are selected from the sets of conformallyaugmented meshes derived from the same base shape in the train split. In the fine-tuning phase, the encoder weights are unfrozen and are jointly optimized with the equivariant head, representing the practical implementation of our proposed approach for equivariant tasks.

Results are shown in Tab. 2, averaged over five randomly initialized pre-training and fine-tuning runs with standard errors. NIso outperforms the NFT and the autoencoder baseline (with random Möbius transformations applied during the fine-tuning phase) in addition to Möbius Convolutions ( MC ) [7], a SoTA handcrafted spherical network equivariant to Möbius transformations. We consider this dataset to present a significant challenge as shape classes are roughly conformally-related and thus the maps between their spherical parameterizations are only approximated by Möbius transformations.

Frame Skip

0

1

3

5

7

9

NIso

0.023 (

±

0.001)

0.031 (

±

0.001)

0.044 (

±

0.001)

0.057 (

±

0.002)

0.071 (

±

0.002)

0.081 (

±

0.002)

w/o

Ω

0.027 (

±

0.001)

0.045 (

±

0.001)

0.068 (

±

0.000)

0.084 (

±

0.000)

0.098 (

±

0.000)

0.110 (

±

0.000)

NFT [37]

0.022 (

±

0.003)

0.035 (

±

0.002)

0.059 (

±

0.001)

0.077 (

±

0.001)

0.091 (

±

0.001)

0.104 (

±

0.001)

DINOv2 [47]

0.027 (

±

0.003)

0.045 (

±

0.002)

0.068 (

±

0.001)

0.084 (

±

0.001)

0.097 (

±

0.001)

0.110 (

±

0.001)

BeIT [48]

0.028 (

±

0.003)

0.046 (

±

0.002)

0.068 (

±

0.001)

0.085 (

±

0.001)

0.098 (

±

0.001)

0.110 (

±

0.001)

ViT

0.020 (

±

0.001)

0.042 (

±

0.001)

0.066 (

±

0.000)

0.083 (

±

0.000)

0.097 (

±

0.000)

0.109 (

±

0.000)

NIso

NFT

ViT

Latent Maps

Select Eigenfunctions

Predicted Trajectories

Skip 0

1

3

5

7

9

TS

NFT

This makes the performance of NIso particularly notable as it suggests that our framework has the potential to offer a more flexible and effective alternative than specialized, handcrafted equivariant networks whose underlying group-based architectures, though elegant, can only approximate inexact symmetries. Furthermore, while the NFT is relatively more competitive with the encoder backbone unfrozen, its performance and that of the baseline indicate that a lack of existing equivariant structure cannot easily be overcome in the fine-tuning phase.

5.3 Camera Pose Estimation from Real-World Video

Last, we apply NIso to the task of camera pose estimation from real-world video on the CO3Dv2 dataset [49] following the procedure described in sec. 4.4. Due to the varying quality of the groundtruth trajectories in the dataset, we create train and test sets from the top 25 % of sequences in the dataset as ranked by the provided pose quality scores. In particular, we are interested in the general ability of our method to encode information about both small and large scale world space transformations. Thus we train and evaluate in a variable baseline regime, randomly skipping between 0 and 9 frames between pairs during the pre-training and fine-tuning phases. Evaluation is performed by computing the mean absolute trajectory error ( ATE ) between the ground truth trajectories and those recovered by our method over the sequences in the evaluation set. To understand the efficacy of our approach at different scales, we report the mean ATE over six splits consisting of the same sequences in the evaluation set with frame skips of 0, 1, 3, 5, 7, and 9.

We compare against the NFT using the same pre-training and fine-tuning procedure along with a transformer baseline. The latter is inspired by the recent success of such models in 3D vision tasks [50, 51], and uses the same ViT-based architecture proposed in DUSt3R [51], with the decoder modified to directly predict the parameters of the relative camera pose. As the weights of the NIso and NFT backbones are frozen during the fine-tuning phase, the transformer baseline is trained from scratch in the fine-turning phase to more directly compare the descriptiveness of the latent representations relative to the information contained in the raw observations. We also compare

against two strong representation learning baselines. We extract features from both images using two pre-trained state-of-the-art vision foundation models - DINOv2[47] and BeIT[48] - and pass the tokens into the modified DUSt3R-style decoder which is trained to predict the pose.

We additionally train and evaluate an ablative version of NIso which does not learn an operator. Here τ is predicted directly between encodings and is required only to be orthogonal with the equivariance and reconstruction losses alone enforced during training. We note that computing a block-diagonalization loss on τ directly lacks justification as it would promote maps that preserve contiguous chunks of spatial indices in the latent tensors which are ordered arbitrarily. Thus, this regime serves to evaluate the degree to which the regularization through learned Ω -commutativity forces the latent transformations to reflect geometric relationships in world space.

Results are shown in Tab. 3, averaged over five randomly initialized pre-training and fine-tuning runs with standard errors. All methods perform similarly with 0 frame skip but diverge afterwards, with NIso achieving significantly lower ATE values as the skip length increases. Notably, the ablative version of our method is consistently among the worst performers, suggesting that isometric regularization is also critical to encode information about world space transformations. Overall the NFT achieves the second best performance, slightly outperforming NIso at 0 frame skip but diverging thereafter. However, the latent maps it recovers are neither sparse nor exhibit condensed structure (Tab. 3, center left), and we hypothesize that its inability to effectively regularize maps beyond linearity makes it difficult for the network to discover a consistent transformation model that applies across scales. This could cause the network to focus on a specific regime at the expense of others, which may explain its relatively strong performance at 0 frame skip. The transformer baseline and representation learners are also highly flexible, and an analogous line of reasoning could explain their similar error profile.

6 Discussion

Limitations. Akey factor limiting the broader applicability of our approach to geometry processing and graph-based tasks is an inability to learn and transfer an operator between domains with varying connectivity. In addition, NIso does not explicitly handle partiality or occlusion between input pairs which are ubiquitous in real-world data and likely degrade its performance in the pose estimation task. We seek to address these limitations in future work.

Conclusion. In this paper we introduce Neural Isometries, a method which converts challenging observed symmetries into isometries in the latent space. Our approach forms an effective backbone for self-supervised representation learning, enabling simple off-the-shelf equivariant networks to achieve strong results in tasks with complex, non-linear symmetries. Furthermore, isometric regularization produces latent representations that are geometrically informative by encoding information about transformations in world space, and we demonstrate that robust camera poses can be extracted from the isometric maps between latents in a general baseline setting.

Acknowledgements. Use of the CO3Dv2 dataset is solely for benchmarking purposes and limited exclusively to the experiments described in sec. 5.3. We thank Ishaan Chandratreya and Hyunwoo Ryu for helpful discussions and David Charatan for his aesthetic oversight and for granting us permission to use his images captured for FlowMap [52] in Fig. 1-2.

Vincent Sitzmann was supported by the National Science Foundation under Grant No. 2211259, by the Singapore DSTA under DST00OECI20300823 (New Representations for Vision and 3D Self-Supervised Learning for Label-Efficient Vision), by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/ Interior Business Center (DOI/IBC) under 140D0423C0075, by the Amazon Science Hub, and by IBM.

References

Taco Cohen and Max Welling. Group equivariant convolutional networks. In Proceedings of the International Conference on Machine Learning (ICML) , pages 2990-2999. PMLR, 2016.

Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5028-5037, 2017.

Carlos Esteves, Ameesh Makadia, and Kostas Daniilidis. Spin-weighted spherical cnns. Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. In Proceedings of the International Conference on Machine Learning (ICML) , pages 3165-3176. PMLR, 2020.

Daniel E Worrall and Max Welling. Deep scale-spaces: Equivariance over scale. arXiv preprint arXiv:1905.11697 , 2019.

Lachlan E MacDonald, Sameera Ramasinghe, and Simon Lucey. Enabling equivariance for arbitrary lie groups. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8183-8192, 2022.

Thomas W Mitchel, Noam Aigerman, Vladimir G Kim, and Michael Kazhdan. Möbius convolutions for spherical cnns. In ACM Trans. Graph. , pages 1-9, 2022.

Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical cnns. arXiv preprint arXiv:1801.10130 , 2018.

Marc Finzi, Max Welling, and Andrew Gordon Wilson. A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups. In International conference on machine learning , pages 3318-3328. PMLR, 2021.

Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess E Smidt, and Boris Kozinsky. E (3)-equivariant graph neural networks for dataefficient and accurate interatomic potentials. Nature communications , 13(1):2453, 2022.

Vıctor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks. In International conference on machine learning , pages 9323-9332. PMLR, 2021.

Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik J Bekkers, and Max Welling. Geometric and physical quantities improve e (3) equivariant message passing. arXiv preprint arXiv:2110.02905 , 2021.

Alexander Bogatskiy, Brandon Anderson, Jan Offermann, Marwah Roussi, David Miller, and Risi Kondor. Lorentz group equivariant neural network for particle physics. In International Conference on Machine Learning , pages 992-1002. PMLR, 2020.

Erik J Bekkers. B-spline cnns on lie groups. arXiv preprint arXiv:1909.12057 , 2019.

Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. ACM Transactions on Graphics (tog) , 38(5):1-12, 2019.

Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veliˇ ckovi´ c. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478 , 2021.

Omri Puny, Matan Atzmon, Heli Ben-Hamu, Ishan Misra, Aditya Grover, Edward J Smith, and Yaron Lipman. Frame averaging for invariant and equivariant network design. arXiv preprint arXiv:2110.03336 , 2021.

Maks Ovsjanikov, Mirela Ben-Chen, Justin Solomon, Adrian Butscher, and Leonidas Guibas. Functional maps: a flexible representation of maps between shapes. ACM Transactions on Graphics (ToG) , 31(4): 1-11, 2012.

Simone Melzi, Jing Ren, Emanuele Rodola, Abhishek Sharma, Peter Wonka, and Maks Ovsjanikov. Zoomout: Spectral upsampling for efficient shape correspondence. arXiv preprint arXiv:1904.07865 , 2019.

Robin Magnet and Maks Ovsjanikov. Memory-scalable and simplified functional map learning. arXiv preprint arXiv:2404.00330 , 2024.

Mingze Sun, Shiwei Mao, Puhua Jiang, Maks Ovsjanikov, and Ruqi Huang. Spatially and spectrally consistent deep functional maps. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14497-14507, 2023.

Ahmed Abdelreheem, Abdelrahman Eldesokey, Maks Ovsjanikov, and Peter Wonka. Zero-shot 3d shape correspondence. In SIGGRAPH Asia 2023 Conference Papers , pages 1-11, 2023.

Or Litany, Tal Remez, Emanuele Rodola, Alex Bronstein, and Michael Bronstein. Deep functional maps: Structured prediction for dense shape correspondence. In Proceedings of the IEEE international conference on computer vision , pages 5659-5667, 2017.

Mikhail Panine, Maxime Kirgo, and Maks Ovsjanikov. Non-isometric shape matching via functional maps on landmark-adapted bases. In Computer graphics forum , volume 41, pages 394-417. Wiley Online Library, 2022.

Souhaib Attaiki, Gautam Pai, and Maks Ovsjanikov. Dpfm: Deep partial functional maps. In 2021 International Conference on 3D Vision (3DV) , pages 175-185. IEEE, 2021.

Dongliang Cao, Paul Roetzer, and Florian Bernard. Revisiting map relations for unsupervised non-rigid shape matching. arXiv preprint arXiv:2310.11420 , 2023.

Maks Ovsjanikov, Etienne Corman, Michael Bronstein, Emanuele Rodolà, Mirela Ben-Chen, Leonidas Guibas, Frederic Chazal, and Alex Bronstein. Computing and processing correspondences with functional maps. In SIGGRAPH ASIA 2016 Courses , pages 1-60. 2016.

Yilun Du, Katie Collins, Josh Tenenbaum, and Vincent Sitzmann. Learning signal-agnostic manifolds of neural fields. Advances in Neural Information Processing Systems (NeurIPS) , 34:8320-8331, 2021.

Sharut Gupta, Joshua Robinson, Derek Lim, Soledad Villar, and Stefanie Jegelka. Learning structured representations with equivariant contrastive learning. 2023.

Sangnie Bhardwaj, Willie McClinton, Tongzhou Wang, Guillaume Lajoie, Chen Sun, Phillip Isola, and Dilip Krishnan. Steerable equivariant representation learning. arXiv preprint arXiv:2302.11349 , 2023.

Nima Dehmamy, Robin Walters, Yanchen Liu, Dashun Wang, and Rose Yu. Automatic symmetry discovery with lie algebra convolutional network. Advances in Neural Information Processing Systems , 34:2503-2515, 2021.

Artem Moskalev, Anna Sepliarskaia, Ivan Sosnovik, and Arnold Smeulders. Liegg: Studying learned lie group generators. Advances in Neural Information Processing Systems , 35:25212-25223, 2022.

Jianke Yang, Robin Walters, Nima Dehmamy, and Rose Yu. Generative adversarial symmetry discovery. In International Conference on Machine Learning , pages 39488-39508. PMLR, 2023.

Jianke Yang, Nima Dehmamy, Robin Walters, and Rose Yu. Latent space symmetry discovery. arXiv preprint arXiv:2310.00105 , 2023.

Ho Yin Chau, Frank Qiu, Yubei Chen, and Bruno Olshausen. Disentangling images with lie group transformations and sparse coding. arXiv preprint arXiv:2012.12071 , 2020.

Takeru Miyato, Masanori Koyama, and Kenji Fukumizu. Unsupervised learning of equivariant structure from sequences. Advances in Neural Information Processing Systems (NeurIPS) , 35:768-781, 2022.

Masanori Koyama, Kenji Fukumizu, Kohei Hayashi, and Takeru Miyato. Neural fourier transform: A general approach to equivariant representation learning. Proceedings of the International Conference on Learning Representations (ICLR) , 2023.

Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas J Guibas. Vector neurons: A general framework for so (3)-equivariant networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 12200-12209, 2021.

user1551 (https://math.stackexchange.com/users/1551/user1551). Solve ∥ xa -b ∥ subject to xc = cx . Mathematics Stack Exchange. URL https://math.stackexchange.com/q/4886103 . URL:https://math.stackexchange.com/q/4886103 (version: 2024-03-23).

Nicolas Donati, Abhishek Sharma, and Maks Ovsjanikov. Deep geometric functional maps: Robust feature learning for shape correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8592-8601, 2020.

Souhaib Attaiki and Maks Ovsjanikov. Understanding and improving features learned in deep functional maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1316-1326, 2023.

Jiashi Feng, Zhouchen Lin, Huan Xu, and Shuicheng Yan. Robust subspace segmentation with blockdiagonal prior. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3818-3825, 2014.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Nicholas Sharp, Souhaib Attaiki, Keenan Crane, and Maks Ovsjanikov. Diffusionnet: Discretization agnostic learning on surfaces. ACM Transactions on Graphics (TOG) , 41(3):1-16, 2022.

Mircea Mironenco and Patrick Forré. Lie group decompositions for equivariant neural networks. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/ forum?id=p34fRKp8qA .

Z Lian, A Godil, B Bustos, M Daoudi, J Hermans, S Kawamura, Y Kurita, G Lavoua, P Dp Suetens, et al. Shape retrieval on non-rigid 3d watertight meshes. In Eurographics workshop on 3d object retrieval (3DOR) . Citeseer, 2011.

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 , 2023.

Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254 , 2021.

Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10901-10911, 2021.

Philippe Weinzaepfel, Vincent Leroy, Thomas Lucas, Romain Brégier, Yohann Cabon, Vaibhav Arora, Leonid Antsfeld, Boris Chidlovskii, Gabriela Csurka, and Jérôme Revaud. Croco: Self-supervised pretraining for 3d vision tasks by cross-view completion. Advances in Neural Information Processing Systems , 35:3502-3516, 2022.

Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy, 2023.

Cameron Smith, David Charatan, Ayush Tewari, and Vincent Sitzmann. Flowmap: High-quality camera poses, intrinsics, and depth via gradient descent. arXiv preprint arXiv:2404.15259 , 2024.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Michael Kazhdan, Jake Solomon, and Mirela Ben-Chen. Can mean-curvature flow be modified to be non-singular? In Computer Graphics Forum , volume 31, pages 1745-1754. Wiley Online Library, 2012.

Jian Sun, Maks Ovsjanikov, and Leonidas Guibas. A concise and provably informative multi-scale signature based on heat diffusion. In Computer graphics forum , volume 28, pages 1383-1392. Wiley Online Library, 2009.

A Derivations and Implementation Details

A.1 Isometries in the Eigenbasis

Here we show that isometries manifest as orthogonal matrices that commute with the diagonal matrix of eigenvalues in the operator eigenbasis. Suppose we are given a PSD operator Ω and diagonal mass matrix M , with the former expressed in terms of its eigendecomposition as in Equation (4) for eigenfunctions Φ and eigenvalues Λ .

Let τ be an isometric functional map satisfying the conditions in Equation (3) with τ Ω its projection into the eigenbasis as in Equation (5). Noting that the condition Φ ⊤ M Φ = I | N | implies that Φ -1 = Φ ⊤ M and Φ -⊤ = M Φ , it follows that

and thus τ Ω is orthogonal. Furthermore, it follows that

so τ Ω Λ = Λ τ Ω .

A.2 Recovering τ Ω

Here we derive the approximate solution to the least squares problem in Equations (7 - 8). Specifically, let Λ = diag ( { λ i } 1 ≤ i ≤ k ) ∈ R k × k be a diagonal matrix of non-negative eigenvalues with Λ ii ≤ Λ i +1 ,i +1 . Furthermore, suppose that there are q distinct eigenvalues { ˜ λ i } 1 ≤ i ≤ q , ˜ λ i ≤ ˜ λ i +1 each with multiplicity m i . Thus, Λ can be expressed as the direct sum

with I m i denoting the m i × m i identity matrix. It follows that any k × k matrix π satisfying π Λ = Λ π must be of the form

That is, π must be a block diagonal matrix with the size of each block determined by the multiplicity of the eigenvalues of Λ .

Now, given any A,B ∈ R k × d consider the minimization problem

consisting of finding an orthogonal, Λ -commuting map minimizing the distance between A and B under the Frobenius norm. Writing

the solution to the minimization problem in Equation (14) is equivalently expressed as the direct sum of the solutions to the orthogonal sub-problems corresponding to each block [39] with

Now, for arbitrary n let κ : R n × n → O ( n ) denote the Procrustes projection to the nearest orthogonal matrix such that for any M ∈ R n × n with SVD M = U Σ V ⊤ , κ ( M ) ≡ UV ⊤ . Thus, the solution to the minimization problem is given by

which, following from the properties of the SVD, can be expressed equivalently via a single k × k Procrustes projection such that

Here we observe that by defining P Λ ∈ R k × k to be the eigenvalue mask given by

In practice we substitute P Λ in Equation (19) with the fuzzy analogue [ P Λ ] ij = exp( -| λ i -λ j | ) , which we find better facilitates the flow of backwards gradients to the parameters of Ω . Replacing A and B with E Ω ( ψ ) and E Ω ( Tψ ) we arrive at the approximate solution for τ Ω in Equation (8).

A.3 Graph Laplacian of P Λ

Given the k × k eigenvalue mask P Λ with [ P Λ ] ij = exp( -| λ i -λ j | ) , we form its graph Laplacian as

we have and thus

with 1 ∈ R k the vector of ones.

B Reproducibility

Code and experiments are available at https://github.com/vsitzmann/neural-isometries .

B.1 Hardware

All experiments were performed on a single NVIDIA A6000 GPU with 48 GB of memory.

B.2 Approximating the Laplacian

In the toric/spherical experiments, the model consists of a single NIso layer. which learns a fullrank/low-rank operator and mass matrix, with the former parameterized by an orthogonal matrix of eigenvectors Φ ∈ R 256 × 256 / Φ ∈ R 256 × 64 and diagonal matrix of non-negative eigenvalues Λ . During training, input features are created by stacking 86 16 × 16 × 3 examples from ImageNet into a single 16 × 16 × 258 tensor representing an observation ψ . In the toric experiments, random circular shifts about both spatial axes are applied to form Tψ . In the spherical experiments, the images are assumed to lie on a 16 × 16 Driscoll-Healy spherical grid and random rotations in SO (3) are sampled and applied to form Tψ . As the encoder and decoders are the identity maps, the equivariance and reconstruction losses are equivalent, and the objective in Equation (11) is minimized with α = 0 and β = 0 . 1 . The models is trained with spectral dropout for 100,000 iterations with a batch size of 1 using the AdamW optimizer [53] with a weight decay of 10 -4 . The learning rate follows a schedule consisting of a 2,000 step warm up from 0 . 0 to 5 × 10 -4 and afterwards decays to 5 × 10 -5 via cosine annealing. Training takes approximately 6 hours.

B.3 Homography-Perturbed MNIST

Architecture. The encoder and decoder are mirrored 2-level ConvNets. Specifically, each level consists of three convolutional ResNet blocks with 128 and 256 channels at the finest and coarsest resolutions respectively. A mean pool (nearest neighbor unpool) layer halving (doubling) the spatial resolution bridges the two levels.

The latent space has 32 dimensions. A rank k = 32 operator Ω and mass matrix M are learned, with the former parameterized by 32 eigenfunctions and non-negative eigenvalues.

The equivariant classification head consists of a 2-layer VN-MLP [38] with 128 channels, followed by an output layer which computes invariant features from the inner products between vectors which are passed to a standard linear layer to output class predictions.

Pre-Training. Following [6], MNIST digits are padded to 40 × 40 , and pairs are produced by warping digits with respect to homographies T sampled from the proposed distribution. In the triplet regimes, tuples are created by applying T 2 in addition to T . Both the pairwise and triplet regimes are trained with respect to the composite loss with α = 0 . 5 and β = 0 . 1 . The two ablative regimes are trained with α = 0 and β = 0 , respectively. In the triplet regime, denoting τ and σ to be the maps estimated between ( E ( ψ ) , E ( Tψ )) and ( E ( Tψ ) , E ( T 2 ψ )) respectively, the equivariance and reconstruction losses are formulated as

and

The autoencoders are trained without spectral dropout for 50 , 000 steps with a batch size of 16 using the AdamW optimizer with a weight decay of 10 -4 . The learning rate follows a schedule consisting of a 2,000 step warm up from 0 . 0 to 5 × 10 -4 and afterwards decays to 5 × 10 -5 via cosine annealing. Training takes approximately 30 minutes.

Fine-Tuning. During the fine tuning phase, the decoder is discarded, and the weights of the encoder, operator Ω , and mass M are frozen. Examples ψ are encoded, and their eigenspace projections E Ω are passed to the classification head to form class predictions under a standard softmax cross-entropy loss. Training is performed for 10,000 iterations with a batch size of 16 using the AdamW optimizer the same weight decay and learning rate schedule as used previously. Training takes approximately one minute. Evaluation is performed on the fixed test set proposed in [6].

Baselines. We implement the NFT following the procedure described in the paper [37]. While a ViT-based encoder is proposed in the original paper, we find that it produces worse evaluation results than the 2-layer ConvNet encoder described above and thus use the latter in our evaluations to provide a fair comparison. We follow the authors' proposed implementation and consider only a reconstruction loss and learn a diagonalization transform in a post-processing step. Otherwise, the NFT is pre-trained, fine-tuned, and evaluated identically to our own approach, and uses the same equivariant classification head. We note that the dimension of the latent map constructed by the NFT is the same as τ Ω (32 × 32) and that the tensor passed to the classification head is the same size for both our method and the NFT.

The autoencoder baseline consists of the same encoder and decoder, pre-trained identically but with respect to the standard reconstruction loss ∥D ( E ( ψ )) -ψ ∥ without considering T -related pairs. During the fine-tuning phase, we apply data augmentation to the input images by transforming them by homographies sampled from the distribution proposed in [6] before encoding. Latents are passed directly to a standard non-equivariant 2-layer MLP with 128 channels to form class predictions. Otherwise, fine-tuning and evaluation are performed in the same manner.

B.4 Conformal Shape Classification

Dataset. Experiments are performed using the conformally extended SHREC '11 shape classification dataset proposed in [7], with 30 random conformal transformations are applied to each shape to extend the original dataset [46]. The augmented dataset contains 25 distinct shape classes. Each shape is mapped to the sphere using the method of [54]. Input features are taken to be the values of the heat kernel signature [55] computed on the original mesh at 16 timescales logarithmically distributed in the

range [ -2 , 0] and rasterized to 96 × 192 spherical grids. Pairs of T -related observations are formed by randomly selecting two conformal augmentations of the same shape. For each set of training and fine-tuning runs, train and evaluation splits are randomly generated by selecting respectively 10 and 4 of the 20 sets of conformally augmented shapes per class.

Architecture. The encoder and decoder are mirrored 4-level ConvNets, with two ResNet blocks per level and 32, 64, 128, 256 channels per layer from finest to coarsest resolution. Mean pooling and nearest neighbor upsampling are used to halve and double the resolution between layers.

The latent space has 64 dimensions. A rank k = 64 operator Ω and mass matrix M are learned, with the former parameterized by 64 eigenfunctions and non-negative eigenvalues.

The equivariant classification head consists of a 4-layer VN-MLP with 128 channels, using the same output layer as described in sec. (B.3) to produce class predictions.

Pre-Training. Training is performed with respect to the composite loss with α = 0 . 5 and β = 0 . 01 . The model is trained without spectral dropout for 50,000 iterations with a batch size of 16 using the AdamW optimizer with a weight decay of 10 -4 . The learning rate follows a schedule consisting of a 2,000 step warm up from 0 . 0 to 5 × 10 -4 and afterwards decays to 5 × 10 -5 via cosine annealing. Training takes approximately four hours.

Fine-Tuning. During the fine tuning phase, the decoder is discarded and the encoder, operator Ω , and mass matrix M are jointly optimized with the equivariant classification head using the standard softmax cross-entropy loss. Training is performed for 10,000 iterations with a batch size of 16 using the AdamW optimizer, with the same weight decay and learning rate schedule as used previously. Training takes approximately one hour.

Baselines. We pre-train, fine-tune, and evaluate the NFT and autoencoder baseline using the same autoencoder architecture, classification head, and training regimes in the same manner as sec. (B.3). Here data augmentation is applied during the fine-tuning phase by randomly sampling and applying Möbius transformations with scale factors proportional to those estimated between the spherical parameterizations of the shapes. Again, the both the size of the latent map and size of the tensor passed to the classification are the same size for both our method and the NFT. Here, the weights of the encoder are left unfrozen, and are jointly optimized with the classification head during the fine-tuning phase.

B.5 Camera Pose Estimation from Real-World Video

Dataset. Experiments are performed with a subsection of the CO3Dv2 dataset [49]. Given the high-degree of variability in the quality of the ground truth trajectories, we consider only the top 25% of sequences in the dataset as ranked by the provided pose quality score, slightly under 9,000 trajectories in total. An evaluation set is formed by withholding 10% of said trajectories, with the rest used for training. The list sequences in the train and evaluation sets will be made available along with the code release. As the videos in the dataset are of different resolutions, we center crop each frame to the minimum of its height and width dimensions and resize to 144 × 144 .

Architecture. The encoder and decoder are mirrored 3-level ConvNets, with four ResNet blocks per level and 64, 128, 256 channels per layer from finest to coarsest resolution. Mean pooling and nearest neighbor resolution are used to halve and double the resolution between layers.

The latent space has 128 dimensions. A rank k = 128 operator Ω parameterized by and mass matrix M are learned, with the former parameterized by 128 eigenvectors and eigenvalues.

The pose extraction network consists of a standard 5-layer MLP with 512 channels, which outputs 9-dimensional tensor representing the parameters of an SE (3) transformation (translation + two vectors used to form the rotation matrix through cross products).

Pre-Training. During pre-training, T -related pairs of observations are formed by randomly selecting adjacent frames from sequences with a frame skip between 0 and 10 . Training is performed with respect to the composite loss with α = 0 . 5 and β = 0 . 025 . The model is trained for 200,000 iterations with a batch size of 8 using the AdamW optmizer with a weight decay of 10 -4 . The learning rate follows a schedule consisting of a 2,000 step warm up from 0 . 0 to 5 × 10 -4 and afterwards decays to 5 × 10 -5 via cosine annealing. Training takes approximately 12 hours.

Fine-Tuning. During the fine-tuning phase, the weights of the encoder, operator Ω , and mass matrix M are frozen. As in the pre-training phase, pairs of observations are passed to the decoder and τ Ω ∈ R 128 × 128 is estimated between the eigenspace projection of the encodings. Then, τ Ω is vectorized and passed to the MLP head to predict the parameters of the camera pose.

Training is performed with respect to a two term loss measuring the degree to which the predicted translation and rotation deviate from those of the ground truth pose. Denoting R Ω and R to be the predicted and ground truth poses respectively, the rotational component of the loss is given by

The transitional component of the loss must be scale invariant as the depth scale factor is unknown between sequences. To handle this, we consider sub-batches formed from pairs of frames all belonging to the same sequence. Denoting t Ω and t to be the matrices consisting of the sub-batched predicted and ground truth translations stacked column-wise, we form the translational component of the loss via

is the scale factor that minimizes L T over the pairs of frames in the sub-batch. The total loss is given by L = L R + L T

The prediction head is trained without spectral dropout for 20,000 iterations with a batch size of 16 (consisting of 4 sub-batches of pairs of frames from 4 sequences) using the AdamW optimizer with the same weight decay and learning rate schedule as used previously. Training takes approximately one 80 minutes.

Baselines. The NFT is pre-trained, fine-tuned, and evaluated using the same autoencoder architecture, prediction head, and training regime as our own method in the same manner as is done in the preceding experiments. We note that both our method and the NFT construct latent maps of the same dimension, which are passed to identical prediction heads.

The transformer baseline is based on the CroCo/DUSt3R[50, 51] architectures. As with our approach, input frames are passed to the same decoder. Here the encoder consists of 12-layer VITs with a patch size of 16, with 16 heads and 512 channels per layer. Adjacent frames are independently encoded, and then concatenated along the spatial dimension in addition to a single learned token. The resulting tensor is passed to a 12-layer transformer decoder which mirrors the encoder with 16 heads and 512 channels per layer. Subsequently, the learned token is extracted from the output and passed through a linear layer to recover the predicted pose parameters.

Additionally, we also evaluate the efficacy of DINOv2[47] and BeIT[48] as representation learners for the pose estimation task. Pairs of adjacent frames are passed through the pre-trained models and the feature tokens for each image from the final layers are extracted. These are used as input to the same DUSt3R[47]-style decoder described above which is trained to predict the parameters of the relative camera pose.

The VIT model and DINOv2/BeIT prediction heads are trained from scratch in the fine-tuning phase, using the same loss and optimization procedure as described above.

C Additional Results

Below we provide additional results including a quantiative evaluation of learned equivariance, visualizations of the eigenfunctions Φ and mass matrices M learned in each experiment, and comparisons of estimated camera trajectories recovered by each method in the pose prediction experiments. For the latter, examples of failure cases are also included.

where

Homography MNIST

Conformal SHREC `11

CO3D

Learned Eigenfunctions

Learned Masses

Predicted Trajectories

Skip 0

1

3

5

7

9

onarcor

Tnnursier

F

Predicted Trajectories

Skip 0

1

3

5

7

9

可

写

Predicted Trajectories

Skip 0

1

3

5

7

9

ADULT COLORIHG

BOOK

EKEFIEEO

H

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims made in the abstract and introduction are supported by experimental results and are contextualized with respect to competing methods.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations are discussed in the final section of the paper (6).

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Derivations of relevant expressions are provided in the supplement.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide reproducibility details for each experiment in the supplement.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experiment details are provided in the paper and supplement.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report all quantitative results in terms of the mean over five random initializations (and dataset splits, where applicable) with the standard error.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All experiments were performed on the same device, the details of which are described in the supplement.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: To the best of our knowledge, we conform to the Code of Ethics. At this time we do not see our method providing a straightforward avenue for abuse by bad actors.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Due to the low-level nature of our method, we do not see it directly facilitating any negative social impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All data used is cited in accordance with the provided licenses.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an autoencoder framework that encodes the input symmetries into isometries in the latent space. The equivariance in the latent space is captured by a functional map $\\tau$, which is regularized to be an isometry. Instead of hard constraints, the equivariance of the system is encouraged via the optimized $\\tau$ and the loss function design. The proposed method is evaluated on a homography-perturbed MNIST, conformal shape classification, and camera pose estimation tasks. The experiments show the proposed method performs on par with the handcrafted equivariant baselines and outperforms a baseline with a similar approach (Neural Fourier Transform). \n\nI’m positive about the paper. However, there are some main concerns that I would like to know the answers to. I’m willing to raise the scores once they are addressed.\n\nSTRENGTHS:\n1. The paper is well-written and easy to follow. \n2. The paper proposes a novel framework that softly models the latent space equivariance as functional maps. \n3. Unlike handcrafted equivariant networks that are limited to certain groups, the proposed framework can model different input transformations. \n4. The proposed method is shown to perform on par with the baselines in homNIST and camera pose estimation tasks, and outperforms the baselines in the conformal shape classification task.\n\nWEAKNESSES:\n1. Although being shown to be effective in the experiments, the constraint of $\\tau$ being an isometry seems a bit heuristic. The paper can benefit from some theoretical investigation on why constraining to being isometries helps the performance.\n2. Following up on the previous point, can all transformations be modeled as isometric functional maps in the latent space? That being said, does restricting $\\tau$ to isometries affect the types of transformations the network is able to model? \n3. Since the equivariance is only obtained softly via the loss function design, it’s essential to see how much equivariant is maintained/lost in the latent space. I recommend the authors report some measures of equivariance and compare them with the handcrafted methods. (For example, equivariant error/loss in the latent space.)\n4. There are several heuristic designs in the proposed framework, and some of them affect the performance significantly. For example, it is not explained clearly why a smooth multiplicative mask is required to ensure $\\tau_\\Omega$ is semi-diagonal. The effect of such approximation is also not discussed/evaluated. The second is the multiplicity loss, which greatly affects the performance of the framework. \n5. I appreciate the comparisons with handcrafted equivariant networks. However, I believe the paper can be greatly strengthened by comparing it with the same autoencoder framework while not enforcing the latent equivariance and training with data augmentation. This can demonstrate the benefits of the proposed latent equivariant design.\n\nQUESTIONS:\n1. Since there are no hard-coded constraints (except for the regularization) in the system, the proposed method can work with different input transformations. Do you think this work can be applied to automatic symmetry discovery tasks? \n2. I understand that the inner product needs to be preserved to construct an isometry, but why does $\\tau$ have to commute with $\\Omega$?\n3. Ln 207, “we show in experiments that a major benefit of our isometric regularization is that our multiplicity loss (promoting a diagonal-as-possible and thus sparse and compact $\\tau_\\Omega$) can serve as an effective substitute for access to triples.” Is this true for all three experiments, or is it only validated on homNIST?\n\nLIMITATIONS:\nAs mentioned in the paper, the proposed network is unable to perform graph-based tasks. In addition, the equivariance in the latent space is not evaluated. As a result, it is unclear how the equivariance is preserved in the latent space. Lastly, by restricting the functional maps to be isometries, the transformations that the network can model might be limited.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-30"}

comments: SUMMARY:
This paper introduces Neural Isometries, which is an autoencoder framework learning to map the observation space to a general-purpose latent space wherein encodings are related by isometries whenever their corresponding observations are geometrically related in world space. Several experiments, including camera pose estimation, are conducted.

STRENGTHS:
The idea of transforming complicated equivariances in observation space into isometries in latent space is intuitive and interesting. Experiments have demonstrated the potentials of this method in practical tasks. The paper is well written and easy to understand.

WEAKNESSES:
This is an interesting paper but in-depth discussion is lacking. My main concern is that there is no quantitative or theoretical description of the equivariance. For example, 
1.  How equivariant is the model? i.e. How large is the equivariance loss?
2. What type of equivariance in the observation space can be modeled? Will the model fail when the pertubations are too strong?

In addition, the experiment results are relatively weak. Only 3 small scale experiments are conducted, and only a few baselines are considered.

QUESTIONS:
1. If the observation space is already isometric, e.g. rotate by 90 degrees, what can be said about the latent sapce?
2. How robust is this model? For example, how will the latent vector change when the inputs are partially observed (masked)?

LIMITATIONS:
See the weakness section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a generic equivariant ML framework by learning latent representations are are modelled to be related by an isometry. There are several important design choices made by the authors: (1.) An autoencoder framework that keeps the spatial structure (i.e. images get encoded into images) (2.) The isometric map is represented compactly with a basis that is the outcome of an eigen- -decomposition of a PSD operator (3.) The isometry manifests as an orthogonal matrix with a sparse block diagonal structure in its reduced projected form. \n\nExperiments are shown for (1.) Learning the spectrum of a Toric Laplacian and comparing it qualitatively to the real one  in Fig 3 (2.) Homography perturbed MNIST (3.) Conformal shape classification and (4.) Camera Pose Estimation from real-world video. The results (1) show a good proof of concept and (2), (3) and (4) demonstrate a clear superiority over the closest conceptual baseline NFT. (2.) Also makes the case for achieving comparable performance with simpler equivariant architectures using the NIso latent space.\n\nSTRENGTHS:\n- Overall the biggest strength of this paper is that it was an enjoyable read. I found the hypothesis (i.e. learning latent codes related by an isometry for arbitrary transformations in the observation space) to be interesting and reasonable. The overall writing and description of the method is very nice. \n- The Experiments especially (1.) was a good proof of concept.\n\nWEAKNESSES:\n- I am actually quite confused on how the operator \\Omega and Mass matrix M is learned? The Laplace Beltrami is a very structured object. See all requirements enumerated in Wardetzky, Max, et al. \"Discrete Laplace operators: no free lunch.\" Symposium on Geometry processing. Vol. 33. 2007.). It is unclear what the nature of this operator is and its learning feels ad-hoc and unmotivated. \n- Do the authors make sure that the optimization for the functional map is well-posed? Typically in the original formulation a \"good\" functional map is the outcome of using many-many descriptors in addition to regularization constraints like commutativity with laplacian. If I understand correctly the proposed formulation seems to be confident in learning a structured functional map (like Fig 2) using just one pair of corresponding functions ? \n- As a high-level opinion: Overall the experimental section could be stronger. The \"discovery\" of the toric laplacian was nice, but a demonstration on roto-translations perhaps with scale and/or spherical images with SO(3) would be extremely convincing. I would also recommend an evaluation that focuses beyond NFT and actually quantifying the complexity of different equivariant models. \n- As mentioned by the authors, the requirement of generalizing to domains with diverse connectivity is indeed a hard one, but important.\n\nQUESTIONS:\n- The basis functions visualized in Fig3 (third column) are learned or from the actual toric laplacian? what are the various rows? \n- It would also be useful to report some feature of complexity in Table 1 indicating \"meticulously-engineered, handcrafted networks\"\n- More examples of evidence showing good basis functions \n- What is a perturbation in Section 5.1?\n\nLIMITATIONS:\nI think the authors make a fair declaration of the limitations of their work in Section 6.\nI would make a stronger vote after the rebuttal. I think this paper has many ingredients to warrant acceptance: clever idea, good argumentation, and decent proof of concept. It lacks clarity in some places and the evaluation does not appear equally general as is the description of the method. For this reason, I maintain a borderline acceptance with a full intention to re-assess in the next round.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe main motivation behind Neural Isometry is as follows: most of real world transformations in vision and geometry processing lack identifiable group structure and therefore challenging for prior work in equivariant learning that assumes such knowledge apriori. The paper proposes an auto encoder framework that maps observations (related by some geometric transformations) to a latent space where embeddings are  related by a linear transform. The framework leverages the existing functional map framework and corresponding regularization to learn this latent space. The paper validates and compares their performance with the baseline NFT (Neural Fourier Transform) that assumes the knowledge of the group a priori.\n\nSTRENGTHS:\n-  The problem of learning structured latent space in a self supervised equivariant learning set up is relevant.\n\n- The proposed framework is formalized clearly and in detail in Section 4.\n\nWEAKNESSES:\nAs mentioned many times in paper, Instead of theoretical justification ( as in NFT), this paper seeks to validate the efficacy of approach experimentally on real world tasks.\n\n- Limited Experimental Set up: The datasets used are still very small scale (MNIST,) or synthetic (SHREC) and therefore far from the real world data.  The submission mainly follows and compares with one baseline (NFT) that was implemented by authors themselves.  The submission could motivate the problem/solution by probing/playing with the learned structured space e.g. in geometry processing tasks [1]. \n\n- Limitation of Functional map: One of the biggest limitation of functional map framework is its applicability to real world data where partiality is ubiquitous (due to occlusion etc). Therefore, Neural Isometry by design also inherits this limitation.  experiments are shown on MNIST/SHREC where there is no occlusion or partiality. In the third benchmark, instead of skipping the odd frame, does the performance decays rapidly between distant frames?\n\n- Presentation: The submission contains several typos, convoluted sentences (Line 85-88) and statements without a citation. e.g.\n\n-- Line 49: < Lorentz transformations with d'almbert operator of Minkowski space> citation is missing here. Also, why do we need to know this fact in the introduction of this paper? \n\n-- Line 42: <by preserving the spatial dimensions> not sure what does this mean after reading the next 2 lines. Please clarify\n\n-- Line 89: <orthogonal relaxation of FM> citation or please detail what those are.\n\n- Related work: The submission cites a dozen variants of deep functional map paper [2] but not the deep functional map paper itself [2] that inspired them. Please justify how [20,21,22,39] are related to this work or inspired this work given other deep functional pipeline cited in paper. The reviewer would instead relate/distinguish this work with [3].\n\n\n1. Composite Shape Modeling via Latent Space Factorization, Dubrovina et al, 2021\n2. Deep Functional Maps Litany et al. 2017\n3. Map-based exploration of intrinsic shape differences and variability  Rustamov et al. 2013\n\nQUESTIONS:\nplease see above.\n\nLIMITATIONS:\nThe submission should state the scalability of this approach given its reliance on Laplacian Eigen basis.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-15"}