Neural Isometries: Taming Transformations for Equivariant ML

Authors: thomas w mitchel
michael  taylor
vincent  sitzmann
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 00:16:06

Abstract: Real-world geometry and 3D vision tasks are replete with challenging symmetries that defy tractable analytical expression. In this paper, we introduce Neural Isometries, an autoencoder framework which learns to map the observation space to a general-purpose latent space wherein encodings are related by isometries whenever their corresponding observations are geometrically related in world space. Specifically, we regularize the latent space such that maps between encodings preserve a learned inner product and commute with a learned functional operator, in the same manner as rigid-body transformations commute with the Laplacian. This approach forms an effective backbone for self-supervised representation learning, and we demonstrate that a simple off-the-shelf equivariant network operating in the pre-trained latent space can achieve results on par with meticulously-engineered, handcrafted networks designed to handle complex, nonlinear symmetries. Furthermore, isometric maps capture information about the respective transformations in world space, and we show that this allows us to regress camera poses directly from the coefficients of the maps between encodings of adjacent views of a scene.

Main paper: 
 related work: 
 Geometric Deep Learning. Geometric deep learning is generally concerned with hand-crafting architectures that are equivariant to (i.e. that commute with) known transformations acting on the data [1] [2] [3] [4] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] . To this end, many successful architectures exploit group representations by using established mappings, such as the Fourier or spherical harmonic transforms, to map features onto domains where the group actions manifest equivalently as linear transformations [1] [2] [3] 7] . In most cases, the representations considered are finite-dimensional and irreducible which, loosely speaking, means that the group action in observation space can be expressed exactly by frequency-preserving block-diagonal matrices acting on the transform coefficients. While finite-dimensional irreducible representations (IRs) are attractive building blocks for equivariance due to their computationally exploitable structure, they often don't exist for non-compact groups, precluding generalizations to most non-linear symmetries, let alone those ill-modeled by groups. We instead avoid a heuristic choice of symmetry model and seek an approach that enables robustness to arbitrary transformations that may not even be group-structured.Functional Maps. Essential to our approach is the parameterization of transformations between observations in the latent space not as group representations, but instead as functional maps. Introduced in the seminal work of Ovsjanikov et al. [18] , functional maps (FMs) provide a powerful medium for interpreting, manipulating, and constructing otherwise intractable mappings between 3D shapes via their realization as linear transformations on spaces of functions on meshes, forming the basis for state-of-the-art pipelines in shape matching and correspondence [19] [20] [21] [22] [23] . Beyond 3D shapes, FMs can be seen as a tool to parameterize transformations between observations viewed as functions, in the sense that images, for instance, are functions that map 2D pixel coordinates to RGB colors. Integral to their study and implementation is the Laplace-Beltrami operator (the generalization of the Laplace operator to function spaces on manifolds), and in particular the expression of FMs in its eigenbasis which can expose specific geometric properties of deformations. In particular, isometries (distance-preserving transformations) manifest as highly-structured matrices not unlike IRs, being orthogonal and commuting with the diagonal matrix of eigenvalues and are thus approximately block-diagonal. That said, FMs are recovered through regularized linear solves and as such lack the consistency inherent in the analytical expressions of IRs for compact groups. However, freed from nagging theoretical constraints, FMs have displayed remarkable representational capacity, well modeling a variety of highly-complex non-rigid deformations [24] including those without group structure, such as partial [25] and inter-genus correspondence [26] . Please see [27] for an outstanding introduction to functional maps.Discovering Latent Symmetries. A number of recent approaches have proposed autoencoder frameworks wherein given symmetries in the base space manifest as simple operations in the latent space [28] [29] [30] [31] [32] [33] [34] [35] . Perhaps most similar to our approach is recent work on a Neural Fourier Transform (NFT) that has sought to manifest group actions in observation space as IRs in the the latent space [36, 37] . These methods offer impressive theoretical guarantees under the conditions that the observed transformations are either known or are a group action, although these assumptions may not hold for real-world data with complex symmetries. In contrast, our method is wholly unsupervised, and assumes no knowledge of the transformations in observation space nor that they even form a group.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Limitations are discussed in the final section of the paper (6) . Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. Justification: All experiments were performed on the same device, the details of which are described in the supplement.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
method overview: 
 Neural Isometries are an architecture-agnostic autoencoder framework which learns to map pairs of observations related in world space to latents which are related by an approximately isometric FM τsee Fig. 2 . We first formulate encoding and decoding, and define a FM in the latent space. Next, we show how a functional operator Ω and mass matrix M can be learned in the latent space to regularize τ by requiring it to be an isometry. Specifically, for observations sharing some potentially unknown relationship in world space, we enforce there exist a FM τ between their encodings satisfying two key properties: 1) That τ preserves the functional inner product in the latent space determined by M; and 2). τ commutes with the functional operator Ω. Subsequently, we show that such maps can be recovered analytically through a differentiable, closed-form least squares solve in the operator eigenbasis. Last, we formulate NIso as an optimization problem incorporating both the strictness of the isometric correspondence between latents and the eigenvalue multiplicity of the operator, the latter of which controls the structure of the maps.In experiments, we demonstrate that NIso are capable of both discovering approximations of known operators and constructing latent spaces where complex, non-linear symmetries in the observation space manifest equivalently as isometries. We show how the latter property can be exploited by demonstrating that a simple vector neuron MLP [38] acting in our pre-trained latent space can achieve results on par with state-of-the-art handcrafted equivariant networks operating in observation space. Subsequently, we consider the task of pose estimation, and demonstrate that robust SE(3) camera poses can be extracted from latent transformations, serving as evidence that NIso encourages models to encode information about transformations in world space. 
neural isometries: 
 We consider an observation space O ⊂ L 2 (M, R n ) consisting of functions defined over some domain M (e.g. with M the plane and n = 3 for RGB images). Here, elements of O are in fact captures from some world space W with σ : W → O representing the mechanism from which O is formed from W . Furthermore, we assume there is a potentially unknown collection of phenomena {T } acting on the world space that relates observations. That is, for some w ∈ W , ψ = σ(w) ∈ O, and denoting T ψ ≡ σ(T w), we assume that T ψ is also in O and that we are able to associate it with ψ.Additionally, we consider an autoencoder consisting of an encoder and decoderE : L 2 (M, R n ) → L 2 (N, R d ) and D : L 2 (N, R d ) → L 2 (M, R n ) (1)mapping between observation space and a space of latent functions over some domain Goal: Equivariance of Latent Functions. Our aim is to train the autoencoder such that for any T acting in the world space and corresponding observations ψ, T ψ ∈ O, there exists a linear map τ :EQUATIONIn other words, we desire our latent space to be equivariant under world space transformations. As our problem is discrete, τ is a functional map -an |N | × |N | matrix representation of maps on L 2 (N, R d ). In the case of latents with |N | = H × W pixels, τ is a matrix whose rows express each pixel in E(T ψ) as a linear combination of pixels in E(ψ), similar to the weight matrix one might obtain from a cross-attention operation.We will find τ by solving a least-squares problem of the form τ = min π ∥E(T ψ) -πE(ψ)∥. Unfortunately, as we will show in experiments, a direct solve without additional regularization leads to uninformative maps that capture little information about the actual world-space transformations T . To add structure, we might ask that τ be orthogonal with τ ⊤ τ = I |N | , generating gradients promoting latent codes having the property ∥E(ψ)∥≈∥E(T ψ)∥.However, we can obtain more structure yet. We propose to learn a representation of the latent geometry by jointly regressing a diagonal mass matrix M and positive semi-definite (PSD) operator Ω ∈ R |N |×|N | such that τ manifests as an isometry. That is, τ preserves the functional inner product defined by M -⟨f,EQUATIONTogether, the conditions in Equation (3) form a strong regularizer, the effects of which are best seen in the expression of τ in the eigenspectrum of Ω. As Ω is a PSD matrix with respect to the inner product defined by M, it can be expressed in terms of its spectral decomposition asEQUATIONwhere Λ = diag({λ i } 1≤i≤|N | ) is the diagonal matrix of (non-negative) eigenvalues and Φ ∈ R |N |×|N | is the matrix whose columns are the M-orthogonal eigenfunctions of Ω. DenotingEQUATIONas the projection of τ into the eigenbasis, it can be shown that the conditions in Eq. (3) reduce to τ Ω being orthogonal and Λ-commutative [27] . This is equivalent to asking that τ Ω be both sparse and condensed in that it forms an orthogonal, block-diagonal matrix, with the size of each block determined by the multiplicity of the eigenvalues in Λ.Fig. 2 visualizes Neural Isometries's training loop. First, τ is estimated between pairs of encoded T -related observations such that it approximately satisfies the conditions for an Ω-isometry as in Eq. (3). Second, the weights of the autoencoder, M, and Ω are jointly updated with respect to a combined loss term, promoting: a) latent equivariance as in Eq. ( 2), b) the ability of the decoder to reconstruct observations, and c) distinct eigenvalues Λ which encourage a diagonal-as-possible τ Ω .Recovering τ Between Latents. Instead of estimating τ directly, we equivalently estimate τ Ω in the eigenbasis of Ω, motivated by the corresponding simplification of the conditions in Eq. (3) . LetE Ω ≡ Φ ⊤ M • E (6)be the map given by encoding followed by projection into the eigenbasis of Ω. Then, given observations ψ and T ψ, we define τ Ω to be the solution to the least squares problemEQUATION)While Eq. ( 7) has an exact analytical solution [39] , we instead approximate τ Ω with a fuzzy analogue which we find better facilitates backwards gradient flow to the parameters of Ω.Specifically, letting κ : R |N |×|N | → O(|N |) denote the Procrustes projection to the nearest orthogonal matrix (e.g. through the SVD), we recover τ Ω via the approximationEQUATIONwhere[P Λ ] ij = exp(-|λ i -λ j |) is a smooth multiplicity mask over the eigenvalues Λ applied element-wise. See the supplement for details.To facilitate the recovery of τ Ω , we parameterize M directly by its diagonal elements and Ω in terms of its spectral decomposition, learning a M-orthogonal matrix of eigenfunctions Φ and nonnegative eigenvalues Λ. This has the added benefit of enabling a low-rank approximation of Ω by parameterizing only the first k eigenvalues and eigenfunctions, i.e. Φ ∈ R |N |×k and Λ = diag({λ i } 1≤i≤k ) with k ≤ |N |, mirroring similar approaches in SoTA FM pipelines [19, 40, 41] . This reduces the complexity of the orthogonal projection in Eq. (8) from |N | × |N | to k × k.Optimization. During training, the autoencoder is given pairs of T -related observations (ψ, T ψ), which are mapped to the latent space and τ Ω is estimated as in Eq. ( 8) giving τ = Φ τ Ω Φ ⊤ M. First, an equivariance loss is formed between the eigenspace projections of the encodings,EQUATIONWe note that for full rank Ω, this loss is equivalent to measuring the degree to which the equivariance condition in Eq. ( 2) holds due to the orthogonality of Φ. Next we compute a reconstruction loss,EQUATIONwith τ -1 = Φτ Ω ⊤ Φ ⊤ M, forcing the decoder to map the transformed latents to the corresponding T -related observations. Last, we formulate a multiplicity loss which promotes distinct eigenvalues and ensures Ω is "interesting" by preventing it from regressing to the identity. We observe that the eigenvalue mask P Λ can be viewed as a graph in which the number of connected components (i.e. the number of distinct eigenvalues) is equivalent to the dimension of the nullspace of the graph Laplacian ∆ PΛ formed from the mask [42] . As a measure of the nullspace dimension, we use the norm of the eigenvalues of ∆ PΛ , given by L M = ∥∆ PΛ ∥. The NFT [36, 37] takes a similar approach, wherein a diagonalization loss is imposed on estimated transformations themselves, though our experiments show it be far less effective in enforcing structure. The total loss is the sum of aforementioned termsEQUATIONwith α, β ≥ 0 weighting the contributions of the equivariance and multiplicity losses.In experiments, we also consider a similar triplet regime as proposed in [36, 37] , where the autoencoder is given triples of T -related observations (ψ, T ψ, T 2 ψ) (assuming T is composable) and the estimated map τ between the encodings of (ψ, T ψ) is used to form equivariance and reconstruction losses between (T ψ, T 2 ψ) and vice-versa. This works to prevent τ from "cheating" by encoding In particular, the estimated spherical τ Ω manifest exactly the same structure as the ground truth Wigner-D matrices corresponding to the rotation, with square blocks of size (2ℓ + 1) × (2ℓ + 1) for the ℓ-th distinct eigenvalue. Please zoom in to view structural details.privileged information about the relationship between pairs beyond T , a property we show to be critical in enforcing a useful notion of latent equivariance. However, triples of T -related observations are rare in practical settings, and we show in experiments that a major benefit of our isometric regularization is that our multiplicity loss (promoting a diagonal-as-possible and thus sparse and condensed τ Ω ) can serve as an effective substitute for access to triples.Spectral Dropout. While the composite loss in Equation ( 11) promotes latent equivariance and distinct eigenvalues, it does not, however, promote a physically meaningful ordering of the eigenvalues i.e. that small eigenvalues correspond to smooth, low-frequency eigenfunctions and larger eigenvalues to eigenfunctions with sharper, high-frequency details. Though such an ordering amounts to a permutation in the spectral dimension and is not a necessary condition for the existence of diagonalized isometric maps, it could be potentially useful in downstream applications where a classical notion of eigenvalues as frequencies is desired.To this end we propose an optional spectral dropout layer applied after computing the equivariance loss L E and before the basis unprojection. The layer is implemented as follows: During training, there is a 50% chance that dropout will be applied to given example in a batch. Then, for each such example, a spectral index 1 < i ≤ k is randomly chosen and all features with index j ≥ i are masked out. Intuitively, this forces the decoder to produce the best possible reconstructions with the lowest frequency eigenfunctions, which appear most often and thus must capture large scale features, while reserving higher frequencies for filling in fine details.We demonstrate that NIso are able to learn a compact representation of isometries that reflect the dynamics of transformations in world space. To do so, we perform two experiments in which we consider pairs of observations formed by 16 × 16 images from ImageNet [43] viewed functions on toric and spherical grids, and respectively transformed by random circular shifts and SO(3) rotations to form pairs. Thus pairs are related by the isometries of the torus and sphere which commute with the Laplacian on each domain. Taking the encoder and decoder to be the identity map (making the equivariance and reconstruction losses equivalent) we optimize for M, Ω ∈ R 256×256 via the pairwise training procedure described in sec. 4.2, including the use of spectral dropout. For the toric experiments, we learn a full-rank parameterization of Ω with k = 256; for the sphere, we learn a low-rank approximation with k = 64. As seen in Fig 3 , NIso regresses operators with significant structural similarities to the Laplacian matrices formed by the standard 3 × 3 stencil on the torus and the low rank approximation using the first 64 spherical harmonics. In both cases, NIso recovers an eigenspace that diagonalizes τ Ω between shifted images, with eigenfunctions ordered by their energy.As our approach is data-driven and our estimated maps are only approximately isometric, our learned operator and its eigenspectrum do not perfectly correspond to the ground-truth Laplacian. Instead, we are able to characterize similar, non-trivial spatial relationships that are preserved under shifts.Viewed in terms of representation learning, NIso can be seen as a recipe for the self-supervised pre-training of a network backbone E satisfying the equivariance condition in Eq. ( 2) such that transformations T in the world space manifest as isometries τ in the sense of Eq. ( 3).Exploiting Equivariance in Latent Space As we demonstrate in experiments, a simple off-theshelf isometry-equivariant head can be appended to the pre-trained backbone and fine-tuned to achieve competitive results in tasks with challenging symmetries. We employ a simple strategy wherein a low rank k approximation of Ω is learned during the pre-training stage. Thus, the eigenspace projections of the encodings of T -related observations are k × d tensors that are nearly equivalent up to an orthogonal transformation τ Ω . As such, we pass the projected encodings to a head consisting of an O(k)-equivariant vector neuron (VN) MLP [38] . We note that for large-scale tasks NIso is potentially well-suited to pair with DiffusionNet [44] , which can make use of the learned eigenbasis to perform accelerated operations in the latent space, though we do not consider this regime here.Pose Extraction from Latent Isometries. We propose that the recovered functional maps τ encode information about world-space transformations T . To test this, we consider a simple pose estimation paradigm consisting of a pre-training phase and fine-tuning phase. In the first phase, a NIso autoencoder is trained using T -related pairs of observations consisting of adjacent frames from video sequences. Subsequently, the decoder is discarded and the same pairs of observations are considered during fine-tuning. In the second phase, isometries τ Ω are estimated between the eigenspace projections of the encoded observations, vectorized, and passed directly to an MLP which predicts the parameters of the SE(3) transformation corresponding to the relative camera motion in world space between the adjacent frames. In our experiments, the weights of the NIso backbone are frozen during fine-tuning to better evaluate the information about world space transformations encoded during the unsupervised pre-training phase. At evaluation, trajectories are recovered by composing estimated frame-to-frame poses over the length of the sequence. 
experiments: 
 In this section, we provide empirical evidence through experiments that NIso 1) recovers a generalpurpose latent space that can be exploited by isometry-equivariant networks to handle challenging symmetries (5.1, 5.2); and 2) NIso encodes information about transformations in world space through the construction of isometric maps in the latent space from which geometric quantities such as camera poses can be directly regressed (5.3). Here we pre-train NIso without spectral dropout, as the ordering of the learned spectrum is irrelevant in our target applications and we find it slightly decreases accuracy of the prediction head. We provide reproducibility details in the supplement in addition to experiments quantitatively evaluating the degree of learned equivariance. We note that a consistent theme in our experiments are comparisons against the unsupervised variant of the NFT [37] (the semisupervised variants cannot be applied because the symmetries we consider have no finite-dimensional IRs). Like our approach, the NFT seeks to relate latents via linear transformations though, it differs fundamentally in that maps are guaranteed additional structure only if the world space transformations are a compact group action. While not originally proposed by the authors, we evaluate it in place of our approach in the same self-supervised representation learning regimes discussed in sec. 4.4. Thus the role of these comparisons is to show that our proposed isometric regularization better and more consistently provides a tractable and informative latent space.In our first set of experiments, we consider classification on the homNIST dataset [6] consisting of homography-perturbed MNIST digits. Following the procedure outlined in sec. 4.4, the classification network consists of a pre-trained NIso encoder backbone followed by a VN-MLP. Specifically, pre-training is performed by randomly sampling homographies from the distribution proposed in [6] which are applied to the elements of the standard MNIST training set to create pairs of observations. Here the weights of the encoder backbone are frozen and the equivariant head is trained only on the original (unperturbed) MNIST training set and evaluated on the perturbed test set. Thus the aim of these experiments is to directly quantify the degree to which pre-trained latent space is both equivariant and distinguishable. With this in mind, we perform three ablations. In the first, we train the NIso autoencoder in a triplet regime (4.2) made possible by the synthetic parameterization of T as homographies. In the second and third, we train the autoencoder in the standard pairwise regime without considering the equivariance loss L E and multiplicity loss L M , respectively. Additionally we compare the efficacy of our approach versus an NFT backbone pre-trained in the same manner. Last, we pre-train and evaluate a baseline backbone which considers only a reconstruction loss without respect to T -related pairs, but trains with data-augmentation during the fine-tuning phase by applying randomly sampled homographies.Results are shown in Tab. 1, averaged over five randomly initialized pre-training and fine-tuning runs with standard errors. Also included are those reported by homConv [6] and LieDecomp [45] , top-performing homography-equivariant networks which serve as a handcrafted baseline. While NIso pre-trained with the triplet regime produces results on par with the handcrafted baselines, pre-training with the pairwise regime-which reflects a real-world scenario-achieves a classification accuracy above 90%, significantly better than all but the three aforementioned approaches. Critically, performance drops when the multiplicity loss L M is omitted, which corresponds to a regime where τ must only preserve the inner product and Ω converges to a multiple of the identity operator. This suggests that sparsifying τ Ω by filtering it through a low-multiplicity eigenspace (i.e. enforcing that it is an "interesting" isometry) is fundamental in forcing the network to disentangle the structure of the observed transformations from the content of the observations themselves. In the same vein, the equivariance loss L M is also clearly instrumental, as the reconstruction loss alone does not explicitly enforce that E(ψ) is in fact mapped to E(T ψ) under the estimated τ . Furthermore, while the authors report that latent maps tend to converge to orthogonal maps for compact group actions in world space [36, 37] , both NFT regimes preform poorly, implying that the learned maps do not replicate the properties of finite-dimensional IRs when the group is non-compact.Acc.NIso 90.26 (± 1.27) NFT [37] 83.24 (± 2.03) AE w/ aug. 69.36 (± 2.81) MC [7] 86.5 Next, we apply NIso to classify conformally-related 3D shapes from the augmented SHREC '11 dataset [7, 46] . We follow [7] by mapping each mesh to the sphere and subsequently rasterizing to a grid. During pre-training, T -related pairs are selected from the sets of conformallyaugmented meshes derived from the same base shape in the train split.In the fine-tuning phase, the encoder weights are unfrozen and are jointly optimized with the equivariant head, representing the practical implementation of our proposed approach for equivariant tasks.Results are shown in Tab. 2, averaged over five randomly initialized pre-training and fine-tuning runs with standard errors. NIso outperforms the NFT and the autoencoder baseline (with random Möbius transformations applied during the fine-tuning phase) in addition to Möbius Convolutions (MC) [7] , a SoTA handcrafted spherical network equivariant to Möbius transformations. We consider this dataset to present a significant challenge as shape classes are roughly conformally-related and thus the maps between their spherical parameterizations are only approximated by Möbius transformations. NIso 0.023 (± 0.001) 0.031 (± 0.001) 0.044 (± 0.001) 0.057 (± 0.002) 0.071 (± 0.002) 0.081 (± 0.002) w/o Ω 0.027 (± 0.001) 0.045 (± 0.001) 0.068 (± 0.000) 0.084 (± 0.000) 0.098 (± 0.000) 0.110 (± 0.000)NFT [37] 0.022 (± 0.003) 0.035 (± 0.002) 0.059 (± 0.001) 0.077 (± 0.001) 0.091 (± 0.001) 0.104 (± 0.001) DINOv2 [47] 0.027 (± 0.003) 0.045 (± 0.002) 0.068 (± 0.001) 0.084 (± 0.001) 0.097 (± 0.001) 0.110 (± 0.001) BeIT [48] 0.028 (± 0.003) 0.046 (± 0.002) 0.068 (± 0.001) 0.085 (± 0.001) 0.098 (± 0.001) 0.110 (± 0.001) ViT 0.020 (± 0.001) 0.042 (± 0.001) 0.066 (± 0.000) 0.083 (± 0.000) 0.097 (± 0.000) 0.109 (± 0.000) This makes the performance of NIso particularly notable as it suggests that our framework has the potential to offer a more flexible and effective alternative than specialized, handcrafted equivariant networks whose underlying group-based architectures, though elegant, can only approximate inexact symmetries. Furthermore, while the NFT is relatively more competitive with the encoder backbone unfrozen, its performance and that of the baseline indicate that a lack of existing equivariant structure cannot easily be overcome in the fine-tuning phase.Last, we apply NIso to the task of camera pose estimation from real-world video on the CO3Dv2 dataset [49] following the procedure described in sec. 4.4. Due to the varying quality of the groundtruth trajectories in the dataset, we create train and test sets from the top 25% of sequences in the dataset as ranked by the provided pose quality scores. In particular, we are interested in the general ability of our method to encode information about both small and large scale world space transformations. Thus we train and evaluate in a variable baseline regime, randomly skipping between 0 and 9 frames between pairs during the pre-training and fine-tuning phases. Evaluation is performed by computing the mean absolute trajectory error (ATE) between the ground truth trajectories and those recovered by our method over the sequences in the evaluation set. To understand the efficacy of our approach at different scales, we report the mean ATE over six splits consisting of the same sequences in the evaluation set with frame skips of 0, 1, 3, 5, 7, and 9.We compare against the NFT using the same pre-training and fine-tuning procedure along with a transformer baseline. The latter is inspired by the recent success of such models in 3D vision tasks [50, 51] , and uses the same ViT-based architecture proposed in DUSt3R [51] , with the decoder modified to directly predict the parameters of the relative camera pose. As the weights of the NIso and NFT backbones are frozen during the fine-tuning phase, the transformer baseline is trained from scratch in the fine-turning phase to more directly compare the descriptiveness of the latent representations relative to the information contained in the raw observations. We also compare against two strong representation learning baselines. We extract features from both images using two pre-trained state-of-the-art vision foundation models -DINOv2 [47] and BeIT [48] -and pass the tokens into the modified DUSt3R-style decoder which is trained to predict the pose.We additionally train and evaluate an ablative version of NIso which does not learn an operator.Here τ is predicted directly between encodings and is required only to be orthogonal with the equivariance and reconstruction losses alone enforced during training. We note that computing a block-diagonalization loss on τ directly lacks justification as it would promote maps that preserve contiguous chunks of spatial indices in the latent tensors which are ordered arbitrarily. Thus, this regime serves to evaluate the degree to which the regularization through learned Ω-commutativity forces the latent transformations to reflect geometric relationships in world space.Results are shown in Tab. 3, averaged over five randomly initialized pre-training and fine-tuning runs with standard errors. All methods perform similarly with 0 frame skip but diverge afterwards, with NIso achieving significantly lower ATE values as the skip length increases. Notably, the ablative version of our method is consistently among the worst performers, suggesting that isometric regularization is also critical to encode information about world space transformations. Overall the NFT achieves the second best performance, slightly outperforming NIso at 0 frame skip but diverging thereafter. However, the latent maps it recovers are neither sparse nor exhibit condensed structure (Tab. 3, center left), and we hypothesize that its inability to effectively regularize maps beyond linearity makes it difficult for the network to discover a consistent transformation model that applies across scales. This could cause the network to focus on a specific regime at the expense of others, which may explain its relatively strong performance at 0 frame skip. The transformer baseline and representation learners are also highly flexible, and an analogous line of reasoning could explain their similar error profile. 
discussion: 
 Limitations. A key factor limiting the broader applicability of our approach to geometry processing and graph-based tasks is an inability to learn and transfer an operator between domains with varying connectivity. In addition, NIso does not explicitly handle partiality or occlusion between input pairs which are ubiquitous in real-world data and likely degrade its performance in the pose estimation task. We seek to address these limitations in future work.In this paper we introduce Neural Isometries, a method which converts challenging observed symmetries into isometries in the latent space. Our approach forms an effective backbone for self-supervised representation learning, enabling simple off-the-shelf equivariant networks to achieve strong results in tasks with complex, non-linear symmetries. Furthermore, isometric regularization produces latent representations that are geometrically informative by encoding information about transformations in world space, and we demonstrate that robust camera poses can be extracted from the isometric maps between latents in a general baseline setting.Here we show that isometries manifest as orthogonal matrices that commute with the diagonal matrix of eigenvalues in the operator eigenbasis. Suppose we are given a PSD operator Ω and diagonal mass matrix M, with the former expressed in terms of its eigendecomposition as in Equation ( 4) for eigenfunctions Φ and eigenvalues Λ.Let τ be an isometric functional map satisfying the conditions in Equation ( 3) with τ Ω its projection into the eigenbasis as in Equation (5) . Noting that the conditionΦ ⊤ MΦ = I |N | implies that Φ -1 = Φ ⊤ M and Φ -⊤ = MΦ, it follows that τ Ω ⊤ τ Ω = [Φ ⊤ τ ⊤ M Φ]Φ ⊤ M τ Φ = Φ ⊤ τ ⊤ M τ Φ = Φ ⊤ MΦ = I |N |, and thus τ Ω is orthogonal. Furthermore, it follows thatτ Ω Λ = Φ ⊤ M τ ΦΛ = Φ ⊤ M τ ΦΛΦ ⊤ MΦ = Φ ⊤ M τ ΩΦ = Φ ⊤ MΩ τ Φ = ΛΦ ⊤ M τ Φ = Λτ Ω , so τ Ω Λ = Λτ Ω .Here we derive the approximate solution to the least squares problem in Equations (7 -8) . Specifically, let Λ = diag({λ i } 1≤i≤k ) ∈ R k×k be a diagonal matrix of non-negative eigenvalues with Λ ii ≤ Λ i+1,i+1 . Furthermore, suppose that there are q distinct eigenvalues { λ i } 1≤i≤q , λ i ≤ λ i+1 each with multiplicity m i . Thus, Λ can be expressed as the direct sumEQUATIONwith I mi denoting the m i × m i identity matrix. It follows that any k × k matrix π satisfying πΛ = Λπ must be of the formEQUATION)That is, π must be a block diagonal matrix with the size of each block determined by the multiplicity of the eigenvalues of Λ.EQUATIONconsisting of finding an orthogonal, Λ-commuting map minimizing the distance between A and B under the Frobenius norm. WritingEQUATIONthe solution to the minimization problem in Equation ( 14) is equivalently expressed as the direct sum of the solutions to the orthogonal sub-problems corresponding to each block [39] withEQUATIONNow, for arbitrary n let κ : R n×n → O(n) denote the Procrustes projection to the nearest orthogonal matrix such that for any M ∈ R n×n with SVD M = U ΣV ⊤ , κ(M ) ≡ U V ⊤ . Thus, the solution to the minimization problem is given byEQUATIONwhich, following from the properties of the SVD, can be expressed equivalently via a single k × k Procrustes projection such thatEQUATION)Here we observe that by defining P Λ ∈ R k×k to be the eigenvalue mask given byEQUATIONwe haveEQUATIONand thusEQUATIONIn practice we substitute P Λ in Equation ( 19) with the fuzzy analogue[P Λ ] ij = exp(-|λ i -λ j |),which we find better facilitates the flow of backwards gradients to the parameters of Ω. Replacing A and B with E Ω (ψ) and E Ω (T ψ) we arrive at the approximate solution for τ Ω in Equation (8) .Given the k × k eigenvalue mask P Λ with [P Λ ] ij = exp(-|λ i -λ j |), we form its graph Laplacian asEQUATIONwith 1 ∈ R k the vector of ones.Code and experiments are available at https://github.com/vsitzmann/neural-isometries.All experiments were performed on a single NVIDIA A6000 GPU with 48 GB of memory.In the toric/spherical experiments, the model consists of a single NIso layer. which learns a fullrank/low-rank operator and mass matrix, with the former parameterized by an orthogonal matrix of eigenvectors Φ ∈ R 256×256 /Φ ∈ R 256×64 and diagonal matrix of non-negative eigenvalues Λ.During training, input features are created by stacking 86 16 × 16 × 3 examples from ImageNet into a single 16 × 16 × 258 tensor representing an observation ψ. In the toric experiments, random circular shifts about both spatial axes are applied to form T ψ. In the spherical experiments, the images are assumed to lie on a 16 × 16 Driscoll-Healy spherical grid and random rotations in SO(3) are sampled and applied to form T ψ. As the encoder and decoders are the identity maps, the equivariance and reconstruction losses are equivalent, and the objective in Equation ( 11) is minimized with α = 0 and β = 0.1. The models is trained with spectral dropout for 100,000 iterations with a batch size of 1 using the AdamW optimizer [53] with a weight decay of 10 -4 . The learning rate follows a schedule consisting of a 2,000 step warm up from 0.0 to 5 × 10 -4 and afterwards decays to 5 × 10 -5 via cosine annealing. Training takes approximately 6 hours.Architecture. The encoder and decoder are mirrored 2-level ConvNets. Specifically, each level consists of three convolutional ResNet blocks with 128 and 256 channels at the finest and coarsest resolutions respectively. A mean pool (nearest neighbor unpool) layer halving (doubling) the spatial resolution bridges the two levels.The latent space has 32 dimensions. A rank k = 32 operator Ω and mass matrix M are learned, with the former parameterized by 32 eigenfunctions and non-negative eigenvalues.The equivariant classification head consists of a 2-layer VN-MLP [38] with 128 channels, followed by an output layer which computes invariant features from the inner products between vectors which are passed to a standard linear layer to output class predictions.Pre-Training. Following [6] , MNIST digits are padded to 40 × 40, and pairs are produced by warping digits with respect to homographies T sampled from the proposed distribution. In the triplet regimes, tuples are created by applying T 2 in addition to T . Both the pairwise and triplet regimes are trained with respect to the composite loss with α = 0.5 and β = 0.1. The two ablative regimes are trained with α = 0 and β = 0, respectively. In the triplet regime, denoting τ and σ to be the maps estimated between (E(ψ), E(T ψ)) and (E(T ψ), E(T 2 ψ)) respectively, the equivariance and reconstruction losses are formulated asEQUATIONandEQUATION)The autoencoders are trained without spectral dropout for 50, 000 steps with a batch size of 16 using the AdamW optimizer with a weight decay of 10 -4 . The learning rate follows a schedule consisting of a 2,000 step warm up from 0.0 to 5 × 10 -4 and afterwards decays to 5 × 10 -5 via cosine annealing.Training takes approximately 30 minutes.Fine-Tuning. During the fine tuning phase, the decoder is discarded, and the weights of the encoder, operator Ω, and mass M are frozen. Examples ψ are encoded, and their eigenspace projections E Ω are passed to the classification head to form class predictions under a standard softmax cross-entropy loss. Training is performed for 10,000 iterations with a batch size of 16 using the AdamW optimizer the same weight decay and learning rate schedule as used previously. Training takes approximately one minute. Evaluation is performed on the fixed test set proposed in [6] .Baselines. We implement the NFT following the procedure described in the paper [37] . While a ViT-based encoder is proposed in the original paper, we find that it produces worse evaluation results than the 2-layer ConvNet encoder described above and thus use the latter in our evaluations to provide a fair comparison. We follow the authors' proposed implementation and consider only a reconstruction loss and learn a diagonalization transform in a post-processing step. Otherwise, the NFT is pre-trained, fine-tuned, and evaluated identically to our own approach, and uses the same equivariant classification head. We note that the dimension of the latent map constructed by the NFT is the same as τ Ω (32 × 32) and that the tensor passed to the classification head is the same size for both our method and the NFT.The autoencoder baseline consists of the same encoder and decoder, pre-trained identically but with respect to the standard reconstruction loss ∥D(E(ψ)) -ψ∥ without considering T -related pairs. During the fine-tuning phase, we apply data augmentation to the input images by transforming them by homographies sampled from the distribution proposed in [6] before encoding. Latents are passed directly to a standard non-equivariant 2-layer MLP with 128 channels to form class predictions. Otherwise, fine-tuning and evaluation are performed in the same manner.Dataset. Experiments are performed using the conformally extended SHREC '11 shape classification dataset proposed in [7] , with 30 random conformal transformations are applied to each shape to extend the original dataset [46] . The augmented dataset contains 25 distinct shape classes. Each shape is mapped to the sphere using the method of [54] . Input features are taken to be the values of the heat kernel signature [55] computed on the original mesh at 16 timescales logarithmically distributed in the range [-2, 0] and rasterized to 96 × 192 spherical grids. Pairs of T -related observations are formed by randomly selecting two conformal augmentations of the same shape. For each set of training and fine-tuning runs, train and evaluation splits are randomly generated by selecting respectively 10 and 4 of the 20 sets of conformally augmented shapes per class.Architecture. The encoder and decoder are mirrored 4-level ConvNets, with two ResNet blocks per level and 32, 64, 128, 256 channels per layer from finest to coarsest resolution. Mean pooling and nearest neighbor upsampling are used to halve and double the resolution between layers.The latent space has 64 dimensions. A rank k = 64 operator Ω and mass matrix M are learned, with the former parameterized by 64 eigenfunctions and non-negative eigenvalues.The equivariant classification head consists of a 4-layer VN-MLP with 128 channels, using the same output layer as described in sec. (B.3) to produce class predictions.Training is performed with respect to the composite loss with α = 0.5 and β = 0.01.The model is trained without spectral dropout for 50,000 iterations with a batch size of 16 using the AdamW optimizer with a weight decay of 10 -4 . The learning rate follows a schedule consisting of a 2,000 step warm up from 0.0 to 5 × 10 -4 and afterwards decays to 5 × 10 -5 via cosine annealing.Training takes approximately four hours.Fine-Tuning. During the fine tuning phase, the decoder is discarded and the encoder, operator Ω, and mass matrix M are jointly optimized with the equivariant classification head using the standard softmax cross-entropy loss. Training is performed for 10,000 iterations with a batch size of 16 using the AdamW optimizer, with the same weight decay and learning rate schedule as used previously.Training takes approximately one hour.Baselines. We pre-train, fine-tune, and evaluate the NFT and autoencoder baseline using the same autoencoder architecture, classification head, and training regimes in the same manner as sec. (B.3).Here data augmentation is applied during the fine-tuning phase by randomly sampling and applying Möbius transformations with scale factors proportional to those estimated between the spherical parameterizations of the shapes. Again, the both the size of the latent map and size of the tensor passed to the classification are the same size for both our method and the NFT. Here, the weights of the encoder are left unfrozen, and are jointly optimized with the classification head during the fine-tuning phase.Dataset. Experiments are performed with a subsection of the CO3Dv2 dataset [49] . Given the high-degree of variability in the quality of the ground truth trajectories, we consider only the top 25% of sequences in the dataset as ranked by the provided pose quality score, slightly under 9,000 trajectories in total. An evaluation set is formed by withholding 10% of said trajectories, with the rest used for training. The list sequences in the train and evaluation sets will be made available along with the code release. As the videos in the dataset are of different resolutions, we center crop each frame to the minimum of its height and width dimensions and resize to 144 × 144.Architecture. The encoder and decoder are mirrored 3-level ConvNets, with four ResNet blocks per level and 64, 128, 256 channels per layer from finest to coarsest resolution. Mean pooling and nearest neighbor resolution are used to halve and double the resolution between layers.The latent space has 128 dimensions. A rank k = 128 operator Ω parameterized by and mass matrix M are learned, with the former parameterized by 128 eigenvectors and eigenvalues. The pose extraction network consists of a standard 5-layer MLP with 512 channels, which outputs 9-dimensional tensor representing the parameters of an SE(3) transformation (translation + two vectors used to form the rotation matrix through cross products). Fine-Tuning. During the fine-tuning phase, the weights of the encoder, operator Ω, and mass matrix M are frozen. As in the pre-training phase, pairs of observations are passed to the decoder and τ Ω ∈ R 128×128 is estimated between the eigenspace projection of the encodings. Then, τ Ω is vectorized and passed to the MLP head to predict the parameters of the camera pose.Training is performed with respect to a two term loss measuring the degree to which the predicted translation and rotation deviate from those of the ground truth pose. Denoting R Ω and R to be the predicted and ground truth poses respectively, the rotational component of the loss is given byEQUATIONThe transitional component of the loss must be scale invariant as the depth scale factor is unknown between sequences. To handle this, we consider sub-batches formed from pairs of frames all belonging to the same sequence. Denoting t Ω and t to be the matrices consisting of the sub-batched predicted and ground truth translations stacked column-wise, we form the translational component of the loss viaEQUATIONwhereEQUATIONis the scale factor that minimizes L T over the pairs of frames in the sub-batch. The total loss is given byL = L R + L TThe prediction head is trained without spectral dropout for 20,000 iterations with a batch size of 16 (consisting of 4 sub-batches of pairs of frames from 4 sequences) using the AdamW optimizer with the same weight decay and learning rate schedule as used previously. Training takes approximately one 80 minutes.Baselines. The NFT is pre-trained, fine-tuned, and evaluated using the same autoencoder architecture, prediction head, and training regime as our own method in the same manner as is done in the preceding experiments. We note that both our method and the NFT construct latent maps of the same dimension, which are passed to identical prediction heads.The transformer baseline is based on the CroCo/DUSt3R [50, 51] architectures. As with our approach, input frames are passed to the same decoder. Here the encoder consists of 12-layer VITs with a patch size of 16, with 16 heads and 512 channels per layer. Adjacent frames are independently encoded, and then concatenated along the spatial dimension in addition to a single learned token. The resulting tensor is passed to a 12-layer transformer decoder which mirrors the encoder with 16 heads and 512 channels per layer. Subsequently, the learned token is extracted from the output and passed through a linear layer to recover the predicted pose parameters.Additionally, we also evaluate the efficacy of DINOv2 [47] and BeIT [48] as representation learners for the pose estimation task. Pairs of adjacent frames are passed through the pre-trained models and the feature tokens for each image from the final layers are extracted. These are used as input to the same DUSt3R[47]-style decoder described above which is trained to predict the parameters of the relative camera pose.The VIT model and DINOv2/BeIT prediction heads are trained from scratch in the fine-tuning phase, using the same loss and optimization procedure as described above.Below we provide additional results including a quantiative evaluation of learned equivariance, visualizations of the eigenfunctions Φ and mass matrices M learned in each experiment, and comparisons of estimated camera trajectories recovered by each method in the pose prediction experiments. For the latter, examples of failure cases are also included. NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Justification: To the best of our knowledge, we conform to the Code of Ethics. At this time we do not see our method providing a straightforward avenue for abuse by bad actors.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: Due to the low-level nature of our method, we do not see it directly facilitating any negative social impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification:Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All data used is cited in accordance with the provided licenses.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.38th Conference on Neural Information Processing Systems (NeurIPS 2024). 
: 
 We constantly capture lossy observations of our world -images, for instance, are 2D projections of the 3D world. Observations captured consecutively in time are often related by transformations which are easily described in world space, but are intractable in the space of observations. For instance, video frames captured by a camera moving through a static scene are fully described by a combination of the 3D scene geometry and SE(3) camera poses. In contrast, the image space transformations between these frames can only be characterized by optical flow, a highdimensional vector field that does not itself have any easily tractable low-dimensional representation.Geometric deep learning seeks to build neural network architectures that are provably robust to transformations acting on their inputs, such as rotations [1] [2] [3] , dilations [4, 5] , and projective transformations [6, 7] . However, such approaches are only tractable for transformations that have group structure, and, even in those cases, still require meticulously handcrafted and complex architectures. Yet, many real-world transformations of interest, for instance in vision and geometry processing, altogether lack identifiable group structure, such as the effect of camera motion in image space-see Fig. 1 to the right. Even when group-structured, they are often non-linear and non-compact, such as is the case of image homographies and non-rigid shape deformations where existing approaches can be prohibitively expensive.In this paper, we take a first step towards a class of models that learns to be equivariant to unknown and difficult geometric transformations in world space. We propose Neural Isometries (NIso), an autoencoder framework which learns to map the observation space to a latent space in which encodings are related by tractable, highly-structured linear maps whenever their corresponding observations are geometrically related in world space. Specifically, observations are encoded into latents preserving their spatial dimensions. Images, for instance, can be encoded into latent functions defined over a lower resolution grid or the patch tokens of a ViT. For observations sharing some potentially unknown relationship in world space, we enforce their encodings be related by a functional map τ -a linear transformation on the space of latent functions. In particular, we require that τ is an isometry such that it preserves a learned inner product and commutes with a learned functional operator, in the sense that rigid body transformations commute with the Laplace operator in Euclidean space.Neural Isometries exhibit unique properties that make them a promising step towards an architectureagnostic regime for self-supervised equivariant representation learning. We experimentally validate two principle claims regarding the efficacy and applicability of our approach:• Neural Isometries recover a general-purpose latent space in which challenging symmetries in the observation space can be reduced to compact, tractable maps in the latent space. We show that this can be exploited by simple isometry-equivariant networks to achieve results on par with leading hand-crafted equivariant networks in tasks with complex non-linear symmetries.• The latent space constructed is geometrically informative in that it encodes information about the transformations in world space. We demonstrate that robust camera poses can be regressed directly from the isometric functional maps between encodings of adjacent views of a scene. 
 
 Appendix:  


Decision: This paper received mixed reviews in the first round, was intensively discussed between reviewers and authors, and between reviewers. After the discussion phases, the scores were 3x borderline reject and 1x weak accept.

The overall consensus is that the paper is a great read and proposes an interesting angle for obtaining equivariant representations. On the negative side, there were several concerns with respect to the experimental setup and if they support the general claims of the work. Some of them were successfully rebutted by the authors:
- the paper does indeed evaluate on a relevant real world task (CO3D camera pose estimation)
- it also successfully shows in the ablations in Section 5.1 that the proposed losses are necessary to recover the correct transformations
- the claim that the method performs on par with hand-designed methods is supported (if only on limited experiments)

However, there are also concerns left, which have not been adequately resolved:
- It is true that the paper lacks theoretical justifications for the many "intuitive" heuristics that it employs
- It is also true that the paper mostly invents its own baselines and datasets. Only the measurements of equivariance error (rebuttal document) seem to be in the same framework as previous works. Meanwhile, the NFT authors provide a benchmark set (https://github.com/masomatics/NFTPublic/tree/main), which is not used by the authors of this work.

Regarding the last point it has to be said that the presented approach is indeed quite novel and the used datasets and evaluation methods seem to be quite creative to me. In this borderline situation, I tend to recommend acceptance, favoring the explorative nature of this work.

Reviews: review: paper_summary: paper_summary: The main motivation behind Neural Isometry is as follows: most of real world transformations in vision and geometry processing lack identifiable group structure and therefore challenging for prior work in equivariant learning that assumes such knowledge apriori. The paper proposes an auto encoder framework that maps observations (related by some geometric transformations) to a latent space where embeddings are  related by a linear transform. The framework leverages the existing functional map framework and corresponding regularization to learn this latent space. The paper validates and compares their performance with the baseline NFT (Neural Fourier Transform) that assumes the knowledge of the group a priori.
strength_weakness: strength_weakness: As mentioned many times in paper, Instead of theoretical justification ( as in NFT), this paper seeks to validate the efficacy of approach experimentally on real world tasks.

- Limited Experimental Set up: The datasets used are still very small scale (MNIST,) or synthetic (SHREC) and therefore far from the real world data.  The submission mainly follows and compares with one baseline (NFT) that was implemented by authors themselves.  The submission could motivate the problem/solution by probing/playing with the learned structured space e.g. in geometry processing tasks [1]. 

- Limitation of Functional map: One of the biggest limitation of functional map framework is its applicability to real world data where partiality is ubiquitous (due to occlusion etc). Therefore, Neural Isometry by design also inherits this limitation.  experiments are shown on MNIST/SHREC where there is no occlusion or partiality. In the third benchmark, instead of skipping the odd frame, does the performance decays rapidly between distant frames?

- Presentation: The submission contains several typos, convoluted sentences (Line 85-88) and statements without a citation. e.g.

-- Line 49: < Lorentz transformations with d'almbert operator of Minkowski space> citation is missing here. Also, why do we need to know this fact in the introduction of this paper? 

-- Line 42: <by preserving the spatial dimensions> not sure what does this mean after reading the next 2 lines. Please clarify

-- Line 89: <orthogonal relaxation of FM> citation or please detail what those are.

- Related work: The submission cites a dozen variants of deep functional map paper [2] but not the deep functional map paper itself [2] that inspired them. Please justify how [20,21,22,39] are related to this work or inspired this work given other deep functional pipeline cited in paper. The reviewer would instead relate/distinguish this work with [3].


1. Composite Shape Modeling via Latent Space Factorization, Dubrovina et al, 2021
2. Deep Functional Maps Litany et al. 2017
3. Map-based exploration of intrinsic shape differences and variability  Rustamov et al. 2013
questions: questions: please see above.
limitations: limitations: The submission should state the scalability of this approach given its reliance on Laplacian Eigen basis.
paper_summary: paper_summary: This paper proposes a generic equivariant ML framework by learning latent representations are are modelled to be related by an isometry. There are several important design choices made by the authors: (1.) An autoencoder framework that keeps the spatial structure (i.e. images get encoded into images) (2.) The isometric map is represented compactly with a basis that is the outcome of an eigen- -decomposition of a PSD operator (3.) The isometry manifests as an orthogonal matrix with a sparse block diagonal structure in its reduced projected form. 

Experiments are shown for (1.) Learning the spectrum of a Toric Laplacian and comparing it qualitatively to the real one  in Fig 3 (2.) Homography perturbed MNIST (3.) Conformal shape classification and (4.) Camera Pose Estimation from real-world video. The results (1) show a good proof of concept and (2), (3) and (4) demonstrate a clear superiority over the closest conceptual baseline NFT. (2.) Also makes the case for achieving comparable performance with simpler equivariant architectures using the NIso latent space.
strength_weakness: strength_weakness: - I am actually quite confused on how the operator \Omega and Mass matrix M is learned? The Laplace Beltrami is a very structured object. See all requirements enumerated in Wardetzky, Max, et al. "Discrete Laplace operators: no free lunch." Symposium on Geometry processing. Vol. 33. 2007.). It is unclear what the nature of this operator is and its learning feels ad-hoc and unmotivated. 
- Do the authors make sure that the optimization for the functional map is well-posed? Typically in the original formulation a "good" functional map is the outcome of using many-many descriptors in addition to regularization constraints like commutativity with laplacian. If I understand correctly the proposed formulation seems to be confident in learning a structured functional map (like Fig 2) using just one pair of corresponding functions ? 
- As a high-level opinion: Overall the experimental section could be stronger. The "discovery" of the toric laplacian was nice, but a demonstration on roto-translations perhaps with scale and/or spherical images with SO(3) would be extremely convincing. I would also recommend an evaluation that focuses beyond NFT and actually quantifying the complexity of different equivariant models. 
- As mentioned by the authors, the requirement of generalizing to domains with diverse connectivity is indeed a hard one, but important.
questions: questions: - The basis functions visualized in Fig3 (third column) are learned or from the actual toric laplacian? what are the various rows? 
- It would also be useful to report some feature of complexity in Table 1 indicating "meticulously-engineered, handcrafted networks"
- More examples of evidence showing good basis functions 
- What is a perturbation in Section 5.1?
limitations: limitations: I think the authors make a fair declaration of the limitations of their work in Section 6.
I would make a stronger vote after the rebuttal. I think this paper has many ingredients to warrant acceptance: clever idea, good argumentation, and decent proof of concept. It lacks clarity in some places and the evaluation does not appear equally general as is the description of the method. For this reason, I maintain a borderline acceptance with a full intention to re-assess in the next round.
paper_summary: paper_summary: This paper introduces Neural Isometries, which is an autoencoder framework learning to map the observation space to a general-purpose latent space wherein encodings are related by isometries whenever their corresponding observations are geometrically related in world space. Several experiments, including camera pose estimation, are conducted.
strength_weakness: strength_weakness: This is an interesting paper but in-depth discussion is lacking. My main concern is that there is no quantitative or theoretical description of the equivariance. For example, 
1.  How equivariant is the model? i.e. How large is the equivariance loss?
2. What type of equivariance in the observation space can be modeled? Will the model fail when the pertubations are too strong?

In addition, the experiment results are relatively weak. Only 3 small scale experiments are conducted, and only a few baselines are considered.
questions: questions: 1. If the observation space is already isometric, e.g. rotate by 90 degrees, what can be said about the latent sapce?
2. How robust is this model? For example, how will the latent vector change when the inputs are partially observed (masked)?
limitations: limitations: See the weakness section.
paper_summary: paper_summary: The paper proposes an autoencoder framework that encodes the input symmetries into isometries in the latent space. The equivariance in the latent space is captured by a functional map $\tau$, which is regularized to be an isometry. Instead of hard constraints, the equivariance of the system is encouraged via the optimized $\tau$ and the loss function design. The proposed method is evaluated on a homography-perturbed MNIST, conformal shape classification, and camera pose estimation tasks. The experiments show the proposed method performs on par with the handcrafted equivariant baselines and outperforms a baseline with a similar approach (Neural Fourier Transform). 

I’m positive about the paper. However, there are some main concerns that I would like to know the answers to. I’m willing to raise the scores once they are addressed.
strength_weakness: strength_weakness: 1. Although being shown to be effective in the experiments, the constraint of $\tau$ being an isometry seems a bit heuristic. The paper can benefit from some theoretical investigation on why constraining to being isometries helps the performance.
2. Following up on the previous point, can all transformations be modeled as isometric functional maps in the latent space? That being said, does restricting $\tau$ to isometries affect the types of transformations the network is able to model? 
3. Since the equivariance is only obtained softly via the loss function design, it’s essential to see how much equivariant is maintained/lost in the latent space. I recommend the authors report some measures of equivariance and compare them with the handcrafted methods. (For example, equivariant error/loss in the latent space.)
4. There are several heuristic designs in the proposed framework, and some of them affect the performance significantly. For example, it is not explained clearly why a smooth multiplicative mask is required to ensure $\tau_\Omega$ is semi-diagonal. The effect of such approximation is also not discussed/evaluated. The second is the multiplicity loss, which greatly affects the performance of the framework. 
5. I appreciate the comparisons with handcrafted equivariant networks. However, I believe the paper can be greatly strengthened by comparing it with the same autoencoder framework while not enforcing the latent equivariance and training with data augmentation. This can demonstrate the benefits of the proposed latent equivariant design.
questions: questions: 1. Since there are no hard-coded constraints (except for the regularization) in the system, the proposed method can work with different input transformations. Do you think this work can be applied to automatic symmetry discovery tasks? 
2. I understand that the inner product needs to be preserved to construct an isometry, but why does $\tau$ have to commute with $\Omega$?
3. Ln 207, “we show in experiments that a major benefit of our isometric regularization is that our multiplicity loss (promoting a diagonal-as-possible and thus sparse and compact $\tau_\Omega$) can serve as an effective substitute for access to triples.” Is this true for all three experiments, or is it only validated on homNIST?
limitations: limitations: As mentioned in the paper, the proposed network is unable to perform graph-based tasks. In addition, the equivariance in the latent space is not evaluated. As a result, it is unclear how the equivariance is preserved in the latent space. Lastly, by restricting the functional maps to be isometries, the transformations that the network can model might be limited.

Comments: title: Additional question remaining in discussion?
Rebuttal response
Address remaining questions in discussion period?
Address remaining questions in discussion?
Questions remaining?
Response (2/2): Analysis of Partiality Robustness
Response (1/2): Modeling Equivariance
comment: Dear FFcP,

Thank you again for your response and feedback! We believe we were able to address your stated remaining concerns, including a straightforward experimental evaluation of robustness.  As the end of the author / reviewer discussion period is fast approaching, we would like to be sure you are satisfied by our discuss and see if we can address any remaining questions. Please let us know whether we addressed your comments and questions appropriately!

Thank you!

Best, the authors
Thank you for your response and we are extremely glad to hear you are happy to recommend a clear acceptance!

We completely agree that NIso represents a first step towards identifying underlying symmetries and that further investigation is necessary to support claims of broad generalization. The reviewer is correct to point out that answering difficult questions like those regarding data efficiency and effectiveness in the presence of multiple, different non-unitary actions are important prerequisites. We will amend the revision to reflect this and make sure that we communicate limitations on partiality and transformations modeled clearly.

We will incorporate the spherical harmonic transform experiment into the main body of the revision as the results are very compelling (thank you again for suggesting this experiment!). In both this experiment and the toric laplacian experiment, the encoder and decoder are just the identity map. We pass the images directly to the transformation estimation module, learning only the parameterization of the operator with the goal of evaluating the abilities of the module in isolation.
Thank you for your response and for updating your score, we are glad that you find NIso will contribute to the NeurIPS community!

Following your comments, we will update our paper to make sure to not overclaim. Specifically, we will clarify that (1) NIso can only exactly model unitary transformations; and (2) That our experimental results only suggest the potential of NIso to model more complex, non-unitary transformations, and that a rigorous theoretical investigation combined with extensive empirical results is necessary to convincingly demonstrate broad generalization. We will also clarify that (3) NIso does currently explicitly handle partiality.

We agree that a deeper investigation into the multiplicity loss will strengthen our method and we are actively investigating this in follow up work. We would also like to note that an important novel benefit of our formulation is that it allows for us to regularize for block-diagonality *without backpropagating these gradients through the solve for the estimated transformation.*  Specifically, the prior methods including the NFT attempt to impose a block-diagonality loss on the estimated transformation itself. We discussed with the authors who confirmed that this is only possible in a post-processing step as it otherwise destabilizes training. In contrast, by learning to parameterize an operator $\Omega$, we instead impose the regularization for block diagonality on the mask $P_\Lambda$ which is stable during training. Our comparisons with the NFT (see Tab. 3, center left) suggest that this is a key feature enabling self-supervised discovery of sparse and condensed representations, and we are excited to investigate this further.

Thank you again for your comments, which will help to clarify the paper!
Thank you for a refreshing rebuttal. After reading through the rebuttal and other reviews, I am quite happy to raise my score to a clear acceptance. This is an interesting paper with some fascinating observations. However, there is still some way to go before this can really be generalized and put into meaningful action. Sustaining drawbacks are some very hard questions like - how much data is needed for this approach? Equivariance to multiple non-trivial (perhaps non-unitary) transformations of the input etc. But I think as a good proof of concept, I believe this paper tells a story that would be interesting at Neurips. I would highly recommend including the spherical image experiment in the main draft (also what kind of architecture did you use for the encoder?)
I thank the authors for their efforts in the rebuttal. Many of my concerns have been addressed. It's good to see NIso can achieve a similar equivariance error to the handcrafted methods. It's also interesting to see patterns similar to spherical harmonies appear in the SO(3) case. It is also great to see NIso perform better than AE with augmentation. The added experiments strengthen the paper. 

In addition, I kindly disagree with other reviewers on the need to solve occlusion/partiality. Occlusion and imperfect symmetry modeling can be a different research topic on its own in the field. However, I recommend that the authors refrain from claiming that NIso can generally handle occlusion/partiality robustly, as it requires additional proof and verification that is not presented in the paper. 

Although most of my concerns have been addressed, there are still some that remain. The multiplicity loss still seems like a heuristic trick to me, and it has a significant impact on the system's performance. I would love to see a more theoretical probe into it in future work. Secondly, although it is shown experimentally that the proposed method can deal with transformations that are not unitary, it is still not guaranteed theoretically. The authors provide a plausible explanation in response to Reviewer FFcP (there exists a domain in which these transformations are either unitary or isometric, and can be mapped to via a sufficiently expressive autoencoder). Although I personally agree with such an explanation, this is still a speculation without proper proof. I again suggest the authors refrain from making such a claim. 

Overall, the paper is strengthened after the rebuttal, and I believe the paper introduces a novel way to softly model equivariance in the latent space, which is a great addition to the community. Therefore, I have raised my score to reflect the above points.
Dear Reviewer Sfsw,

Thank you again for your comments and feedback! As the end of the author / reviewer discussion period is fast approaching, we would love to hear your thoughts and see if we can address any remaining questions in the remaining time. Please let us know whether we addressed your comments and questions appropriately!

Thank you!

Best, 
the authors
Dear JQqD,

Thank you again for your comments and feedback! As the end of the author / reviewer discussion period is fast approaching, we would love to hear your thoughts and see if we can address any remaining questions in the remaining time. Please let us know whether we addressed your comments and questions appropriately!

Thank you!

Best, 
the authors
Dear 2Npe,

Thank you again for your comments and feedback! As the end of the author / reviewer discussion period is fast approaching, we would love to hear your thoughts and see if we can address any remaining questions in the remaining time. Please let us know whether we addressed your comments and questions appropriately!

Thank you!

Best, 
the authors
## Analysis: Robustness to Partiality
In response to the reviewer’s concerns, we are happy to report that we performed a theoretically grounded experimental evaluation of robustness which we will be happy to include in the revision using the additional page of the camera-ready paper.

Specifically, we seek to evaluate the robustness of the isometry-estimation module in isolation in the presence of occlusions. We consider  the encoder-free paradigm – consisting of the projection to the learned basis, the estimation of the isometric map as in Equation (8), followed by unprojection – which is the same setup in the toric and spherical laplacian experiments. **This setup allows us to study exactly the effect of masking on the latents (as requested by the reviewer) which in this case are the input images.** To do so, we consider two observations $\psi$ and $T\psi$ which only partially correspond and denote  $O_\psi$ and $O_{T\psi}$ to be the diagonal overlap masks such that the $i$-th diagonal element of $O_{\psi}$ is 1 if $i$ is the overlap with $T\psi$ and $0$ otherwise, with $O_{T\psi}$ defined in the same manner. We observe that the equivariance error under the partiality can thus be defined by the magnitude of the difference between the components of $O_{\psi} \psi$ that get mapped to $ O_{T\psi} T\psi $ under $\tau$. That is, we define the partiality equivariance error to be $  \lVert  O_{T\psi} \tau O_{\psi} \psi - O_{T\psi} T\psi \rVert^2 / \lVert O_{T\psi} T\psi \rVert^2$.

Using the toric laplacian experiments as a base, we consider two models of partiality. In the first, we no longer consider the domain to be toric and instead shifted images are clipped at the boundaries with the resulting empty pixels masked to zero – corresponding to the type of partiality often observed in video. In the second, we randomly mask out $2 \times 2$ patches in the shifted image. We train five instances of our model under both partiality regimes, masking out approximately 10%, 20%, 30%, 40%, and 50% of pixels in each instance and measuring the resulting partial equivariance error on the test set. The results are shown in the table below.

| Percent Occluded  | 10% | 20% | 30%| 40%| 50% |
| :------------------------ | :-----: | :-----: | :-----: | :-----: | :-----: |
| Shift Mask    	|   5.66%   | 9.42% | 14.43% | 17.42% | 20.47% |
| Patch Mask      	|   6.69%  | 13.62% | 23.54% | 34.14% | 45.35%|

Notably, we see that the partial equivariance error is consistently lower in the presence of shift-based occlusions, and  increases less as the percentage of occluded pixels increases. We observe that the principal difference between these two regimes is that the unoccluded pixels exist in a contiguous block under the shift mask, whereas the region of unoccluded pixels is fragmented under the patch masking. Intuitively, the contiguous matches between large blocks (comprising the majority of the image) act as a strong regularizer that de-prioritizes matching the occluded areas. Conversely, when the correspondence is interrupted and fragmented it offers weaker regularization and the resulting map is more affected by the spurious matches induced by the occlusion.
Thank you for your response. We are glad to hear that you view our experimental results as convincing. 

## Which Types of Equivariance can be modeled?

We apologize for the lack of clarity in our previous response. In the following, we will make two points: (1) We show that NIso can exactly model equivariance for any unitary transformations, which make up a large chunk of previously proposed equivariant ML architectures. (2) We provide detailed reasoning for our empirical results that clearly suggest that NIso can learn to be equivariant even to certain non-unitary groups.

### Equivariance to unitary transformations
NIso can learn to be equivariant to any transformation that preserves the norm of its input. Examples for this include:
Shifts on the torus, as demonstrated in Fig. 3 of the main paper.
SO(3) with spherical inputs, as demonstrated in Fig. 1 of the rebuttal PDF.
The group of 90 degree rotations mentioned by the reviewer, which we empirically validated but were not able to include in the rebuttalPDF due to space constraints (we will include this in the camera-ready paper!)

We note that prior work in equivariant ML at top conferences is often equivariant to only **a single one** of these transformations, with significant expert crafting. For instance, SO(3) is a non-abelian group and its IURs are substantially complex – nevertheless we are able to recover a basis and representation of transformations with  *exactly* the same properties. 

The fact that NIso learns to be equivariant to all these transformations is a significant strength - we are not aware of any previous work in symmetry discovery that has demonstrated this capacity. 

We hope that this is absolutely un-ambiguous: **NIso can learn to be equivariant to a transformation that preserves a norm of its input**.

### Equivariance to non-unitary transformations
We agree that the statement “Any transformation can be approximated with a sufficiently strong encoder” was too loose. We specify it in the following. 

As discussed above, NIso can closely approximate **unitary** transformations on its input. 
For other more complex transformations, our experiments suggest that a sufficiently expressive autoencoder can, in fact, discover a latent space where the manifestation of the transformation is approximately unitary and even isometric.  We believe this is a surprising and interesting result that is perhaps being overlooked. We note that it is not understood which transformations or groups can be embedded in this way, and point out that this is an exploration of fundamental math, beyond the scope of any ML paper.

However,  a potential explanation of this phenomenon could be related to the fact that for many challenging transformations, there exists a domain in which these transformations are either unitary or isometric. For example, Mobius transformations are isomorphic to the isometries of the hyperbolic ball with their action on the sphere representing their restriction to the boundary. Similarly, camera motions in the image plane are in fact projections of the isometric action of SE(3) on the underlying 3D scene.  

We hypothesize that by forcing the network to find an isometric representation for the observed transformations we are implicitly regularizing the learned latent space to discover information about this underlying geometry. We note that our ability to recover reasonable camera poses from the estimated transformations where other representation learners fail is indicative of this.

An additional illustrative example supporting this hypothesis can in fact be seen in Figure 7 of the supplement, where we show examples of failure cases in which the pose extraction from $\tau_{\Omega}$ fails catastrophically. Here we observe that these failure cases often coincide with sequences of frames where background and foreground objects are respectively far from and close to the camera – both cases where depth-estimation pipelines also often fail. 

### Conclusion
All in all,  NIso is a novel *first step* towards a new paradigm for symmetry discovery and equivariant ML. To the best of our knowledge, no prior work in symmetry discovery has been able to show, for instance, self-supervised discovery of approximate equivariance to a variety complex transformations, some of which do not even form a group in the observation space (such as the action of camera motions on the image plane). 

Using the additional page of the paper, we will add above discussion to the “Methods” section of the paper - we agree with the reviewer that this will make the paper stronger!
Thanks for your clarification. I have read the experiments in the attached pdf, and I am clear now that the equivariance error is at least comparable with the existing methods. The reply on the experiments is also convincing.

However, the other two concerns are not addressed: the type of equivariance can be modeled and the robustness to partial visibility.  The author argued that  "Any other transformation can in theory be approximated with a sufficiently expressive latent space". It is imprecise. 

On the other hand, due to the lack of description of the robustness against partial visibility, and the model does not have any constraint on the equivariance of the input, it is hard to understand why the model is robust or how robust I should expect it to be for practical tasks. The author mentioned that this issue will be investigated in the next step, but since current version of paper does not include any discussion on this issue, I think it is not complete.