Grounding Continuous Representations in Geometry: Equivariant Neural Fields
Authors: David Wessels
David M Knigge
Riccardo Valperga
Samuele Papa
Sharvaree Vadgama
Efstratios Gavves
Erik J Bekkers
Venue: ICLR 2025 Poster (2025)
Keywords: Geometric Deep Learning
Neural Fields
Equivariance
Representation Learning
Latent Point Clouds
Primary area: learning on graphs and other geometries & topologies
TL;DR: Grounding Continuous Representations via Equivariant Neural Fields (ENFs) in Geometry by utilising equivalence relations over (latent, coordinate)-pairs

Abstract: Conditional Neural Fields (CNFs) are increasingly being leveraged as continuous signal representations, by associating each data-sample with a latent variable that conditions a shared backbone Neural Field (NeF) to reconstruct the sample. However, existing CNF architectures face limitations when using this latent downstream in tasks requiring fine-grained geometric reasoning, such as classification and segmentation. We posit that this results from lack of explicit modelling of geometric information (e.g. locality in the signal or the orientation of a feature) in the latent space of CNFs. As such, we propose Equivariant Neural Fields (ENFs), a novel CNF architecture which uses a geometry-informed cross-attention to condition the NeF on a geometric variable—a latent point cloud of features—that enables an equivariant decoding from latent to field. We show that this approach induces a steerability property by which both field and latent are grounded in geometry and amenable to transformation laws: if the field transforms, the latent representation transforms accordingly—and vice versa. Crucially, this equivariance relation ensures that the latent is capable of (1) representing geometric patterns faitfhully, allowing for geometric reasoning in latent space, (2) weight-sharing over similar local patterns, allowing for efficient learning of datasets of fields. We validate these main properties in a range of tasks including classification, segmentation, forecasting, reconstruction and generative modelling, showing clear improvement over baselines with a geometry-free latent space.

Published as a conference paper at ICLR 2025
GROUNDING CONTINUOUS REPRESENTATIONS IN GEOMETRY: EQUIVARIANT NEURAL FIELDS
David R. Wessels*: David M. Knigge Riccardo Valperga Samuele Papa",
Sharvarce Vadgama', Efstratios Gavves 2,3 Erik J. Bekkers
1 AMLab, VVSL University of Amsterdam,
3 Archimedes/Athena RC
d r. wessell s@uva. nl, d. m. kniggeeuva.nl
ABSTRACT
Conditional Neural Fields (CNFs) are increasingly being leveraged continu- ous signal representations, by associating each data-sample with a latent variable that conditions a shared backbone Neural Field (NeF) reconstruct the sample. However, existing CNF architectures face limitations when using this latent down- stream in tasks requiring fine- ane-grained geometric reasoning, such as classification and segmentation. We posit that this results from lack of explicit modelling of geometric information (e.g. locality in the signal or the orientation a feature) the latent space of CNFs. As such, we propose Equivariant Neural Fields (ENFs), a novel CNF architecture which uses a geometry-informed cross-attention to con- dition the NeF enables an equivariant decoding from latent to field. We show that this approach induces steerability property by which both field and latent grounded in ge- ometry and amenable to transformation laws: the field transforms, the latent rep- resentation transforms accordingly- and vice versa. Crucially, this equivariance relation ensures that the latent is capable of (1) representing geometric patterns faitfhully, allowing for geometric reasoning in latent space, (2) weight-sharing over similar local patterns, allowing for efficient learning of datasets of fields. We validate these main properties in a range of tasks including classification, seg- mentation, forecasting, reconstruction and generative modelling, showing clear improvement over baselines with geometry-free latent space. Code attached submission here Code for clean and minimal repo here
geometric variable-2 latent point cloud of features that
INTRODUCTION
Neural Fields (NeFs) Xie et al. 2022) have recently gained prominence in the machine learning community as a novel representation method that models data as continuous functions. These fields, expressed Sa Rd - RR map spatial suurdinates-such corresponding signal, like RGB values fo(x) E R3, with 0 representing the model's parameters. The parameters are optimized approximate target signal f. ensuring Vz fxx fz).. This capability makes NeFs effective for representing continuous spatial, spatio-temporal. and geometric data, particularly in cases where grid-based methods fall short (Dupont et al. 2022). Their promise lies in serving as resolution-free free representation that may be used irrespective of data resolution or discretization (Xie et al. 2022). Moreover, NeF-reprresentations unify downstream models over different data modalities, allowing for transfer of modelling principles between data modalities that
pixel locations
22-10
otherwise require data-specific engineering efforts (Dupont al. 2022 Papa et al. 2023).
Building on this concept, conditional neural fields (CNFs) introduce conditioning variable = E Z to the model. Given a dataset of fields D = {ffs gd + [ each specific field can now be represented by specific conditioning variable z; via Vz f((x) x fo(;; zi). while model weights are shared over the entire dataset. This approach enables CNFs to efficiently model datasets of fields using set of latent variables that condition shared backbone NeF. This allows for representing and analysing fields fi by means of their latent representation 2i, enabling novel approaches for
Equal contrihution.

---

Published as a conference paper at ICLR 2025
1. optimize & to reconstruct (-)
2. use downstream
stap
reand
Fo (1S2E)
Mnnan (122)
context
F Foersa (:51)
"flower"
V,C
Mtp2
pose
fe (135328)
grounded latent latent
field f(-)
Figure I: Equivariant Neural Fields (ENFs) ground Neural Fields (NeFs) in geometry using a latent point cloud. latent set consisting of tuples (ps. c;) of pose information p; and context c; is optimized to reconstruct to the field f(-) as a function fo fo(=;2) 2) using gradient-descent. Due to their explicit positional grounding and locality, the latent retains important geometric features in the input field. The latent can then be used in downstream tasks, e.g. classífication, segmentation, and geometric reasoning, where transformations in the field are mirrored in the latent representation through group actions [f] g--2.
solving tasks involving fields through framework known as learning with functa Dupont et al. 2022]. Applications this include tasks such as classification, segmentation, and the generation of continuous fields (Dupont et al., 2022, Papa et al. 2023). as well as continuous PDE forecasting by solving dynamics in the latent space (Yin al.) 2022 Knigge et al. 2024).
Geometry in CNFs A notable limitation of conventional CNFs, however, lack of explicit geometric interpretability; each field f. is encoded by a "global" variable meaning for instance that any notion of locality or other explicit spatial relationships-whiie have proven a strong in- ductive bias in computer vision-is is lost. Althought this global representation has inherent benefits, e.g. enabling the use of simple MIPss as downstream models and allowing for intuitive interpola- tion between latent signal representations, empirically it has shown limited performance in settings where samples the dataset are not consistently globally aligned (Bauer et al.. 2023). For instance, in classification tasks, spatial organisation of an image's content is crucial for understanding shape and enabling geometric reasoning (Van Quang et al., 2019); current neural fields lack such geo- metric inductive biases, limiting their performance in e.g. classification and generative modelling (Bauer et al. 2023 Papa et al. 2023). To this end, we propose equivariant neural fields (ENFs), class of NeFs that allows for the identification of continuous fields with concrete geometric representations (Fig.
Grometry-grounderr neural fields When the goal is to utilise field representations 2; in down- stream tasks, it is crucial that these representations capture both textural/appearancee information and explicit geometric information. Our approach is inspired by the idea of neural -ecaraabel geometric representations (Vadgama et al.) 2022 2023), and addresses the pervasive issue of texture bias, which causes typical deep learning systems to overfit to textural patterns and ignore important geometric cues (Geirhos et al., 2018 Hermann et al. 2020). To address this challenge, we pro- pose defining representations that (1) explicitly separate aspects of snometry--pecificee
the pose of features-frow appearance and (2) are localized the input signal such that geometric concepts like orientation and distance are maintained from input to latent space. This necessitates that the ge- ometric components of the representations have meaningful structure, adhering to the same group transformation laws applicable to the fields. Geometrically, this means that distortions in the field translate to corresponding distortions the latent space, ensuring that geometric (shape) variations preserved and representable in latent space.
To establish an explicit grounding in geometry, we propose modelling the conditioning variables geometric point clouds cp comprising N pose-appearance tuples, with p, G being a pose (element) in a group G, and c; E Re an appearance vector. This representation space has well-defined group action, namely g= ((g,, c;) ) allowing to formalise the notion of
2

---

Published as a conference paper at ICLR 2025
to flax) (ar
ps canaaa
IDV moodas
3m
(lop..;lff
(eleca)
Figure 3: ENFs are local signal encoding; latent 2, optimized to represent local signal patch. We show that this inductive bias allows for leveraging weight-sharing, and improves down- stream performance by retaining important geo- metric features.
Figure 2: ENFs preserve transformations through their steerability property; if the field transforms with a group action the latents transform ac- cordingly via the following group action the pointcloud; {gp,, C; Cifi-l-
grounding a neural field through the following
Steerability property: Vg E G: folg :
fo(:; gz)
This property ensures that if the field transforms, the latent transforms accordingly (Fig. 2 This property has also been shown in (Atzmon et al.] 2022, Chatzipantazis et al., (0022) for learning implicit shape representations and well known in equivariance literature more broadly (Bekkers) 2019
Cohen et al., 2019: Weiler & Cesa 2019 Deng al. 2021)
Contributions
With this work we present the following contributions: new class of geometry- grounded equivariant neural fields that posses
the steerability property and thus proveable generalization over group actions
weight sharing which enables more efficient learning
localized representations in a latent point set which enables unique editing properties
We verify these properties through range of experiments that (1) support the claim that latents are geometrically meaningful, (2) show competitive reconstruction and representation capacity on segmentation, classification, forecasting and super-resolution tasks on image and shape data.
BACKGROUND
Neural Fields and conditioning variables Neural Fields (NeFs) are learned functions fo mapping
signal coordinates x to signal values f(x), parameterized by a neural network with parameters 0. Due to their flexibility they have emerged as prominent continuous data representation, applied on datatypes varying from object or scene data (Park et al., 2019 Mescheder et al. 2019 Sitzzann
al. 2020al Mildenhall al., 2021) to audio and images Tancik et al. (2020); : Sitzmann et al.
(2020b). In order to more efficiently represent whole datasets of signals, Conditional Neural Fields only optimise a latent conditioning variable z! per signal-instance f D instead of optimising a separate set of neural network parameters 0,. The two most common approaches for obtaining latents z are autodecoding (Park et al. 2019)- where latents 2 are initialized and optimized alongside NeF parameters 0 and Meta-Learning based encoding where optimization is split into outer loop that optimizes the backbone 0 and an inner loop that optimizes latents zf_ We explain both in detail in Appx. A.I.I and use both in experiments described in Sec.

---

Published as a conference paper at ICLR 2025
In seminal work by Dupont et al. 2002), datatype agnostic approach for learning these contin- uous signal representations was proposed involving first the optimization set of conditioning variables to reconstruct dataset of signals D == [f] رر= and afterward using these variables as a surrogate for the data in downstream tasks such as classification, generation and com- pletion. Although this work highlighted the flexible data-agnostic nature of Conditional Neural Fields (CNFs) by representing signals through single "global" condition variable 24, later work by Bauer et al. (2023) showed their limitations in performance for more complex tasks (i.e. involving higher-resolttion more varied natural data).
Group theoretical preliminaries The notion of transformatioo-preeevvvandd of an operator - such as the relation between field f and latent = is best expressed through group theory. A group is an algebraic construction (G.-) defined by a set G and a binary operator : x G + G called the group product, satisfying: closure: Vh. g E G h g E G. identity: Be E G Vg E G.g inverse: Vgag E G 9. = e and associativity; Vg, h, G hg
g.
g. (h-
Given such group G with identity element e G. and set X, can define the group action as a map G x X + X. which we will denote with direct multiplication i.c. a group element E G action on coordinate I E X is denoted gr. Note that when X G the group action equals the group product. For the group action on fields f : X - R we use a separate symbol, namely f(g_1x)) In this work we are interested in the Special Euclidean group (()) Tn x (O(n). SEnn) is the roto-translation group consisting of elements g (t, R) with group operation gg' = (t, R) (t', R') (t Rt RR'); the left-regular action on function spaces is defined by (z) = fl x) f( ()) t)).
[Z) (z)
Equivariant graph neural networks for downstream tasks. A key property of our framework is its ability to associate geometric representations with fields. This capability unlocks rich toolset for field analysis through the lens of geometric deep learning (GDL) (Bronstein et al.] (021). The GDL field has made significant advancements in the analysis and processing of geometric data, including the study of molecular properties (Batzner et al., 2022 Batatia et al.] 2022 Gasteiger al.) 2021: Brandstetter et al. (021) and the generation of molecules (Hoogeboom et 2022
Bekkers et 2023) and protein backbones (Corso et al. 2022 Yim et al. 2023). In essence, these approaches characterise shape. Our encoding scheme makes these tools now applicable to analysing the geometric components of neural fields, shown in Sec.
Equivariant Graph Neural Networks (EGNNs) are class of Graph Neural Networks (GNNs) that imposes voto-translational equivariance constraints on their message passing operators to ensure that the learned representations adhere to specific transformation symmetries of the data. Among the various forms of equivariant graph NNs (Thomas et al., 2018: Brandstetter et al. 2021 Satorras
let al. 202T] Gasteiger et al., (202T] Bekkers et al.] 2023; Bijkelboom et al. 2023 Kofinas et al. equivariant operator process our latent point-set representations of fields. For the neural field, we leverage the same optimal bi-invariant attributes introduced in Bekkers et al. (2023) which of homogeneous spaces- to parameterise our neural fields, allowing for seamless integration. They formalise the notion of weight sharing in convolutional networks as the sharing of message functions (kernels) over point-pairs e.g. relative pixel positions- that should be treated equally. By defining equivalence classes of point-pairs that are identical up to transformation in the group, we too derive attributes that uniquely identify these classes and enable weight sharing in our proposed ENFs.
2024 will utilise PONITA (Bekkers et al.] 2023)
analyse and
which are based on the theory
3 METHOD
In this section we introduce the Equivariant Neural Field (ENF) architecture. We start Sec. by showing how we impose the proposed steerability property (Eq. [ through the definition of bi-imvariant altributes. We construct cross-attention operator conditioned by these attributes. We subsequently constrain the operator to represent local sub-regions of the input domain by applying Gaussian window. Finally, we propose a k-nearest neighbouring approach to cope with the com- putational complexity of the cross-attention operation between pixels and latents. In Sec. 3.2 we finally discuss how we obtain a latent point cloud zf for signal f.
3.1
Bi-invariance constraint Before presenting our equivariant neural field design, we need to under-
stand the constraints imposed by the steerability property (Eq. l). The key result-shown also by
4

---

Published as a conference paper at ICLR 2025
Am,i == a(am.)
Ven v(C,, am,,)
k; = k(c))
C2
atlma
Vem
Tm
Fm
I'
Pt
2x
(pt: c,)
Iontks Ki
Jm
attm.: softmax tallor
folm; 2) :=
attma Vind
v dd
(a)
(b)
(c)
Figure 4: A visual intuition for the proposed cross-attention between coordinate Im and latent ((pi, ca)] (a) Bi-invariant am, is calculated between coordinate 3m and pose p: as P3 Tmm- (b) The query and key functions transforms am,i into a query qm,i, and key function maps context vector c, to key kj- Attention coefficients
calculated through softmax The softmax is taken over the N latents, yielding N attention coefficients attm,,, one for each latent 24. (c) A value m for each latent-coordinate pair is calculated as function of C2 and a, and the resulting values are aggregated, weighted by their corresponding attention coefficients attm,i-
qm. K4
Cohen et al. (2019; Bekkers (2019) in the context of equivariant CNNs-is that for steerability, the field to must be bi-invariant with respect to both coordinates and latents.
Lemma 1. A conditional neural field satisfies the steerability property iff it is bi-invariant, i.e., Vy E G: folga; gz) = [o(a; z).
Proof. If fo satisfies the steerability property, then folgx;g2) folg- gx:²)) fo(x; so is bi-invariant. Conversely, if fo is bi-invariant, then folg 1:000=x;20 2) = folog solog-'x;ga0 satisfying the steerability property equation]]
xx;22) = fo(x; gz),
3.1 EQUIVARIANT NEURAL FIELDS CONDITIONED ON GEOMETRIC ATTRIBUTES
The cross-attention operation enables the use of latent-sets as conditioning variables for CNFs by applying cross-attention between embedded coordinates and latent set of context vectors ==(c,) N (Zhang et al. 2023). Such cross-attention fields assign to each Vm a corresponding value fo (2m; 2), by matching coordinate (query) embedding q(xm) against latent (key) vectors k(c;) to obtain attention coefficients attm, and aggregating associated values v(c;) via
q(xm) k((:))
fo(#m;z)= W. Σ allm, v(c.)
with
albm, softmax
Vdg.
where W. maps cross-attention outputs to NeF output/signal dimension R. Our desired latent representation contains geometric component, namely the poses p, associated with the context vectors c, In order to how this geometric information could be leveraged, highlight how each the three components (q, k, v) could depend on the geometric attributes:
q q(=ms Pk) 7((m Pis C:)
fo(#mi2) W. Σ attm, (T Pt, c,)
with
attm,: softmax 1=1 i=1.....!
vdl
1=1
The steerability condition demands that the field has to be bi-invariant with respect to transforma- tions on both Z' and Ps, and the easiest way to achieve this is to replace any instance of Im. P; by an invariant wair-wise attribute a (m Pt) that is both invariant and maximally informative. With maximally informative we mean that coordinate- -pose pairs that are the same up a group action receive different vector descriptor, i.e., a(Tm: Pc) a (I' 1/) if and only if there exists g E G such that zrm gmm and p¹ gp:-
Equivariant Neural Fields Based on recent results in the context of equivariant graph neural networks Bekkers et al. (2023). we let ( Pr) := p. -I z be the the unique and complete bijective identifier for the equivalence class of all coordinate-pose pairs that the same up group
5

---

signals in the Published as a conference paper at ICLR 2025
action. Bijectivity here implies that the descriptor P; 2m contains all information possible to identify the equivalence classes, and thus the use of those attributes leads to maximal expressivity. In this work we use bi-invariants for translation (a roto-translation (aSB(2)) as well "bi- invariant" that breaks any cquivariance a" (see Appx. B for details). We define the ENF as follows:
N
q(a(2, p.))
fola;2) = W. attm, ((),
with attm, softmax
1-1 1......
ddn
As specific parameterizations for a, q. k, v choose:
a(r,p:) == op, 2z) q(a(p z)) = W W,a(z,p.) k(c,) := WaC,
(2)
(3)
v(a(s,p)),.) = (W.c,) (W Wa, a(x, + (Waja(z,p)
(4)
with @ denoting eiement-wise multiplication and a relative coordinate embedding function which we set to be Gaussian RFF embedding (Tancik et al. (020)) In neural field literature it is known that neural networks suffer from high spectral biases (Rahaman al. 2019). Due smooth input- output mappings it becomes difficult to learn high-frequency information in low-dimensions such as the coordinate inputs for a NeF. Gaussian RFF embeddings introduce embedding to alleviate the spectral bias.
Since the value transform has as goal to in spatially varying signal patches during reconstruction, the value-furee funeefnnniion is also conditioned on the geometric attributes p- lz. To add extra ex expressivity we chose to apply the conditioning via FiLM modulation (Perez et al. (2018) which applies a feature- wise linear modulation with a learnable shift 3 and scale y modulation.
crucial difference with standard transformer-type methods on point clouds that cross-attention between relative position embeddings relative the latent pose p. and that the value transform is of depth-wing wise separable convolutional form (Chollet (2017 [ekkerss et al. (2023), which is stronger form of conditioning (Koishekenov & Bekkers 2023) than additive modulation as is typically done in biased self-attention networks such as Point Transformer (Zhao et al. 2021)
Enforcing and learning locality in the latent point cloud With the current proposed setup, cross- attention is universally applied across the entire set of latents and coordinates. Given the Softmax distribution, each coordinate indiscriminately receives a nonzero attention value for every latent (Pr, ca) E z. Consequently, although latents possess inherent latent space positional attributes, they do not strictly represent localised regions of the signal, breaking the locality we require.
To address this issue, we modify the attention mechanism by incorporating a Gaussian spatial windowing function with parameter aan into the computation of attention coefficients. This ap- proach follows the strategy proposed by [or- donnier et al. (2019). Specifically, the atten- tion scores derived from the dot product be- tween the query and key values are modulated by Gaussian window μe, which is dependent on the Euclidean distance between latent posi- tions poos and the input coordinates, expressed as Po (Xm, Ps)) = - -aa lp,p xm 1 Here, aau is hyperparameter that regulates the size for each latent (Fig. 5) We have:
abkeu
(a)
(b)
Figure (a) Global attention between coordi- nates 3im and latents 2; can result in high atten- tion values for non-local latents. (b) Locality is enforced through a Gaussian window Ho (Im: Pi), attenuating the dot-product qm,ik, k, as a function of the distance between I. and pt.
p()( +=o(t..))).
att(x, 2) softmax s=.....
(5)
Vdl
where p( P.) represents the Gaussian window computed for each latent position. To enhance the expressiveness of the model even further, Jatt can be made latent-specific, encoding for the spatial extent of latent 2,. This extension allows the latents to be expressed point clouds: z! {(p,, Cj o, > effectively coupling position, appearance, and locality attributes within the latent space. However, keep this for further research fixing Jau in experiments.

---

Published as a conference paper at ICLR 2025
A note efficiency A limitation of the proposed method is the considerable computational com- plexity required to calculate output values for large numbers of input coordinates; larger more com- plex signals require a larger number of latents to be represented accurately leading to exponential number of attention coefficients needing to be calculated each forward pass (complexity scales as O(Nittets X N Noooddinates)) Since we localize latents, larger input domains present trade-off. Either we maintain small number of latent points-requiring larger att to cover the entire domain, diminishing locality-or we increase the number of latent points, aggravating computational cost.
To mitigate the computational overhead associated with the cross-attention between the latent points and the sampled coordinates, we propose employing k-nearest neighbours (k-NN) approach for the cross-atie cross-attention operation. Specifically, for each pixel, we first identify its k-nearest latent points and then compute pross-attention for this coordinate only over these nearest latents. This approach reduces computational cost while preserving the advantages of local representations (Appx. D.5
Properties of Equivariant Neural Fields Weight sharing Conditioning ENF's attention operator on attributes a which uniquely iden- tify equivalence classes of latent-coordinater pairs, ensures that the cross-attents cross-attention opera- tors-be it the attention logits the value transform-respond similarly regardless of the pose under which a signal pattern presents it- self. This form of weight-sharing has shown improved data and representation efficiency in GNN-based architectures (Bekkers et al. (2023) and-as as such we hypothesize that our pro- posed ENF architecture similarly benefits from these properties compared to other types of NeFs. We confirm this property in figure I0 which shows that ENFs share weights over group actions G for the geometry in which the point clouds are grounded. In Sec. f
focren (:2())
foreon (#131)
foneon (:3
(a)
(b)
(c)
Figure 6: Latent point cloud editing. Subfigures
(a) and (b) show two reconstructed CIFAR- im- ages with corresponding latents 2 chack Sub- figure (c) shows a reconstruction of the latent set ear-duck when selecting latents from either sar or duck based on their position.
we verify these benefits on downstream tasks.
Locality and (geometric) interpretability- The use of latent point clouds allows for localization of the cross-attention mechanisms around the latent pose, akin to how a convolution operator works with localised kernels. Not only does this improve interpretability and downstream performance (field patterns can be attributed to specific latent points), it enables unique field editing possibilities. Since our method is based on sets of latents and each element is responsible for local neighbourhood of the input domain, we can take arbitrary unions(c.g. stitching) or intersections(c.g latent- meeenn)merging) of point-sets of different samples (Fig. 6
3.2 OBTAINING LATENT POINT CLOUDS
Following Park al. (2019) : Dupont et (022) we obtain latent point cloud 2f for a specific sample I using gradient descent, optimizing 2 for reconstruction of the original signal f. For instance, in images latent point cloud may be optimized for an T-2-loss between fo(,,=) and f()). Of course, this also requires optimizing 0 to meaningfully map latents to fields. The two most common approaches to this end are Autodecoding (Park et al. 2019) where zf and are optimised simultaneously over dataset, or MAML (Finn et 2017; Tancik et al. 2021) where optimization is split into an outer and inner loop, with 0 being optimized in the outer loop and & being re-initialized every outer step to reconstruct the current signal batch in limited number of SGD steps in the inner loop. We detail these approaches in Appx. A.I.I using both in the experiments.
4 EXPERIMENTS
First, we show the ability of Equivariant Neural Fields (ENFs) reconstruct datasets of input fields f - Le. to associate a latent point cloud 3 to a given dataset of samples f, E D that accurately reconstructs them. Then, we validate ENFs as improved NeF-based downstream representation

---

Published as a conference paper at ICLR 2025
Table I: Test-set reconstruction PSNR (dbf) on CIFARIO, CelebA64x64, ImageNet1K, test accu- racy (%1) on CIFARIO.
Test reconstruction (IoU1) on ShapeNet16 and ShapeNet55 and test clas- sification accuracy (%f) on ShapeNet16.
Table 2:
CIFARIO
CELEDA
IMAGENET
SHAPENETI6 VOXEL (OCC)
SHAPENET55 CLOUD (SDF)
MODALITY
TASK
RECON,
CLASS.
RECON.
RECON.
TASK
RECON.
CLASS.
RECON.
ENF =²
ENF a
42.2
36.5
82.1
34.6
24.7
27.5
ENF a*
NF2vec
Functa
92.1
90.7
93.3
25.7
ENF 98222)
41.6
81.5
32.9
26.8
ENF 2²
92.9
96.6
73.2
for various tasks requiring geometrie reasoning; classification, segmentation and forecasting. To show the flexibility of NeF-bbaed representations, we perform these tasks on range of modalities.
Each downstream experiment consists of two stages: (1) fitting ENF backbone fo for reconstruc- tion of the input fields fj E D to obtain latents pointclouds 24, and (2) training a downstream model-that takes 2 as input-for each specific task. The bi-invariant am, we choose to con- dition our ENF, as well the downstream model, varies depending one the type of task we'r deeeciibedd per experiment below. Since our goal is to assess the impact of grounding continuous representations in geometry, besides -ppecfc baselines, we also compare against Functa (Dupont et al. 2022), the framework that originally proposed functional representations 2 as data surrogates.
For experimental details and hyperparameters we refer to Appx.
4.1 RECONSTRUCTION CAPACITY
First we evaluate our proposed ENFs on their reconstruction capabilities. Image data We show results for reconstruction trained with Meta-Learning CIFARIO Krizhevsky et al. (2009),
CelcbA64x64 (Liu et al. 2013) and ImageNetIK (Deng et al., 2009) using different bi-invariant at- tributes am. resulting equivariance different corresponding transformation groups in Tab.
We provide results for a Functa baseline, following the setup described in (Dupont et al. 2022).
Notably, translational weight sharing (a R outperforms settings with no-transformation (a") and roto-translational weight sharing (a (E(())). Moreover, it seems that locality alone is itself a useful inductive bias when moving to higher resolution, more varied images; Functa outperforms ENP a on CIFAR10 reconstruction, but CelebA and ImageNet all ENF parameterizations outperform Functa. These results reinforce locality and equivariance as inductive biases in image-based con- tinuous reconstruction tasks. Shape data We show shape reconstruction results (Tab. 2 on two common shape representations; voxels (3D occupancy grids) and meshes. For voxel data we take train and test splits from the 16-class ShapeNet- segmentation dataset (Yi et al., 2016) (which we denote ShapeNet16) and fit their corresponding voxel-based representation also using the obtained representations in the ShapeNet- segmentation experiment detailed below, For mesh data we opt instead to fit the full 55-class ShapeNetCore (v2) object dataset Chang et al. 2015), fitting these with 150k points sampled from the signed distance function the mesh (more details in Appx. C.3) Unlike Dupont et al. (2022), we were unable to get sufficient quality reconstructions with meta- learninn and instead obtain latents & using autodecoding on shape data (finding discussed in Appx. A.I.T. Results show that ours well as the baseline models struggle with accurately reconstructing the underlying shape from the SDF point clouds, we think due to the more complex optimization objective.
occupancy function
R ( 1)
4.2 DOWNSTREAM TASKS
Image classification One major limitation of Cuncta-noted by Bauer et al. (2023) was lacking performance on complex image tasks such as classification. To show performance of our model in this setting. we reproduce the CIFAR10 classification experiment listed in Bauer et al. menting CIFARIO with 50 random crops and flips per image, and training an ENF to reconstruct these using meta-learning, obtaining latent point clouds z. We do this for different bi-invariants corresponding to equivariance (a"), translational Ra and roto-translational (a (E(2) equivari-
aug-
8

---

Published as a conference paper at ICLR 2025
ance. We then train PONITA classifier (Bekkers al. 2023 to classify these latent point clouds conditioning the message passing function on the same bi ii-inaarsans, now calculated between poses Ps- Results (Tab. I show a test-accuracy improvement of 13.8 percentage points (68.7%--2..'%) over Functa (Dupont et al. 2022), and also indicate that in this setting (roto- (ioo-)translationne equivari- ance is a strong inductive bias with a SE(22) aR² -ENFs outperforming a -EEEss.
Shape classification Highlighting the flexibility of NeF-representations we apply the same setup to shape classification, training PONITA classifiers on the aforementioned ShapeNer16 dataset. Re- sults in Tab. X show that relevant geometric features are better preserved in localized latent space. Here, the performance difference between equivariant (a and non-equivariant (a?) ENFs are neg- ligible. This is to be expected due to the global alignment of the ShapeNet dataset, and shows ENF able to perform under equivariance constraints even in non-equivariant tasks.
ShapeNet-Part segmentation Where classification primarily evaluates how well the latent point clouds captures global information, we also want to evaluate ENFs performance on fine-gres fine-grained tasks. such, we evaluate on the ShapeNet part segmentation task (Yi et al., (016). The ShapeNet-Part dataset consists of point clouds for 16 ShapeNet object-classes, each with a varying number of annotated parts for a total of 50 segmentation classes. We use the ENF a² backbone defined in the voxel reconstruction task above (fe to obtain a latent = for a shape. Then, second ENF (fe..) is trained to map points on this shape to one-hot encoding of their corre- sponding segmentation classes, i.e. So (Tmi =) maps I' to its class label ym- Results (Tab. B} somewhat surprisingly show ENF and Functa perform comparably in this task (detailed results and visualizations in Appx. D.6). We again think this attributable to the fact that all shapes are aligned and centered we further investigate these results in Appx. D. We additionally include results for point cloud-specific architectures, and NF2 Vec a framework for self-supervised representation learning on 3D data from (non-conditional NeFs. These results show that ENF only slightly underperforms modality -ppeciiic baselines.
Table 3: Mean class and instance IoU (t) on ShapeNet.
aerstoo
Monn
PountNer PoietNats
N3.3
79.0
84.9
82.7
DOCNN
83.6
30.9
NF2vee
81.3
76.9
Fanesa
82.8
74.8
ENP
82.2
Flood Map Segmentation For more chal- lenging segmentation task we apply ENFs on multi-modal flood mapping dataset (Drakon- akis et al.] 2022). This small dataset (759/85
STtwior
Statter
12
$2atte
GT
train/test split) provides dual-modal temporal data; aligned Synthetic Aperture Radar (SAR) and optical satellite images at 256x256 reso- lution obtained by satellites Sentinel and 2 (S1,S2), of disaster sites before and after their flooding, along with corresponding masks that segment the flooded area. The goal is to predict binary segmentation mask given these differ- ent input fields. We first train a reconstruction a (E(() fowe. with MAML to obtain a la- tent reon that decodes into the four observa- tions. Next, a segmentation ENF feag is trained to predict, given a latent grecon the binary mask at each location.
ENF
Figure 7: Top- OMBRIA test sample of SAR (S1), optical (S2) before and after flooding with ground truth flood map. Bottom- ENF reconstructions fa (-3 reeus) and predicted mask fe.., (:: reou)).
Table 4: Test IOU (t) for flood map segmen- tation on OMBRIA, for different observation
Table 5: Test IOU
(t) zero-shot resolution transfer OMBRIA. Sans So were trained on 128x resolu-
rates.
Mones
tion.
We evaluate and compare our model against the multi-modal U-Net proposed by Drakon- akis et al. (202)). As suggested by Dupont to fit
100% fm OBSERVED
OuabriaNet
16.77
N.A.
42.75
72.36
PSN
ENF
31,65
256-2256
TEST IESIE
Faseua
16:72
$7.14
et | (2022) we trained Functa-unaie the training set with MAML-using autode- coding instead. However, we found Functa un- able to generalize to test images in this com- plex low-data setting, collapsing to remember- ing the training dataset (achieving 31.5 recon PSNR and 93.7 loU the 256 256 train set). Results (Tab. 4
or fs DUSERVED
28.11
OmbriaNet
N.A.
27.02
Functa
16.71
42.74
ENF
31,37
73.65
29.31
73.21
10% oF fa OBSERV 2D
64 64 725T RESOLUTIION
DelviaNot
N.A.
0.0
Purera ENF
16.5%
MD
Functa
16.77
42.92
33.31
72.59
ENF
Fig. 7 show the importance of
9

---

Published as a conference paper at ICLR 2025
Ground truth AT - -Taa() Iil)
Predicted Af= fol: fo(;))
Table 6: ERA5 reconstruction T--MSEL and 1-hour forecast- ing T++1-MSEJ. oween ground truth observa-
*MSE be-
tions at T and T2+1-
M.ES
MSEE
Identity*
2,42E-05
Figure 8: Visualization of ground truth ERAS test sample and ENF prediction of the change in temperature between observa- tions T, and T We show the true (AT) and predicted (A?') difference between the temperature maps at / and +++ 1.
Functa
5.75E-05 8,04E-06
3.45E-03
ENF
9.44E-06
inductive biases in complex limited-data regimes. We provide results for subsampled observations simulate missing data, showing the robustness of NeF-baaed methods sparsity ENF performs well even at 10% observation rate where classical convolution- methods fail. Moreover, we show zero-shot resolution transfer results (Tab. (5) where fe fe... are trained on 128 x 128 resolu- tion data are deployed on 64 x 64 and 256 x 256 resolution data without fine-tuning, showing the resolution agnostic nature of NeF-representations.
a
ERA5 Climate forecasting Following (Yin et al. 2022 Knigge et al. 2024
we evaluate our NeF-
based representation dynamics forecasting. ERAS (Hersbach et al. 2019 is a dataset of hourly global temperature observations. We use the dataset as described in Dupont et al. (202T), which contains data defined over 46 x 90 latitude-longitude grids. From the training and test sets, we extract 5693 and 443 pairs of subsequent observations T,, T2+1 for train and test sets respectively. Using MAML, we train an ENF fo with bi-invariant a (no symmetries exist in this data) to reconstruct the global temperature state T, using 2t, and optimize PONITA MPNN to predict update As, that maps 24 to a latent 1++1 = 2 + Az which decodes into the state at + 1, i.e. Te+1 2++1) Ro fe(:; 2 22)) Training is done sequentially, i.e. first the backbone is optimized and afterward the MPNN is trained, keeping fe fixed. Results (Tab, 6 Fig. 8 show that the latent space of ENF lends itself well for modelling such complex dynamics where global latent representation such
as Functa seems unable to model the relevant fine-graine fine-grained details needed for forecasting.
Image generation Following Dupont al. 2022);; Bauer et al. (2023), we provide results for diffusion applied
ENF Funeta
Table 7: FID for generative modelling on CIFAR-I0 and 6eleba64x64
dataset of latents obtained from pre- trained ENFs on CIFARIO and Celeba64 x 64. As down- stream diffusion model, we uti- lize DiT-B (Peebles & Xie 2023), a natural choice for our set-latent (training detailed in Appx. C.4. We provide results in FID (Heusel al. (017) for unconditional generation in Tab. and samples in Fig. 9 We provide comparison to Functa Dupont al., 2022). as well as other frameworks for generative modelling over fields (Du al.) 2021 Dupont et al. 2021; Zhuang et al.) nnoaaly each of these methods is trained on generative objective and does not support self-supervised pre-training like Funeta or ENF. On globally aligned CelebA64 64, both Functa and ENF produce perceptually qualitative samples, but unlike ENF, Functa is unable to generalize to CIFAR-IO, where data is less homogeneous and not aligned. These results again show clear benefit of geometrically interpretable latent space for downstream tasks, though previously proposed frameworks specific to generative modelling over fields achieve better performance than ENF. The latter points to possible area of improvement, and future work could look into incorporating insights from these works into a generative adaptation of the ENF frame- work, e.g. through latent-space regularization of perceptual consistency per Du al. 2021.
ENF Functa
CelehAG1x61 RID
CIFAR-10
Monm
GASP
13.5
DPF
13.2
15.1
Figure 9: Qualitative sam-
Functa ENF
78.2
ples for generative modelling on
23.5
CIFAR-10 and Celeba64x64..
10

---

Published as a conference paper at ICLR 2025
5 CONCLUSION
Building upon fascinating work using Neural Fields (NeFs) continuous data surrogates, this paper introduces Equivariant Neural Fields (ENFs); novel NeF parameterization that re-introduces induc- tive biases (locality, equivariance) into NeF-based representations. ENF uses geometry-grounnd conditioning variable-a latent attributed point cloud-to achieve an equivariant decoding process, ensuring that transformations in input field are preserved the latent space and enabling steering of the latent to transform the output signal. This steerability property allows for the accurate repre- sentation of geometric information, and for efficient weight-sharing over spatially similar patterns, significantly improving learning efficiency and generalization, as validated on range of experi- ments with varying data modalities and objectives.
6 REPRODUCIBILITY
All datasets be downloaded via their cited references without any effort except for the ShapeNet dataset which requires registration and approval. The pre-processing steps are described in ap- pendix C.3 For all model parameter settings for the ENFs, downstream models or Functa to the appendix C As supplementary material we added a codebase containing code to reproduce results for the CIFAR 10 and OMBRIA experiments. Code for all other experiments will be released during the rebuttal phase of the review process, containing all settings to reproduce the experiments in config files.
refer
7
ACKNOWLEDGEMENTS
David Knigge is partially funded by Elekta Oncology Systems AB and RVO public-private part- nership grant (PPS2102). David Wessels partially funded Ellogon.AI and public grant of the Dutch Cancer Society (KWF) under subsidy (15059/202-PPSS2) This work used the Dutch na- tional e-infrastructure with support of the SURF Cooperative using grant no. EINF-9549 and EI5F-10544.
REFERENCES
Matan Atzmon, Koki Nagano, Sanja Fidler, Sameh Khamis, and Yaron Lipman. Frame averaging for equivariant shape space learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 631-641, 2022.
Ilyes Batatia, David Kovacs, Gregor Simm, Christoph Ortner, and Gábor Csányi. Mace: Higher order equivariant message passing neural networks for fast and accurate force fields. Advances in Neural Information Processing Systems, 35:11423-114 1436, 2022.
Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P Mailoa, Mordechai Ko- mbluth, Nicola Molinari, Tess E Smidt, and Boris Kozinsky. E (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature communications, 13(1): 2453, 2022.
Matthias Bauer, Emilien Dupont, Andy Brock, Dan Rosenbaum, Jonathan Richard Schwarz, and Hyunjik Kim. Spatial functa: Scaling functa to imagenet classification and generation. arXiv preprint arXiv:2302.031300 2023.
Erik Bekkers. B-spline cnns on lie groups. arXiv preprint arXiv:1909. 2019.
Erik J Bekkers, Sharvaree Vadgama, Rob D Hesselink, Putri A van der Linden, and David w Romero. Fast, expressive se (n) equivariant networks through weight-sharing in position- orientation space. arXiv preprint arXiv:2310.02970. 2023.
Johannes Brandstetter, Rob Hesselink, Elise der Pol, Erik J Bekkers, and Max Welling. Geomet- ric and physical quantities improve (3) equivariant message passing. In International Conference Learning Representations, 2021.

---

Published as a conference paper at ICLR 2025
Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličkovié. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478 2021.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, al. Shapenet: An information-rich 3d model repository. arXiv preprint 1r22v:15/..03012. 2015.
Evangelos Chatzipantazis, Stefanos Pertigkiozoglou, Edgar Dobriban, and Kostas Danilliis. Se (3)-equivariant attention networks for shape reconstruction in function space. arXiv preprint arXiv:2204.02394, 2022.
François Chollet. Xception:
Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251-1258, 2017.
Taco Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant cnns on homo- geneous spaces. Advances in neural information processing systems, 32, 2019.
lean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self- attention and convolutional layers. arXiv preprint
2019.
Gabriele Corso, Hannes Stärk, Bowen Jing, Regina Barzilay, and Tommi S Jaakkola. Diffdock: Diffusion steps, twists, and turns for molecular docking. In The Eleventh International Conference on Learning Representutions, 2022.
Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas J Guibas. Vector neurons: A general framework for so -e)eeuriari networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12200-12200 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Georgios I Drakonakis, Grigorios Tsagkatakis, Konstantina Fotiadou, and Panagiotis Tsakalides. Ombrianet--supervieed flood mapping via convolutional neural networks using multitemporal sentinel-1 and sentinel-2 data fusion. IEEE Journal of Selected Topics in Applied Earth Observa- tions and Remote Sensing, 15:2341-2356 2022.
Yilun Du, Katie Collins, Josh Tenenbaum, and Vincent Sitzmann. Learning signal-agnostic mani- folds of neural fields. Advances in Neural Information Processing Systems, 34:8320-8331, 2021.
Emilien Dupont, Yee Whye Teh, and Arnaud Doucet. Generative models as distributions of func- tions. arXiv preprint arXiv:2102.04776. 2021.
Emilien Dupont, Hyunjik Kim, SM Eslami, Danilo Rezende, and Dan Rosenbaum. From data to functa: Your data point is a function and you can treat it like one. arXiv preprint arXiv:2201.12200 12204 2022.
Floor Eijkelboom, Rob Hesselink, and Erik J Bekkers. E (n) equivariant message passing simplicial networks. In International Conference Machine Learning, pp. 9071-9081. PMLR, 2023.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In International conference on machine learning, pp. 1206-1135. PMLR, 2017.
Johannes Gasteiger, Florian Becker, and Stephan Günnemann. Gemnet: Universal directional graph neural networks for molecules. Advances Neural Information Processing Systems, 34:6790- 6802, 2021.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias im- proves accuracy and robustness. arXiv preprint arXiv:1811.12233 2018.
Katherine Hermann, Ting Chen, and Simon Kornblith. The origins and prevalence of texture bias in convolutional neural networks. Advances Neural Information Processing Systems, 33:19000- 19015, 2020.
12

---

Published as a conference paper at ICLR 2025
Hans Hersbach, Bill Bell, Paul Berrisford, G Biavati, András Horányi, Joaquín Muñoz Sabater, Julien Nicolas, C Peubey, R Radu, Rozum, et al. Era5 monthly averaged data on single levels from 1979 to present. Copernicus Climate Change Service (C3S) Climate Data Store (CDS), 10: 252-266, 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffu- sion for molecule generation in 3d. In International conference on machine learning, pp. 8867- 8887. PMLR. 2022.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 11X.6::412.6980,, 2014.
David M Knigge, David Wessels, Riccardo Valperga, Samuele Papa, Jan-Jakob Sonke, Efstratios Gavves, and Erik Bekkers. Space-time continuous pde forecasting using equivariant neural fields. arXiv preprint urXiv:2406.06660 2024.
Miltiadis Kofinas, Boris Knyazev, Yan Zhang, Yunlu Chen, Gertjan J. Burghouts, Efstratios Gavves, Cees G. M. Snoek, and David W. Zhang. Graph neural networks for learning equivariant represen- tations of neural networks. In The Twelfth International Conference on Learning Representations, 2024. URLAALpp::/oppeeevieee.. http:///weee..tttooooooooooooooe sMYDDB
Yeskendir Koishekenov and Erik Bekkers. An exploration conditioning methods graph neural networks. arXiv preprint (rXXix23005009333 2023.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Oc- cupancy networks: Learning reconstruction in function space. In Proceedings the IÉEE/CVF conference on computer vision and pattern recognition, pp. 4460-4470, 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Samuele Papa, Riccardo Valperga, David Knigge, Miltiadis Kofinas, Phillip Lippe. Jan-Jakol Sonke, and Efstratios Gavves. How to train neural field representations: A comprehensive study and benchmark. arXiv preprint arXiv:2312.10531 2023.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 165-174, 2019.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks, In International conference on machine learning, pp. 5301-5310 PMIR, 2019.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.0055/2 2022.
13

---

Published as a conference paper at ICLR 2025
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural net- works. In International conference on machine learning, pp. 9323 93322. PMLR, 2021.
Vincent Sitzmann, Eric Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein. Metasdf: Meta-learning signed distance functions. Advances in Neural Information Processing Systems, 33:10136-10147. 2020a.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances neural information processing systems, 33:7462-7473, 2020b.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.025500 025002, 2020.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich- Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in neural information processing systems, 33:7537-7547, 2020.
Matthew Tancik, Ben Mildenhall, Terrance Wang. Divi Schmidt, Pratul Srinivasan, Jonathan T Barron, and Ren Ng. Learned initializations for optimizing coordinate-based neural representa- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2846-2855, 2021.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and and translation-cquivriaan neural networks for 3d point clouds. arXiv preprint arXiv:1802.082/19 2018.
Sharvaree Vadgama, Jakub Mikolaj Tomezak, and Erik J Bekkers. Kendall shape-vae: Learning shapes in generative framework. In NeurIPS 2022 Workshop on Symmetry and Geometry in Neural Representations, 2022.
Sharvaree Vadgama, Jakub M Tomezak, and Erik Bekkers. Continuous kendall shape variational autoencoders. International Conference on Geometric Science of Information, pp. 73-81. Springer, 2023.
Nguyen Van Quang, Jinhee Chun, and Takeshi Tokuyama. Capsulenet for micro-expression recog- nition. In 2019 14th IEEE International Conference Aulomatic Face & Gesture Recognition (FG 2019), pp. 1-7. IEEE, 2019.
Maurice Weiler and Gabriele Cesa. General e (2)-equivariant steerable cnns. Advances in neural information processing systems, 32, 2019.
Francis Williams. Point cloud utils, 2022. http://wwwwggiiinnoooo//lliiliie
Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual comput- ing and beyond. In Computer Graphics Forum, volume 41, pp. 641-676. Wiley Online Library, 2022.
Li Yi, Vladimir Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections. ACM Transactions on Graphics (ToG), 35(6):1-12, 2016.
Jason Yim, Andrew Campbell, Andrew YK Foong, Michael Gastegger, José Jiménez-Lume Sarah Lewis, Victor Garcia Satorras, Bastiaan S Veeling, Regina Barzilay, Tommi Jaakkola, et al. Fast protein backbone generation with se (3) flow matching. arXiv preprint 4rX1v.2310.05297, 2023.
Yuan Yin, Matthieu Kirchmeyer, Jean- Yves Franceschi, Alain Rakotomamonjy, and Patrick Galli- nari. Continuous pde dynamics forecasting with implicit neural representations. arXiv preprint arXiv:2209. 14855, 2022.
14

---

Published as a conference paper at ICLR 2025
Biao Zhang, Jiapeng Tang. Matthias Niessner, and Peter Wonka. 3dshape2vecset: A 3d shape representation for neural fields and generative diffusion models. ACM Transactions on Graphics (TOG), 42(4):1-11, 2023.
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 16259-16268, 2021.
Peiye Zhuang, Samira Abnar, Jiatao Gu, Alex Schwing, Joshua M Susskind, and Miguel Angel Bautista. Diffusion probabilistic fields. In The Eleventh International Conference on Learning Representations, 2023.
Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning. In International Conference Machine Learning, pp. 7693-7702. PMLR, 2019.
A APPENDIX
A.1 AUTODECODING AND META-LEARNING
A.1.1 META-LEARNING
When fitting samples with (Conditional) NeFs using autodecoding (gradient-descent based optimi- sation the latent at test-time) (Park et al. 2019), two key challenges emerge: (1) optimising the sample-specific parameters/latent for a novel sample can be time-consuming -taking e.g. up to 500 gradient updates (Yin et al. 2022) and (2) more gradient updates to NeF weights may impede downstream performance through phenomenon known overtraining (Papa al. 2023) -where the relationship between field f and 2 is obscured by oversensitivity to high-frequency details. To
address the first point, Sitzmann et al. (2020a); Tancik al. (202T): ) Dupont al. (2022) propose Model-Agnostic Meta-neearning (MAML.) (Finn et al. 2017 based optimisation method, enabling the network or latent initialisation be learned such that each sample can be fitted with just a few gradient steps. More specifically, Dupont et al. (022) proposes an inner-outer loop structure where the modulations are updated in the inner loop, while the base network weights are updated in the outer loop. This method corresponds to an instance of learning subset of weights with MAML, also known as Contextual Variable Interaction Analysis (CAVIA) Zintgraf et al. (2019). More recently, Knigge et al. (2024) note that this meta-learning approach also improves downstream performance by imposing structure on the NeF's latent-space. We provide pseudocode for this approach Alg.
Algorithm 1 Meta-learning ENF
Randomly initialize shared base network fo while not done do Sample batch of signals f Sample random coordinates x
Initialize latents zí (pi, C;) c;)).N 1im for batch of signals.
for all step E 1, ..., Ninner and j E do
z! 2f - mmsc ( (x (.).
end for Update ENF: 0 VeVe end while
A.1.2 AUTODECODING
During our experiments, we found that not all types of signals lend themselves easily to this encoding approach when using ENFs (specifically SDFs and occupancy functions). Although it saves time in inference and adds structure the latent space, (Dupont et al. 2022) also remark on the limited expressivity of Meta-Learning due to the small number of gradient descent steps used to optimize a latent As such, for all shape experiments we instead opt for autodecoding (Park et al.] 2019), in
15

---

Published as a conference paper at ICLR 2025
which latents and backbone optimized simultaneously. We provide pseudocode for this approach in Alg.
Algorithm 2 Autodecoding ENF
Randomly initialize shared base network fe
Initialize latents zf +- {(p:, c,)li1 for all signals
while not done
Sample batch of signals Sample random coordinates x
Update latent: ={+1 +- 2i. V. fume (fo(x, f())))
Update ENF: Be+1 + 0₂ -v Lime (fo, (z, f(x))
end while
A.1.3 A NOTE ON POSE INITIALIZATION
We noted during our experiments that initialization of the latent poses-i.e. their initial posi- tion/orientation in the inner loop-has significant impact on the reconstruction capacity and stability space them as equidistantly as of the ENF. We found that a good way initialize the latents possible and then adding small Gaussian noise ((, le - 3)), e.g. for 2D images on perturbed 2D grid. Any orientations are initialized canonically, i.e. all latents are initialized with the same orientation. When defining an equidistantly spaced grid is hard, for example point clouds or data defined on a sphere, we propose using Farthest Point Sampling on a training grid to initialize positions for the latents.
B BI-INVARIANT FUNCTION PARAMETERIZATIONS am,,
The bi-invariants attributes that are used in the experiments section are listed here. Translational symmetries R" In this setting, poses correspond to translations t; R"
Fmm
Roto-translational symmetries SE(2). In this setting, poses p: correspond group elements g (0,, t.) E SE(2). We adopt the invariant attribute introduced by (Bekkers et al. 2023):
SE(2)
Re -t))
No transformation symmetries. A simple "bi- invariaaintt for this setting that preserves all geometric information given by simply concatenating coordinates p with coordinates r::
a a. = Pt + Jmm
Parameterizing the cross-attention operation in Eq. 3.1 as function of this bi-invariant results in framework without any equivariance constraints. We use this in experiments to ablate over equivari- constraints and its impact on performance.
C EXPERIMENTAL DETAILS
We provide hyperparameters per experiment. We optimize the weights of the neural field Sa in all experiments with Adam (Kingma & Bal 2014) with a learning rate of Ic-4, and an inner step size of 30.0 for and 1.0 for (increasing inner step size in general speeds up convergence and improves reconstruction but may also lead to instabilities). For downstream classification we train an equivariant MPNN Fg, using 3 message passing layers in the architecture specified in [ekkers) ct al. 2023) conditioned on the same bi-invariant which was used to fit the ENF, with a hidden dimensionality of 256, always trained with learning rate le-4. The std parameters og. o, of the RFF embedding functions Sg:: Pu are chosen per experiment based on an ablation. In general, increasing both values leads to increased frequency response of the ENF, though generally the model is more susceptible to small change in o,. as well as hidden dim size and the number of attention heads are chosen per experiment, detailed below. We all experiments single H100.
16

---

Published as a conference paper at ICLR 2025
C.I IMAGE RECONSTRUCTION, CLASSIFICATION, SEGMENTATION, FORECASTING
CIFARIO reconstruction and classification For CIFARIO Krizhevsky et al. 2009 reconstruc- tion and classification we use hidden dim of 128 with 3 heads, 25 latents of size 64, batch size of 32 and restrict the cross-attention operator to k=4 nearest latents for each input coordinate x. For Jg. Te we choose 1.0 and 3.0 respectively. We train the ENF model and the classifier for 100 epochs.
CelebA64x64 For CelebA Liu et al. 2015) we use a hidden dim of 256, 36 latents of size 64, a batch size of and restrict the cross cross-attention operator to k=4 nearest latents for each input coordi- nate z. For g. Te we choose 2.0 and 10.0. We train the model for 30 epochs.
ImageNetIK reconstruction For ImageNetIK (Deng et al. 2009) reconstruction we use hidden dim of 128 with 3 heads, 169 latents of size 64, a batch size of 2, restricting the cross-attention operator to k=4 nearest latent for each input coordinate x. For og. oo we choose 2.0 and 10.0 respectively. We train the model for 2 epochs.
Ombria For OMBRIA Drakonakis et al. (2022) we trained a reconstruction model frr... with, 256 hidden dim, 4 heads, 169 latents of size 128, batch size 8. Restricting the cross-attention operator to k=1 nearest latent for each input coordinate æ. For ag. we choose 2.0 and 10.0 respectively. The segmentation model fagg has hidden size of 128, 8 heads, with cross-attention restricted to k-4 nearest latents, trained with batch size 16. For Og. Ty we choose 2.0 and 3.0 respectively. We train both models, sequentially, for 500 epochs.
ERAS forecasting For ERA5 forecasting (Hersbach et al. 2019) we train reconstruction model fe.... with 128 hidden dim, 3 heads, 36 latents of size 64, batch size of 32. Restricting the cross- attention operator to k=4 nearest latent for each input coordinate æ. For ogs d. we choose 2.0 and 8.0 respectively. Inputs are defined over latitude longitude 6, & grid, which coordinates per x = [cos f cos o, cos sin o sin 6). We first train the ENF for 800 epochs. As forecasting model, we train a PONITA MPNN of layers with 256 hidden dim for 1000 epochs. Both models are trained with a batch size of 32.
map to 3D euclidean
As objective, since don't want overfit the reconstruction error incurred by fitting latent z to the initial state, we supervise the forecasting model with Lg loss between the decoded output for predicted latent ε per:
L.fveeustt
|10 (5 + AT) (Fo (:; 1111
AT being the ground truth change in temperature, and Ê
F Prrrocu (zt).
C.2 FUNCTA BASELINE MODELS
For the Functa baselines (Dupont al. 2022 we try to keep as close possible to the setup defined by the original authors. However, we found that training deeper models in the shape experiments (2 8 layers) lead to very unstable training. Instead, for these experiments, we opted go for shallower models, up to 6 layers. For all experiments we use hidden dim of 512 and latent modulation size of 512 as used (Bauer al. 2023), except for ImageNetIK reconstruction, where we use a 1024 latent modulation. We would like to note here that in all experiments, the Functa baseline has larger parameter count than the ENF models applied each task (e-g. for CIFARTO 10, ENF has 522K params where Functa has 2.6M params). Although we did not explore this in-depth, it seems that the proposed ENF representation is more parameter efficient compared to the deep SIREN model defined in (Dupont et al. 2022).
For downstream models we follow (Bauer et al., 2023) and use a 1024 hidden dim 3 layer residual MLP (~2.1MM params). Like in our ENF experiments, we use the same architecture across tasks, only changing the output head to accommodate.
C.3 SHAPE RECONSTRUCTION AND CLASSIFICATION
Voxel Dataset and Segmentation The voxels are given with the ShapeNet dataset where the seg- mentation labels are given as point clouds. However, the coordinate frames of the voxels are differ- ent, to align them mapped both between -1 and 1.
17

---

Published as a conference paper at ICLR 2025
We trained model fe.... with a hidden dim of 128, 3 heads, 27 latents of size 32. We set oq J. for the RFF embedding functions &g. Oe to and 10 respectively. For Functa we used, a latent dim of 864 to have the same latent parameters as the conditioning variable used for ENF. As NeF we used a 5-layer Siren with a hidden-dim of 512, wo is set to 10. The modulation network is a two-layer ML.P of hidden sizes 256 and 512.
As is customary for ShapeNet-pan -art segmentation, we condition on the object class and supervise over all segmentation classes with a cross-entropy loss, but only calculate test IoU based on segmentation classes that correspond to the object class. We chose the class-emb dim to be 32 for all settings.
The segmentation NeFs are all trained for 500 epochs with the same parameters the reconstruction NeF. However, for ENF, we chose qq to be I, 1 for extra stability.
SDF Dataset To create the signed distance functions from Shape V2 Chang et al. (2015) objects, we took their meshes and made them water-tight using Point Cloud Utils (Williams) 2022] Afterwards, we sampled point cloud of 150,000 points from the surface. To create the actual SDF, perturbed the points with Gaussian noise along the mesh normals, and recalculate the signed distances to the surface. Finally, the dataset consisted of 55 classes with a total number of 42.472 and 5.000 samples for the train and test set respectively.
For ENF we used a latent point cloud of 27 points with context vectors of dimension 32. The std parameters 0g. By for the RFF embedding functions &y O. are 2 and 10 respectively. The hidden dim of the ENF was set to 128 and we used 3 attention-heads The
used latent modulation of 864 which corresponds to ENFs chosen 27+32 parameters for the conditioning variable. As a NeF we used a 5 layer Siren with an hidden-dim of 512 and we parameter of 15. As modulation network used a two-layer MLP with 256,512 hidden-dim.
For Functa (Dupont et al. 2022)
C.4 GENERATIVE MODELLING ON ENF LATENT SPACE
For generative modelling experiments CelebA64x64 and CIFAR-10 we train a Diffusion Trans- former (DiT-B) (Peebles & Xie 2023) on ENF latents, utilizing the context vectors input tokens and their positions as input for an RFF position embedding added to the tokens. We use the same approach for CelebA for CIFAR 10; train a ENF with MAML on the image dataset, and use this model to obtain sets of "ground truth" latents 2 == (pr, C for each image. We then train DiT-B on diffusion objective on this latent space, where the forward diffusion kernel is given by:
= (pi, Varcen + - [
with e E NO, 1). i.c. we only add noise to the latent vectors, as we find adding noise to the poses leads to unstable training (something to be investigated in future work). To generate a sample, we take random set of "ground truth" poses from the training set, and attach a context vector C₁t E NO, 1) to each pose to denoise. We supervise the DiT-B with the v objective (Salimans & Ho 2022). Like (Peebles& & Xiel 2023). we use mmax = 1000 linear noise schedule ranging from Ie 4 to 1e-2, 2, and generate samples using DDIM (ong et al. 2020 in 512 steps.
In both settings, we train the diffusion model for 200 epochs using Adam, constant learning rate of 1e-4, no weight-decay or dropout, and generate 50k samples to calculate FID.
D ADDITIONAL RESULTS
D.1 SIZE OF LATENT POINT-CLOUD
In this section, we delve deeper into the tlype--araamtert of the latent point clouds used as condi- tioning variables. Equivariant Neural Fields can increase the number of parameters used to represent signal in two ways: by increasing the number of latent points or by expanding the dimensionality of the context vectors. Intuitively, we can either enhance the representational capability of a single region in the input domain create more, smaller regions with lower representational dimensions.
18

---

Published as a conference paper at ICLR 2025
To gain further insights into how these conditioning variables behave, train multiple CNFs using different latent point-cloud configurations. In these experiments, the chosen latent dimension or number of latent points is adjusted to keep the total number of parameters close as possible across configurations. We trained each model for 400 epochs, as only small improvements occurred beyond this point and the overall trend was already clear. After fitting the ENF, we used the meta-learned representation to perform classification tasks. We employed a simple message-passing GNN, which trained for 20 epochs, after which performance improvements began to degrade. Below, present the ablation results for these different approaches to increasing representational capabilities.
Table 8: Reconstruction PSNR (db1) and ACC (%1) on CIFARIO for different parametrisations of the latent point-clouds, i.c. varying N, in 2 = {c, E )1=1-
# LATENTS(N)
LATENT DIM(D)
PARAMS
PSNR
ACC (%)
1600
1600
22.69
53,21
400
1600
29. 14
64.98
178
1602
73.54
16
100
1600
39.93
77.09
D.2 ABLATION ON GAUSSIAN SPATIAL WINDOWING AND KNN APPROXIMATION
To evaluate the impact of Gaussian spatial windowing (GSW) and the k-Nearest Neighbors (kNN) approximation in the proposed method, trained four models on the CIFARIO dataset: one with both features disabled, one with only kNN enabled, one with only GSW enabled, and one with both enabled. Besides evaluating the difference in reconstruction capabilities, we mainly interested in the downstream performance. We argue that the introduced locality enhances latent-space structure by improving veight-sharingg across ooca-pptchhss.
After training the models, we used the different ENF models to generate latent representations for CIFARIO classification. The results are shown Table 9 While reconstruction performance re- mains almost consistent across the different setups, GWS significantly improves downstream classi- fication accuracy. Moreover, the kNN approximation does not negatively affect either reconstruction nor classification performance. Interestingly, kNN even provides a slight improvement even with- out GWS. We hypothesize that this improvement comes from kNN introducing an implicit form of windowing not by modifyyng attention values directly but by limiting the set of attention values considered. To conclude, there can be observed that the introduced locality in CNF latents does improve the downstream performance.
Table 9: Reconstruction PSNR (dbf) and ACC (%1) on CIFAR10 to ablate the Gaussian spatial windowing and kNN approximation.
ENF SETUPS
PSNR (DR
ACC (% f)
ENF
39.1
70.1
ENF KNN
40.8
72.8
ENF + GWS
TBD
TBD
ENF GWS LNN
D.3 TRANSFORMING THE LATENT POINT-CLOUD
We provide visualizations for transformations applied to the latent pointclouds for different bi- invariants a Fig.
D.4 ENF WITH GEOMETRY-FREE LATENT SETS
To further investigate what design choices the performance of ENF results from, we provide an ablation 10 over geometry-free implementation of ENF, We do this by removing the pose information from the latent set, i.e. we set 2 := [ and use a "bi-invariant" that is only function of 2y, a,jj Zy. Since now latents do not have position, windowing and KNN approximation, but keep the rest of the ENF architecture as well as the hyper- parameters used identical to the settings reported in Appx. C under "CIFARIA 10 reconstruction and
remove the Gaussian
19

---

Published as a conference paper at ICLR 2025
Ground Truth
Reconstruction ((;2))
Rotated for Co.3)
Trandated (x)
[(a)
(a)
(b)
(c)
Figure 10: Transformations applied to latents for different bi-invariants (a) am is not bi- invariant to any transformations, (b) a is bi-invariant to translations, producing distorted patterns on rotation and (c) an.i rotations and translations applied to z.
SB(2) is bi-invariant to toto-translation:; the output fo(2; equivaries with both
classification'. We observe highly unstable training during the reconstruction phase, and reconstruc- tion performance on the test set converges to 22.3. We think this attributable to the fact that now any update to one of the latent codes affects the output of the NEF globally, leading complex optimization landscape. This highlights another advantage of either having a single global latent, using locality inductive bias; optimization of single or locally responsible latents seems to lead to a simpler optimization landscape compared to optimizing a set of global latents.
much more
We subsequently train a simple 4 layer transformer with hidden dim 256 and 4 heads as classifier. Note that this transformer uses no positional encoding, since the latent 2 in this setting has no associated geometry/positional information. We train for 500 epochs the augmented dataset, after which training accuracy has converged to 95%. We observed overfitting early into training. Utilizing early stopping, best performance was achieved after just epochs, yielding 0.47. These observations (Tab. 10 are in line with the outcome of our other experiments; geometry- grounded latents are more informative for downstream tasks.
accuracy of
Table 10: Reconstruction PSNR (dbf) and classification test accuracy (%+) CIFAR10 when ablating over latent geometry.
PSNR
ACC (%)
Functa
38.1
68.3
ENF w/ posc-free latents
22.3
47.9
ENP w/ g latents
42.2
82.1
D.5 DETAILS ON COMPUTATIONAL EFFICIENCY
To allow for more fine- mneignai comparison of our method with previous work, we provide details on time and memory efficiency of our approach on the CIFARIO classification experiments listed in Tab. with a²² when both Functa and ENF are fit using MAML with inner loop steps. Moreover, we compare efficiency also when ablating over the KNN approximation of the attention operation. We report estimated FLOPs (obtained through JAX's AOT api), GPU memory usage per sample and training time per epoch for a batch size of 32. We see (Tab. IT] that the naive implementation that does not truncate the attention operator is significantly more PLOP-intenste and memory intensive compared to Functa (Dupont et al. 2022) and the KNN approximate implementation. Functa in all settings does have considerably higher runtime, attributable to its relatively deep sequential archi- tecture compared to the shallow single layer architecture of ENF. These results show that, besides
20

---

Published as a conference paper at ICLR 2025
being more performant fine ine-grained downstream tasks, ENF also scales favourably compared to Functa.
Table II: Computational efficiency of ENF with and without KNN approximation to the Functa baseline for the CIFARIO experiment.
FI FLOPS (x10")
GPU MEMORY (GR/SAMPLE)
TIME PER EPOCH (S)
Functa ENF w/ KNN approx.
104.5
1.83
1801
ENF w KNN approx.
22.7
0.40
207
D.6 ADDITIONAL. SHAPENET-PARE SEGMENTATION RESULTS
Below we show the full table with ShapeNet- Part segmentation results with IoUs per class in table
12 and some qualitative examples in figure 11
Figure 11: Qualitative examples drawn randomly from the ShapeNet segmentation
Table 12: Segmentation class and instance averaged IOU (f) on ShapeNet, and mloUs per class.
INS MtoU
AIRPLANE
CHATE
MOO
Mocker
TABLE
PoinNee
X3.1
67.01
95.1
PointNec++
Functa 92.8
74.8
82.1
12.5
40.2
72.5
60.9
80.1
93.9
63.8
90.6
463
76.0
ENF
32.2
75.4
8007
22.2
43.1
73.2
87.7
64.4
89.4
79.6
30.6
938
62.
91.8
76.9
52.5
80.6
D.6.1 SHAPENET- PART SEGMENTATION WITHOUT SHAPE INFORMATION
Further investigating the results obtained in the ShapeNet Part classification task, we train an ENF fex, without conditioning on reom i.c. without any shape-specific conditioning but instead only conditioning on the object class. This model obtains class and instance mloU of 64.3 and 69.2 respectively, indicating that a lot of points in this dataset can be correctly segmented purely based on their absolute position, and as such the backbone NeF model does not need capture to perform decently on this dataset though we would expect additional geometric to help with performance.
21


Reviews:
[{"review_id": "uNjxp1uq5w", "reviewer": "Reviewer_4oVU", "summary": "This paper proposes equivariant conditional neural fields based on steerable networks. Architecture-wise, this paper proposes equivariant  cross-attention layers with Gaussian windowing as the basis of their Equivariant Neural Fields (ENF). The ENF is trained with a two-stage process: in the first stage, the ENF backbone takes in an input signal and outputs a latent point cloud of (pose, context) pairs. Downstream tasks can be accomplished by training a decoder which takes the latent point cloud as input. Experiments are performed on 2D image reconstruction and classification, 3D reconstruction, classification, and part-segmentation, flood map segmentation, and climate forecasting.", "questions": "1. Does it make sense to compare against Inr2Array [1]?\n2. Should NF2vec also be a baseline for the shape classification task (Table 2)?\n3. Can Functa be trained with a cross-attention-based architecture, similar to that proposed for ENF?\n4. For Functa baselines on downstream tasks such as classification, what was the architecture of the decoders used?\n\n[1]: Zhou, Allan, et al. \"Neural functional transformers.\" Advances in neural information processing systems 36 (2024).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper creates a novel equivariant neural field based on the notion of steerability from equivariant networks, which has the advantages of weight-sharing, locality, and geometric interpretability.", "weaknesses": "1. The original Functa paper uses a SIREN neural field architecture but this paper uses an attention-based neural network architecture. This seems like a potentially unfair comparison. \n2. Another weakness of this paper is that there is no way to decide ahead of time whether to train the latent point cloud using MAML or autodecoding. \n3. The only baseline is Functa for most experiments. Is it possible to use NF2vec in Table 2 and Inr2Array [1] for any of the experiments involving downstream tasks? For tasks involving generalization, \n4. ENF performs only comparably to to the baselines on part-segmentation (Table 3), and some experiments (Table 2) don't show the effectiveness of using equivariance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JzwNiHvPUt", "reviewer": "Reviewer_AKXe", "summary": "The paper introduces the Equivariant Neural Fields, a variant of conditional neural field that uses a geometry-informed cross-attention to condition the NeF using geometrical point cloud representation. The method was validated using a variety of applications, including classification, segmentation, forecasting, and reconstruction.", "questions": "Please refer to the weakness section", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "***Clear and Professional Presentation***: The paper is well-written, structured effectively, and easy to follow. Its clear motivation, logical organization, and high-quality visualizations contribute to a polished and professional presentation, making the methodology accessible and engaging.\n\n***Introduction of Equivariant Neural Fields Model***: The authors propose a novel model, Equivariant Neural Fields, which combines conditional neural fields with point cloud conditioning and equivariant decoding from latent space to field. This approach creatively integrates Neural Fields with equivariant models designed for point clouds, expanding on existing techniques. Additionally, the paper introduces specialized attention layers and engineering optimizations that enhance the model's efficiency, showcasing an innovative blend of established methods.\n\n***Comprehensive Experimental Validation***: The method is rigorously tested across a wide range of use cases and downstream tasks spanning various domains. This extensive evaluation demonstrates the versatility and potential real-world applicability of the proposed approach, supporting its robustness and utility across diverse applications.", "weaknesses": "***High Time Complexity***: The proposed approach appears to be computationally intensive. It would be beneficial for the authors to compare the training time and memory usage of their method against a reference model, such as the Functa method, to provide a clearer assessment of its efficiency.\n\n***Lack of Ablation Studies***: The paper would benefit from ablation studies to clarify the contributions of key components, such as Gaussian spatial windowing and the k-nearest neighbors (kNN) efficiency trick. These studies would help demonstrate how each element impacts the model’s training efficiency and overall performance.\n\n***Suboptimal Segmentation Performance****: The segmentation results are weaker than those of traditional point cloud segmentation baselines. A deeper investigation and discussion of these performance differences would help in understanding and potentially addressing the gaps in segmentation accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pOH2vX9mt6", "reviewer": "Reviewer_yjcq", "summary": "The paper introduces a novel class of Conditional Neural Fields (CNFs) called Equivariant Neural Fields (ENFs) which aim to address the limitations of CNFs in tasks requiring geometric reasoning. The authors propose a geometry-informed cross-attention mechanism that conditions on a latent point cloud of features, enabling equivariant decoding from the latents to the field of interest. This approach possess a steerability property where transformations in the field and mirrored in the latent space. Further, this approach ensures that the cross-attention attention operators respond similarly regardless of pose allowing for weight sharing over similar local patterns leading to more efficient learning. These claims are backed with experiments that demonstrate the advantages posed by the formulation and show a clear advantage over the baselines that have a geometry-free latent space.", "questions": "- I'm particularly curious about the use of these equivariant neural fields as a general backbone for any neural field based task? Are there any situations where it's not helpful to enforce equivariance especially for vision / PDE-based applications?\n- Have you considered using this methodology in a generative context? I think the localized latent point clouds are a particularly interesting property that could lead to more structured creation.\n- Did you study the sample efficiency of ENFs against other CNF methodologies in tasks such as classification? One would assume that enforcing equivariance should lead to a better sample efficiency throughout all truncations of the training dataset\n- I'm curious about the computational cost of your experiments. Does it have a similar run time to the other baselines that were discussed?\n\nAdditionally, I believe there are a couple of typos that I may have spotted:\n- In the abstract: faitfhully -> faithfulll\n- Also, on line 103, posses needs to be possess?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper introduces a novel and mathematically sound method to incorporating geometric structure to neural fields through the equivariant cross attention. The steerability property is well formulated with proven bi-invariant constraints.\n- The experimental details demonstrate an advantage over methods that do not incorporate such geometry informed structure in the latent space. Additionally, the locality and weight-sharing properties discussed are clearly demonstrated.\n- The paper is well-written providing clear background on the neural fields, and the motivation for the need for enforcing equivariance in neural fields. The diagrams are informative and highlight the key components of the methodology. Highlighing geometry attributes in Section 3 with a blue text color was particularly helpful in aiding understanding", "weaknesses": "- While the motivation to compare against other CNF based approaches is clear, the methodology seems to be restricted to a discussion and comparison to the results reported in functa (Dupont et al.) and other CNF-based methods but do not provide a thorough comparison against other equivariant methods or other state of the art methods. Perhaps a comparison of ENFs against more comparisons would strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2ZJWzxzZUx", "reviewer": "Reviewer_9X4V", "summary": "The paper presents a method for conditioning a neural field using a set of $SE(n)$ equivariant local latents. The aim is to enhance downstream task performance by operating on the neural field’s learned latent representation, rather than on discrete samples from the continuous signal as in conventional approaches. It outlines the necessary conditions for an equivariant latent representation in neural fields and adapts a cross-attention architecture to support these conditions. The approach is evaluated across a wide variety of tasks.", "questions": "I would appreciate a response regarding the weaknesses and questions mentioned above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and organized, with a clearly defined method supported by formal definitions.\n\nThe proposed solution is simple and intuitive for enhancing CNFs with local equivariant features.\n\nI appreciate the variety of dataset types used in the experiments.", "weaknesses": "Lack of motivation for using CNF latent encodings in downstream tasks\n\nThe paper does not explicitly discuss the motivation for using latent encodings of CNFs for downstream tasks. It seems that one advantage might be the ability to utilize more data for training since the reconstruction training stage does not require labeled data. This raises a follow-up question:\nWhat benefit do latent features learned through continuous reconstruction (decoder) have over latent features learned through reconstructing a discrete sample?  It seems that a continuous decoder could enable the learning of discretization-agnostic features.  Is there another motivation for using latent encodings of CNFs for downstream tasks?  The purpose behind their use in enhancing downstream tasks remains somewhat unclear.\n\nThe motivation for using local CNF latent encodings could be framed more clearly.\n\nThe paper states that a notable limitation for conventional CNF (ln 51): “each field is encoded by a global variable”. However, this statement about cnfs limitations seems to be partially accurate. In fact, this approach for latent space modeling also has some clear advantages. For example, interpolating between two latents to generate novel signals is far more natural with a global latent structure, whereas a local latent structure requires solving the complex problem of finding correspondences between latent points. Thus, the characterization of a tradeoff rather than a limitation may be more appropriate.\nAdditionally, to address the limitations of a global latent, why not employ an encoder-decoder architecture with gradually decreasing spatial dependency in the latent representation (similar to a UNet)? This approach would provide a final latent that incorporates both local and global information. The rationale for restricting the model to an auto-decoder-style architecture remains unclear.\n\nOverclaiming on Geometry-Appearance Separation in Neural Fields\n\nThe paper claims that the proposed method “separates geometry from appearance” in its representation. My understanding is that this refers to the structure of pose-appearance tuples in the latent space. However, how does the method ensure that only appearance information is captured in  $c_i$ ? This seems to rely solely on  $c_i$  being an $SE(n)$ invariant feature. Yet, some relevant geometric features are also invariant (e.g., shape volume), while some equivariant features can relate to appearance (e.g., how an object’s appearance changes are affected by material reflectance features under rotation). Consequently, enforcing a latent structure of invariant and equivariant features may not be sufficient to achieve true separation. Is there empirical evidence to support the above claim about separation?\n\nUnclear reconstruction results\n\nThe paper claims: “Results show that ours as well as the baseline models struggle with accurately reconstructing the underlying shape from the SDF point clouds”. Given the inaccuracies in reconstruction, how can the learned features be effectively used for downstream tasks? Additionally, it’s unclear why this model underperforms in reconstruction compared to [3]. Both architectures appear similar (apart from the equivariant features), yet [3] reports more accurate reconstruction results.\n\nUnclear segmentation results\n\nThe choice of ShapeNet as the dataset for segmentation evaluation is questionable, as it is an aligned dataset (line 468). A better alternative might be to use non-aligned datasets, such as those used for human-body segmentation in [4] and [5]. Another option would be to unalign ShapeNet by applying a random $SE(3)$ transformation to each data point. Additionally, it’s unclear if the point cloud-specific architectures were also trained with a reconstruction pretext stage. \n\nAdditional comments.\n\nFigure 8 is uninformative on its own without comparison to other methods, showcasing some of the proposed method qualitative benefits/limitations. \n\nConditioning with k-nearest neighbors appears to restrict the smoothness of the modeled field to be at most continuous, while the data signals are at least differentiable.\n\nThe Steerability property for CNFs has also been defined and utilized in prior works, such as [1] and [2].\n\n[1] Frame Averaging for Equivariant Shape Space Learning. Matan Atzmon, Koki Nagano, Sanja Fidler, Sameh Khamis, Yaron Lipman.\n\n[2] Vector Neurons: A General Framework for SO(3)-Equivariant Networks. Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, Leonidas Guibas.\n\n[3] Biao Zhang, Jiapeng Tang, Matthias Niessner, and Peter Wonka. 3dshape2vecset: A 3d shape representation for neural fields and generative diffusion models.\n\n[4] Approximately Piecewise E(3) Equivariant Point Networks. Matan Atzmon, Jiahui Huang, Francis Williams, Or Litany.\n\n[5] Generalizing neural human fitting to unseen poses with articulated se (3) equivariance. Haiwen Feng, Peter Kulits, Shichen Liu, Michael J Black, and Victoria Fernandez Abrevaya", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)