Grounding Continuous Representations in Geometry: Equivariant Neural Fields

## Abstract

Abstract Conditional Neural Fields (CNFs) are increasingly being leveraged as continuous signal representations, by associating each data-sample with a latent variable that conditions a shared backbone Neural Field (NeF) to reconstruct the sample. However, existing CNF architectures face limitations when using this latent downstream in tasks requiring fine-grained geometric reasoning, such as classification and segmentation. We posit that this results from lack of explicit modelling of geometric information (e.g. locality in the signal or the orientation of a feature) in the latent space of CNFs. As such, we propose Equivariant Neural Fields (ENFs), a novel CNF architecture which uses a geometry-informed cross-attention to condition the NeF on a geometric variable—a latent point cloud of features—that enables an equivariant decoding from latent to field. We show that this approach induces a steerability property by which both field and latent are grounded in geometry and amenable to transformation laws: if the field transforms, the latent representation transforms accordingly—and vice versa. Crucially, this equivariance relation ensures that the latent is capable of (1) representing geometric patterns faitfhully , allowing for geometric reasoning in latent space, (2) weight-sharing over similar local patterns , allowing for efficient learning of datasets of fields. We validate these main properties in a range of tasks including classification, segmentation, forecasting and reconstruction, showing clear improvement over baselines with a geometry-free latent space. Code available here .

## 1 Introduction

Neural Fields (NeFs) (Xie et al., ) have recently gained prominence in the machine learning community as a novel representation method that models data as continuous functions. These fields, expressed as $f_{\theta}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{c}$, map spatial coordinates—such as pixel locations $x\in\mathbb{R}^{2}$—to a corresponding signal, like RGB values $f_{\theta}(x)\in\mathbb{R}^{3}$, with $\theta$ representing the model’s parameters. The parameters are optimized to approximate a target signal $f$, ensuring $\forall x:f(x)\approx f_{\theta}(x)$. This capability makes NeFs effective for representing continuous spatial, spatio-temporal, and geometric data, particularly in cases where grid-based methods fall short (Dupont et al., ).

Building on this concept, conditional neural fields (CNFs) introduce a conditioning variable $z\in\mathcal{Z}$ to the model. Given a dataset of $N$ fields $\mathcal{D}=\{f_{i}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{c}\}_{i=1}^{N}$, each specific field can now be represented by a specific conditioning variable $z_{i}$ via $\forall x:f_{i}(x)\approx f_{\theta}(x;z_{i})$, while model weights $\theta$ are shared over the entire dataset. This approach enables CNFs to efficiently model datasets of fields using a set of latent variables that condition a shared backbone NeF. This allows for representing and analysing fields $f_{i}$ by means of their latent representation $z_{i}$, enabling novel approaches for solving tasks involving fields through a framework known as learning with functa (Dupont et al., ). Applications of this include tasks such as classification, segmentation, and the generation of continuous fields (Dupont et al., ; Papa et al., ), as well as continuous PDE forecasting by solving dynamics in the latent space (Yin et al., ; Knigge et al., ).

##### Geometry in CNFs

A notable limitation of conventional CNFs, however, is a lack of explicit geometric interpretability; each field $f_{i}$ is encoded by a ”global” variable $z_{i}$, meaning for instance that any notion of locality or other explicit spatial relationships—which have proven a strong inductive bias in computer vision—is lost. For instance, in classification tasks, spatial organisation of an image’s content is crucial for understanding shape and enabling geometric reasoning (Van Quang et al., ); current neural fields lack such geometric inductive biases, limiting their performance in e.g. classification and generative modelling (Bauer et al., ; Papa et al., ). To this end, we propose equivariant neural fields (ENFs), a new class of NeFs that allows for the identification of continuous fields with concrete geometric representations (Fig. 1).

Figure: Figure 1: Equivariant Neural Fields (ENFs) ground Neural Fields (NeFs) in geometry using a latent point cloud. A latent set $z$ consisting of tuples $(p_{i},\mathbf{c}_{i})$ of pose information $p_{i}$ and context $\mathbf{c}_{i}$ is optimized to reconstruct to the field $f(\cdot)$ as a function $f_{\theta}(\cdot;z)$ using gradient-descent. Due to their explicit positional grounding and locality, the latent retains important geometric features in the input field. The latent $z$ can then be used in downstream tasks, e.g. classification, segmentation, and geometric reasoning, where transformations in the field are mirrored in the latent representation through group actions $L_{g}[f]\sim g\cdot z$.
Refer to caption: extracted/5902079/images/figures/enf-fig-1.png

##### Geometry-grounded neural fields

When the goal is to utilise field representations $z_{i}$ in downstream tasks, it is crucial that these representations capture both textural/appearance information and explicit geometric information. Our approach is inspired by the idea of neural ideograms—learnable geometric representations (Vadgama et al., ; ), and addresses the pervasive issue of texture bias, which causes typical deep learning systems to overfit to textural patterns and ignore important geometric cues (Geirhos et al., ; Hermann et al., ). To address this challenge, we propose defining representations that (1) separate shape/geometry from appearance and (2) are localized in the input signal such that geometric concepts like orientation and distance are maintained from input to latent space. This necessitates that the geometric components of the representations have a meaningful structure, adhering to the same group transformation laws applicable to the fields. Geometrically, this means that distortions in the field translate to corresponding distortions in the latent space, ensuring that geometric (shape) variations are preserved and representable in latent space.

Figure: Figure 2: ENFs preserve transformations through their steerability property; if the field transforms with a group action $g$, the latents transform accordingly via the following group action on the pointcloud; $gz=\{gp_{i},\mathbf{c}_{i}\}^{N}_{i=1}$.
Refer to caption: extracted/5902079/images/figures/steerability.png

To establish an explicit grounding in geometry, we propose modelling the conditioning variables as geometric point clouds $z=\{(p_{i},\mathbf{c}_{i})\}_{i=1}^{N}$, comprising $N$ pose-appearance tuples, with $p_{i}\in G$ being a pose (element) in a group $G$, and $\mathbf{c}_{i}\in\mathbb{R}^{c}$ an appearance vector. This representation space has a well-defined group action, namely $gz=\{(gp_{i},\mathbf{c}_{i})\}_{i=1}^{N}$, allowing us to formalise the notion of grounding a neural field through the following

$$ $\boxed{\text{{Steerability property}:}\;\;\;\;\;\;\;\forall g\in G:f_{\theta}( g^{-1}x;z)=f_{\theta}(x;gz)\,.}$ (1) $$

This property ensures that if the field as a whole transforms, the latent will transform accordingly (Fig. 3).

##### Contributions

With this work we present the following contributions: A new class of geometry-grounded equivariant neural fields that posses

- •
the steerability property and thus proveable generalization over group actions
- •
weight sharing which enables more efficient learning
- •
localized representations in a latent point set which enables unique editing properties

We verify these properties through a range of experiments that (1) support the claim that latents are geometrically meaningful, (2) show competitive reconstruction and representation capacity on segmentation, classification, forecasting and super-resolution tasks on image and shape data.

## 2 Background

##### Neural Fields and conditioning variables

Neural Fields (NeFs) are learned functions $f_{\theta}$ mapping signal coordinates $x$ to signal values $f(x)$, parameterized by a neural network with parameters $\theta$. Due to their flexibility they have emerged as prominent continuous data representation, applied on datatypes varying from object or scene data (Park et al., ; Mescheder et al., ; Sitzmann et al., ; Mildenhall et al., ) to audio and images Tancik et al. (); Sitzmann et al. (). In order to more efficiently represent whole datasets of signals, Conditional Neural Fields only optimise a latent conditioning variable $z^{f}$ per signal-instance $f\in D$ instead of optimising a separate set of neural network parameters $\theta_{i}$. The two most common approaches for obtaining latents $z^{f}$ are autodecoding (Park et al., ) - where latents $z^{f}$ are initialized and optimized alongside NeF parameters $\theta$ - and Meta-Learning based encoding - where optimization is split into an outer loop that optimizes the backbone $\theta$ and an inner loop that optimizes latents $z^{f}$. We explain both in detail in Appx. A.1.1, and use both in experiments described in Sec. 4.

In seminal work by Dupont et al. (), a datatype agnostic approach for learning on these continuous signal representations was proposed - involving first the optimization of a set of conditioning variables $z$ to reconstruct a dataset of signals $\mathcal{D}:=\{f_{j}\}_{j=1}^{n}\sim\{z^{f_{j}}\}_{j=1}^{n}$, and afterward using these variables as a surrogate for the data in downstream tasks such as classification, generation and completion. Although this work highlighted the flexible data-agnostic nature of Conditional Neural Fields (CNFs) by representing signals through a single ”global” condition variable $z^{f_{j}}$, later work by Bauer et al. () showed their limitations in performance for more complex tasks (i.e. involving higher-resolution more varied natural data).

##### Group theoretical preliminaries

The notion of transformation-preservation of an operator - such as the relation between field $f$ and latent $z$ - is best expressed through group theory. A group is an algebraic construction $(G,\cdot)$ defined by a set $G$ and a binary operator $\cdot:G\times G\rightarrow G$ called the group product, satisfying: closure: $\forall h,g\in G:h\cdot g\in G$, identity: $\exists e\in G:\forall g\in G,g\cdot e=g$, inverse: $\forall g\exists g^{-1}\in G:g\cdot g^{-1}=e$ and associativity: $\forall g,h,i\in G:(g\cdot h)\cdot i=g\cdot(h\cdot i)$

Given such a group $G$ with identity element $e\in G$, and a set $X$, we can define the *group action* as a map $G\times X\rightarrow X$, which we will denote with direct multiplication i.e. a group element $g\in G$ action on a coordinate $x\in X$ is denoted $gx$. Note that when $X=G$ the group action equals the group product. For the group action on fields $f:X\rightarrow\mathbb{R}$ we use a separate symbol, namely $[\mathcal{L}_{g}f](x):=f(g^{-1}x)$. In this work we are interested in the Special Euclidean group $SE(n)=T_{n}\rtimes SO(n)$. $SE(n)$ is the roto-translation group consisting of elements $g=(\mathbf{t},\mathbf{R})$ with group operation $g\,g^{\prime}=(\mathbf{t},\mathbf{R})\,(\mathbf{t^{\prime}},\mathbf{R}^{\prime
})=(\mathbf{t}+\mathbf{Rt}^{\prime},\mathbf{RR}^{\prime})$; the left-regular action on function spaces is defined by $\mathcal{L}_{g}f(x)=f(g^{-1}x)=f(\mathbf{R}^{-1}(x-\mathbf{t}))$.

##### Equivariant graph neural networks for downstream tasks.

A key property of our framework is its ability to associate geometric representations with fields. This capability unlocks a rich toolset for field analysis through the lens of geometric deep learning (GDL) (Bronstein et al., ). The GDL field has made significant advancements in the analysis and processing of geometric data, including the study of molecular properties (Batzner et al., ; Batatia et al., ; Gasteiger et al., ; Brandstetter et al., ) and the generation of molecules (Hoogeboom et al., ; Bekkers et al., ) and protein backbones (Corso et al., ; Yim et al., ). In essence, these approaches characterise shape. Our encoding scheme makes these tools now applicable to analysing the geometric components of neural fields, shown in Sec. 4.

Equivariant Graph Neural Networks (EGNNs) are a class of Graph Neural Networks (GNNs) that imposes roto-translational equivariance constraints on their message passing operators to ensure that the learned representations adhere to specific transformation symmetries of the data. Among the various forms of equivariant graph NNs (Thomas et al., ; Brandstetter et al., ; Satorras et al., ; Gasteiger et al., ; Bekkers et al., ; Eijkelboom et al., ; Kofinas et al., ) we will utilise P$\Theta$NITA (Bekkers et al., ) as an equivariant operator to analyse and process our latent point-set representations of fields. For the neural field, we leverage the same optimal bi-invariant attributes as introduced in Bekkers et al. ()—which are based on the theory of homogeneous spaces—to parameterise our neural fields, allowing for seamless integration. They formalise the notion of weight sharing in convolutional networks as the sharing of message functions (kernels) over point-pairs - e.g. relative pixel positions - that should be treated equally. By defining equivalence classes of point-pairs that are identical up to a transformation in the group, we too derive attributes that uniquely identify these classes and enable weight sharing in our proposed ENFs.

## 3 Method

In this section we introduce the Equivariant Neural Field (ENF) architecture. We start Sec. 3.1 by showing how we impose the proposed steerability property (Eq. 1), through the definition of bi-invariant attributes. We construct a cross-attention operator conditioned by these attributes. We subsequently constrain the operator to represent local sub-regions of the input domain by applying a Gaussian window. Finally, we propose a k-nearest neighbouring approach to cope with the computational complexity of the cross-attention operation between pixels and latents. In Sec. 3.2 we finally discuss how we obtain a latent point cloud $z^{f}$ for a signal $f$.

##### Bi-invariance constraint

Before presenting our equivariant neural field design, we need to understand the constraints imposed by the steerability property equation 1. The key result is that for steerability, the field $f_{\theta}$ must be bi-invariant with respect to both coordinates and latents.

###### Lemma 1 .

A conditional neural field satisfies the steerability property iff it is bi-invariant, i.e., $\forall g\in G:\;f_{\theta}(gx;gz)=f_{\theta}(x;z).$

###### Proof.

If $f_{\theta}$ satisfies the steerability property, then $f_{\theta}(gx;gz)=f_{\theta}(g^{-1}gx;z)=f_{\theta}(x;z)$, so it is bi-invariant. Conversely, if $f_{\theta}$ is bi-invariant, then $f_{\theta}(g^{-1}x;z)=f_{\theta}(gg^{-1}x;gz)=f_{\theta}(x;gz)$, satisfying the steerability property equation 1.
∎

Figure: (a)
Refer to caption: extracted/5902079/images/figures/catt-1.png

###### Lemma 1 .

###### Proof.

### 3.1 Equivariant Neural Fields conditioned on Geometric Attributes

The cross-attention operation enables the use of latent-sets as conditioning variables for CNFs by applying cross-attention between embedded coordinates $x_{m}$ and a latent set of context vectors $z{=}\{\mathbf{c}_{i}\}_{i=1}^{N}$ (Zhang et al., ). Such cross-attention fields assign to each $x_{m}$ a corresponding value $f_{\theta}(x_{m};z)$, by matching a coordinate (query) embedding $\mathbf{q}(x_{m})$ against latent (key) vectors $\mathbf{k}(c_{i})$ to obtain attention coefficients $\operatorname{att}_{m,i}$, and aggregating associated values $\mathbf{v}(c_{i})$ via

$$ $\vspace{-1mm}f_{\theta}(x_{m};z)=\mathbf{W}_{o}\sum_{i=1}^{N}\operatorname{att }_{m,i}\mathbf{v}(\mathbf{c}_{i})\;\;\;\;\;\;\;\text{with}\;\;\;\;\;\;\; \operatorname{att}_{m,:}=\underset{i{=}1,...,N}{\operatorname{softmax}}\left( \;\;\frac{\mathbf{q}(x_{m})^{T}\mathbf{k}(\mathbf{c}_{i})}{\sqrt{d_{k}}}\right )\,,\vspace{-1mm}$ $$

where $\mathbf{W}_{o}$ maps cross-attention outputs to NeF output/signal dimension $\mathbb{R}^{c}$. Our desired latent representation contains a geometric component, namely the poses $p_{i}$ associated with the context vectors $\mathbf{c}_{i}$. In order to see how this geometric information could be leveraged, we highlight how each of the three components ($\mathbf{q},\mathbf{k},\mathbf{v}$) could depend on the geometric attributes:

$$ $\vspace{-1mm}f_{\theta}(x_{m};z)=\mathbf{W}_{o}\sum_{i=1}^{N}\operatorname{att }_{m,i}\mathbf{v}({\color[rgb]{0,0.0,1.0}x_{m}},{\color[rgb]{0,0.0,1.0}p_{i}}, \mathbf{c}_{i})\;\;\;\;\;\;\;\text{with}\;\;\;\;\;\;\;\operatorname{att}_{m,:} =\underset{i{=}1,...,N}{\operatorname{softmax}}\left(\;\;\frac{\mathbf{q}({ \color[rgb]{0,0.0,1.0}x_{m}},{\color[rgb]{0,0.0,1.0}p_{i}})^{T}\mathbf{k}({ \color[rgb]{0,0.0,1.0}x_{m}},{\color[rgb]{0,0.0,1.0}p_{i}},\mathbf{c}_{i})}{ \sqrt{d_{k}}}\right)\,.$ $$

The steerability condition demands that the field has to be bi-invariant with respect to transformations on both $x_{m}$ and $p_{i}$, and the easiest way to achieve this is to replace any instance of $x_{m},p_{i}$ by an invariant pair-wise attribute ${\color[rgb]{0,0.0,1.0}\mathbf{a}(x_{m},p_{i})}$ that is both invariant and maximally informative. With maximally informative we mean that coordinate-pose pairs that are not the same up to a group action receive a different vector descriptor, i.e., ${\color[rgb]{0,0.0,1.0}\mathbf{a}(x_{m},p_{i})}={\color[rgb]{0,0.0,1.0}\mathbf
{a}(x_{m}^{\prime},p_{i}^{\prime})}$ if and only if there exists a $g\in G$ such that $x_{m}^{\prime}=gx_{m}$ and $p_{i}^{\prime}=gp_{i}$.

##### Equivariant Neural Fields

Based on recent results in the context of equivariant graph neural networks Bekkers et al. (), we let ${\color[rgb]{0,0.0,1.0}\mathbf{a}(x_{m},p_{i}):=p_{i}^{-1}x}$ be the the unique and complete bijective identifier for the equivalence class of all coordinate-pose pairs that are the same up to a group action. Bijectivity here implies that the descriptor ${\color[rgb]{0,0.0,1.0}p_{i}^{-1}x_{m}}$ contains all information possible to identify the equivalence classes, and thus the use of those attributes leads to maximal expressivity. In this work we use bi-invariants for translation ($\mathbf{a}^{\mathbb{R}^{n}}$) roto-translation ($\mathbf{a}^{\rm SE(2)}$) - as well as a ”bi-invariant” that breaks any equivariance $\mathbf{a}^{\emptyset}$ (see Appx. B for details). We define the ENF as follows:
$\boxed{f_{\theta}(x;z)=\mathbf{W}_{o}\sum_{i=1}^{N}\operatorname{att}_{m,i}
\mathbf{v}({\color[rgb]{0,0.0,1.0}\mathbf{a}(x,p_{i})},\mathbf{c}_{i})\;\;\;\;
\;\;\;\text{with}\;\;\;\;\;\;\;\operatorname{att}_{m,:}=\underset{i=1,...,N}{
\operatorname{softmax}}\left(\;\;\frac{\mathbf{q}({\color[rgb]{0,0.0,1.0}
\mathbf{a}(x,p_{i})})^{T}\mathbf{k}(\mathbf{c}_{i})}{\sqrt{d_{k}}}\right)\,.}$
As specific parameterizations for $\mathbf{a},\mathbf{q},\mathbf{k},\mathbf{v},$ we choose:

$$ $\displaystyle\mathbf{a}(x,p_{i})$ $\displaystyle:=\phi(p_{i}^{-1}x)$ (2) $\displaystyle\mathbf{q}(\mathbf{a}(p_{i}^{-1}x))$ $\displaystyle:=\mathbf{W}_{q}{\color[rgb]{0,0.0,1.0}\mathbf{a}(x,p_{i})}\;\;\; \;\;\;\;\mathbf{k}(\mathbf{c}_{i}):=\mathbf{W}_{k}\mathbf{c}_{i}\,,$ (3) $\displaystyle\mathbf{v}({\color[rgb]{0,0.0,1.0}\mathbf{a}(x,p_{i})},\mathbf{c} _{i})$ $\displaystyle:=(\mathbf{W}_{v}\mathbf{c}_{i})\odot(\mathbf{W}_{a_{\gamma}}{ \color[rgb]{0,0.0,1.0}\mathbf{a}(x,p_{i})})+(\mathbf{W}_{a_{\beta}}{\color[rgb ]{0,0.0,1.0}\mathbf{a}(x,p_{i})})$ (4) $$

with $\odot$ denoting element-wise multiplication and $\phi$ a relative coordinate embedding function which we set to be a Gaussian RFF embedding (Tancik et al., ). In neural field literature it is known that neural networks suffer from high spectral biases (Rahaman et al., ). Due to smooth input-output mappings it becomes difficult to learn high-frequency information in low-dimensions such as the coordinate inputs for a NeF. Gaussian RFF embeddings introduce high-frequency signals in the embedding to alleviate the spectral bias.

Since the value transform has as goal to fill in spatially varying signal patches during reconstruction, the value-function is also conditioned on the geometric attributes $p^{-1}x$. To add extra expressivity we chose to apply the conditioning via FiLM modulation (Perez et al., ) which applies a feature-wise linear modulation with a learnable shift $\beta$ and scale $\gamma$ modulation.

A crucial difference with standard transformer-type methods on point clouds is that cross-attention is between relative position embeddings—relative to the latent pose $p_{i}$—and that the value transform is of depth-wise separable convolutional form (Chollet, ; Bekkers et al., ), which is a stronger form of conditioning (Koishekenov & Bekkers, ) than additive modulation as is typically done in biased self-attention networks such as Point Transformer (Zhao et al., ).

##### Enforcing and learning locality in the latent point cloud

With the current proposed setup, cross-attention is universally applied across the entire set of latents and coordinates. Given the Softmax distribution, each coordinate indiscriminately receives a nonzero attention value for every latent $(p_{i},\mathbf{c}_{i})\in z$. Consequently, although latents possess inherent latent space positional attributes, they do not strictly represent localised regions of the signal, breaking the locality we require.

Figure: (a)
Refer to caption: extracted/5902079/images/figures/catt-gwin-0.png

To address this issue, we modify the attention mechanism by incorporating a Gaussian spatial windowing function with parameter $\sigma_{\text{att}}$ into the computation of attention coefficients. This approach follows the strategy proposed by Cordonnier et al. (). Specifically, the attention scores derived from the dot product between the query and key values are modulated by a Gaussian window $\mu$, which is dependent on the Euclidean distance between latent positions $p^{pos}$ and the input coordinates, expressed as $\mu_{\sigma}(x_{m},p_{i})=-\sigma_{\text{att}}||p^{pos}_{i}-x_{m}||^{2}$. Here, $\sigma_{\text{att}}$ is a hyperparameter that regulates the size for each latent (Fig. 5). We have:

$$ $att(x,z)=\underset{i=1,...,N}{\operatorname{softmax}}\left(\frac{\mathbf{q}({ \color[rgb]{0,0.0,1.0}\mathbf{a}(x,p_{i})})^{T}\mathbf{k}(\mathbf{c}_{i})}{ \sqrt{d_{k}}}+\mu_{\sigma}({\color[rgb]{0,0.0,1.0}x},{\color[rgb]{0,0.0,1.0}p_ {i}})\right),$ (5) $$

where $\mu(x_{m},p_{i})$ represents the Gaussian window computed for each latent position. To enhance the expressiveness of the model even further, $\sigma_{att}$ can be made latent-specific, encoding for the spatial extent of a latent $z_{i}$. This extension allows the latents to be expressed as point clouds: $z^{f}=\{(p_{i},\mathbf{c_{i}},\sigma_{i})\}^{N}_{i=1}$, effectively coupling position, appearance, and locality attributes within the latent space. However, we keep this for further research - fixing $\sigma_{\text{att}}$ in our experiments.

##### A note on efficiency

A limitation of the proposed method is the considerable computational complexity required to calculate output values for large numbers of input coordinates; larger more complex signals require a larger number of latents to be represented accurately leading to an exponential number of attention coefficients needing to be calculated each forward pass (complexity scales as $O(N_{\text{latents}}\times N_{\text{coordinates}})$). Since we localize latents, larger input domains present a trade-off. Either we maintain a small number of latent points—requiring a larger $\sigma_{\text{att}}$ to cover the entire domain, diminishing locality—or we increase the number of latent points, aggravating computational cost.

To mitigate the computational overhead associated with the cross-attention between the latent points and the sampled coordinates, we propose employing a k-nearest neighbours (k-NN) approach for the cross-attention operation. Specifically, for each pixel, we first identify its k-nearest latent points and then compute cross-attention for this coordinate only over these $k$ nearest latents. This approach reduces computational cost while preserving the advantages of localized representations.

Figure: Figure 6: Transformations applied to latents $z$ for different bi-invariants $\mathbf{a}$. (a) $\mathbf{a}_{m,i}^{\emptyset}$ is not bi-invariant to any transformations, (b) $\mathbf{a}^{\mathbb{R}^{2}}_{m,i}$ is bi-invariant to translations, producing distorted patterns on rotation and (c) $\mathbf{a}_{m,i}^{\rm SE(2)}$ is bi-invariant to roto-translations; the output $f_{\theta}(x;z)$ equivaries with both rotations and translations applied to $z$.
Refer to caption: extracted/5902079/images/results/weight-sharing.png

##### Properties of Equivariant Neural Fields

Weight sharing - Conditioning ENF’s attention operator on attributes ${\color[rgb]{0,0.0,1.0}\mathbf{a}}$ which uniquely identify equivalence classes of (latent-coordinate)-pairs, ensures that the cross-attention operators—be it the attention logits or the value transform—respond similarly regardless of the pose under which a signal pattern presents itself. This form of weight-sharing has shown improved data and representation efficiency in GNN-based architectures (Bekkers et al., ) and—as such—we hypothesize that our proposed ENF architecture similarly benefits from these properties compared to other types of NeFs. We confirm this property in figure 7, which shows that ENFs share weights over group actions $g\in G$ for the geometry in which the point clouds are grounded. In Sec. 4 we verify these benefits on downstream tasks.

Locality and (geometric) interpretability- The use of latent point clouds allows for localization of the cross-attention mechanisms around the latent pose, akin to how a convolution operator works with localised kernels. Not only does this improve interpretability and downstream performance (field patterns can be attributed to specific latent points), it enables unique field editing possibilities. Since our method is based on sets of latents and each element is responsible for a local neighbourhood of the input domain, we can take arbitrary unions(e.g. stitching) or intersections(e.g. latent-merging) of point-sets of different samples (Fig. 7).

### 3.2 Obtaining latent point clouds z 𝑧 z italic_z

Following Park et al. (); Dupont et al. () we obtain a latent point cloud $z^{f}$ for a specific sample $f$ using gradient descent, optimizing $z^{f}$ for reconstruction of the original signal $f$. For instance, in images a latent point cloud $z$ may be optimized for an $L_{2}$-loss between $f_{\theta}(\cdot,z)$ and $f(\cdot)$. Of course, this also requires optimizing $\theta$ to meaningfully map latents to fields. The two most common approaches to this end are Autodecoding (Park et al., ) - where $z^{f}$ and $\theta$ are optimised simultaneously over a dataset, or MAML (Finn et al., ; Tancik et al., ) - where optimization is split into an outer and inner loop, with $\theta$ being optimized in the outer loop and $z$ being re-initialized every outer step to reconstruct the current signal batch in a limited number of SGD steps in the inner loop. We detail these approaches in Appx. A.1.1, using both in the experiments.

## 4 Experiments

First, we show the ability of Equivariant Neural Fields (ENFs) to reconstruct datasets of input fields $f$ - i.e. to associate a latent point cloud $z^{f_{j}}$ to a given dataset of samples ${f_{j}}\in D$ that accurately reconstructs them. Then, we validate ENFs as an improved NeF-based downstream representation for various tasks requiring geometric reasoning; classification, segmentation and forecasting. To show the flexibility of NeF-based representations, we perform these tasks on a range of modalities.

Each downstream experiment consists of two stages: (1) fitting a ENF backbone $f_{\theta}$ for reconstruction of the input fields $f_{j}\in D$ to obtain latents pointclouds $z^{f_{j}}$, and (2) training a downstream model—that takes $z^{f_{j}}$ as input—for each specific task. The bi-invariant $\mathbf{a}_{m,i}$ we choose to condition our ENF, as well as the downstream model, varies depending one the type of task we’re performing—described per experiment below. Since our goal is to assess the impact of grounding continuous representations in geometry, besides dataset-specific baselines, we also compare against Functa (Dupont et al., ), the framework that originally proposed functional representations $z^{f_{j}}$ as data surrogates.

For experimental details and hyperparameters we refer to Appx. C. Code for all experiments made available shortly, an implementation of ENFs may be found at [this link](https://github.com/david-knigge/enf-min-jax).

### 4.1 Reconstruction capacity

First we evaluate our proposed ENFs on their reconstruction capabilities. •  Image data We show results for reconstruction trained with Meta-Learning on CIFAR10 Krizhevsky et al. (), CelebA$64{\times}64$ (Liu et al., ) and ImageNet1K (Deng et al., ) using different bi-invariant attributes $\mathbf{a}_{m,i}$—resulting in equivariance to different corresponding transformation groups—in Tab. 2. We provide results for a Functa baseline, following the setup described in (Dupont et al., ). Notably, translational weight sharing ($\mathbf{a}^{\mathbb{R}^{2}}$) outperforms settings with no-transformation ($\mathbf{a}^{\emptyset}$) and roto-translational weight sharing ($\mathbf{a}^{\rm SE(2)}$). Moreover, it seems that locality alone is itself a useful inductive bias when moving to higher resolution, more varied images; Functa outperforms ENF $\mathbf{a}^{\emptyset}$ on CIFAR10 reconstruction, but on CelebA and ImageNet all ENF parameterizations outperform Functa. These results reinforce locality and equivariance as inductive biases in image-based continuous reconstruction tasks.
•  Shape data We show shape reconstruction results (Tab. 2) on two common shape representations; voxels (3D occupancy grids) and meshes. For voxel data we take train and test splits from the 16-class ShapeNet-Part segmentation dataset (Yi et al., ) (which we denote ShapeNet16) and fit their corresponding voxel-based representation as occupancy function $\mathbb{R}^{3}{\rightarrow}\{0,1\}$ - we use this dataset in our shape segmentation experiments detailed below. Train and test splits consists of 11.827 and 2.785 samples. For mesh data we opt instead to fit the full 55-class ShapeNetCore (v2) object dataset (Chang et al., ) (which we denote ShapeNet55) with 42.472 and 5.000 samples in train and test splits respectively. we follow Sitzmann et al. () and fit signed-distance functions (SDFs) $\mathbb{R}^{3}{\rightarrow}\mathbb{R}$, sampling 150k points per shape and calculating their signed distance to the mesh. Unlike Dupont et al. (), we found that we were unable to get reconstructions of sufficient quality with meta-learning and instead obtain latents $z$ using autodecoding on these datasets - possibly hinting at a limitation in expressivity of meta-learning for our shallow model architecture. Results show that ours as well as the baseline models struggle with accurately reconstructing the underlying shape from the SDF point clouds, we think due to the more complex optimization objective.

**Table 1: Test-set reconstruction PSNR (db$\uparrow$) on CIFAR10, CelebA64x64, ImageNet1k, test accuracy (%$\uparrow$) on CIFAR10.**
|  | Cifar10 | CelebA | ImageNet |  |
| --- | --- | --- | --- | --- |
| Task | Recon. | Class. | Recon. | Recon. |
| Functa | 38.1 | 68.3 | 28.0 | 7.2 |
| ENF $\mathbf{a}^{\emptyset}$ | 36.5 | 68.7 | 30.6 | 24.7 |
| ENF $\mathbf{a}^{\mathbb{R}^{2}}$ | $\bm{42.2}$ | 82.1 | $\bm{34.6}$ | 27.5 |
| ENF $\mathbf{a}^{\rm SE(2)}$ | 41.6 | 81.5 | 32.9 | 26.8 |

### 4.2 Downstream tasks

##### Image classification

One major limitation of Functa—noted by Bauer et al. ()—was lacking performance on complex image tasks such as classification. To show performance of our model in this setting, we reproduce the CIFAR10 classification experiment listed in Bauer et al. ()—augmenting CIFAR10 with 50 random crops and flips per image, and training an ENF to reconstruct these using meta-learning, obtaining latent point clouds $z$. We do this for different bi-invariants $\mathbf{a}$ corresponding to no equivariance ($\mathbf{a}^{\emptyset}$), translational ($\mathbf{a}^{\mathbb{R}^{2}}$) and roto-translational ($\mathbf{a}^{\rm SE(2)}$) equivariance. We then train a P$\Theta$NITA classifier (Bekkers et al., ) to classify these latent point clouds - conditioning the message passing function on the same bi-invariants, now calculated between poses $p_{i}$. Results (Tab. 2) show a test-accuracy improvement of $13.8$ percentage points ($68.7\%{\rightarrow}82.1\%$) over Functa (Dupont et al., ), and also indicate that in this setting (roto-)translational equivariance is a strong inductive bias—with $\mathbf{a}^{\rm SE(2)}$, $\mathbf{a}^{\mathbb{R}^{2}}$-ENFs outperforming $\mathbf{a}^{\emptyset}$-ENFs.

##### Shape classification

Highlighting the flexibility of NeF-representations we apply the same setup to shape classification, training P$\Theta$NITA classifiers on the aforementioned ShapeNet16 dataset. Results in Tab. 2 show again that relevant geometric features are better preserved in a localized latent space. We also note that the performance difference between equivariant ($\mathbf{a}^{\mathbb{R}^{3}}$) and non-equivariant ($\mathbf{a}^{\emptyset}$) ENF parameterizations are negligible. We attribute this to the fact that ShapeNet’s models are centered and aligned along the same orientation, i.e. no global transformation symmetries are present in the data for the ENF to leverage.

Figure: Figure 8: Qualitative examples drawn randomly from the ShapeNet segmentation test set.
Refer to caption: extracted/5902079/images/results/shapenet-seg.png

##### ShapeNet-Part segmentation

Where classification primarily evaluates how well the latent point clouds captures global information, we also want to evaluate ENFs performance on fine-grained tasks. As such, we evaluate on the ShapeNet part segmentation task (Yi et al., ). The ShapeNet-Part dataset consists of point clouds for 16 ShapeNet object-classes, each with a varying number of annotated parts for a total of 50 segmentation classes. We use the ENF $\mathbf{a}^{\mathbb{R}^{3}}$ backbone defined in the voxel reconstruction task above ($f_{\theta_{\text{recon}}}$) to obtain a latent $z^{\text{recon}}$ for a shape. Then, a second ENF ($f_{\theta_{\text{seg}}}$) is trained to map points on this shape to a one-hot encoding of their corresponding segmentation classes, i.e. $f_{\theta_{\text{seg}}}(x_{m};z^{\text{recon}})$ maps a point $x_{m}$ to its segmentation label $y_{m}$. We report mean class IoU and mean instance IoU in Tab. 8, and mIoU per class. We include dense segmentations sampled from the underlying ENF in Fig. 8. Here, somewhat surprisingly, results show ENF and Functa perform comparably in this task. Again, we think this

Figure: Figure 9: Top- OMBRIA test sample of SAR (S1), optical (S2) before and after flooding with ground truth flood map. Bottom- ENF reconstructions $f_{\theta_{\text{recon}}}(\cdot;z^{\text{recon}}$) and predicted mask $f_{\theta_{\text{seg}}}(\cdot;z^{\text{recon}})$.
Refer to caption: extracted/5902079/images/results/ombria.png

attributable to the fact that all shapes are aligned and centered - we further investigate these results in Appx. D.2. We additionally include results for point cloud-specific architectures, and NF2Vec - a framework for self-supervised representation learning on 3D data from (non-conditional) NeFs. These results show that ENF only slightly underperforms modality-specific baselines.

##### Flood Map Segmentation

For a more challenging segmentation task we apply ENFs on multi-modal flood mapping dataset (Drakonakis et al., ). This small dataset (759/85 train/test split) provides dual-modal temporal data; aligned Synthetic Aperture Radar (SAR) and optical satellite images at $256{\times}256$ resolution obtained by satellites Sentinel 1 and 2 (S1,S2), of disaster sites before and after their flooding, along with corresponding masks that segment the flooded area. The goal is to predict binary segmentation mask given these 4 different input fields. We first train a reconstruction $\mathbf{a}^{\rm SE(2)}$-ENF $f_{\theta_{\text{recon}}}$ with MAML to obtain a latent $z^{\text{recon}}$ that decodes into the four observations. Next, a segmentation ENF $f_{\theta_{\text{seg}}}$ is trained to predict, given a latent $z^{\text{recon}}$, the binary mask at each location.

We evaluate and compare our model against the multi-modal U-Net proposed by Drakonakis et al. (). As suggested by Dupont et al. () we trained Functa—unable to fit the training set with MAML—using autodecoding instead. However, we found Functa unable to generalize to test images in this complex low-data setting, collapsing to remembering the

Figure: Table 4: Test IOU ($\uparrow$) for flood map segmentation on OMBRIA, for different observation rates.

training dataset (achieving 31.5 recon PSNR and 93.7 IoU on the $256{\times}256$ train set). Results (Tab. 4, Fig. 9) show the importance of inductive biases in complex limited-data regimes. We provide results for subsampled observations to simulate missing data, showing the robustness of NeF-based methods to sparsity—ENF performs well even at $10\%$ observation rate where classical convolution-based methods fail. Moreover, we show zero-shot resolution transfer results (Tab. 5) where $f_{\theta_{\text{recon}}},f_{\theta_{\text{seg}}}$ are trained on $128{\times}128$ resolution data are deployed on $64\times 64$ and $256\times 256$ resolution data without fine-tuning, showing the resolution agnostic nature of NeF-representations.

Figure: Table 6: ERA5 reconstruction $T_{t}$-MSE$\downarrow$ and 1-hour forecasting $T_{t+1}$-MSE$\downarrow$. *MSE between ground truth observations at $T_{t}$ and $T_{t+1}$.
Refer to caption: extracted/5902079/images/results/era5-gt.png

##### ERA5 Climate forecasting

Following (Yin et al., ; Knigge et al., ) we evaluate our NeF-based representation on dynamics forecasting. ERA5 (Hersbach et al., ) is a dataset of hourly global temperature observations. We use the dataset as described in Dupont et al. (), which contains data defined over $46\times 90$ latitude-longitude grids. From the training and test sets, we extract 5693 and 443 pairs of subsequent observations $T_{t},T_{t+1}$ for train and test sets respectively. Using MAML, we train an ENF $f_{\theta}$ with bi-invariant ${\mathbf{a}^{\emptyset}}$ (no symmetries exist in this data) to reconstruct the global temperature state $T_{t}$ using $z_{t}$, and optimize a P$\Theta$NITA MPNN to predict an update $\Delta z_{t}$ that maps $z_{t}$ to a latent $\hat{z}_{t+1}=z_{t}+\Delta z_{t}$ which decodes into the state at $t+1$, i.e. $T_{t+1}{\approx}f_{\theta}(\cdot;z_{t+1})\approx f_{\theta}(\cdot;z_{t}+\Delta
z
_{t})$. Training is done sequentially, i.e. first the backbone $f_{\theta}$ is optimized and afterward the MPNN is trained, keeping $f_{\theta}$ fixed. Results (Tab. 6, Fig. 10) show that the latent space of ENF lends itself well for modelling such complex dynamics - where a global latent representation such as Functa seems unable to model the relevant fine-grained details needed for forecasting.

## 5 Conclusion

Building upon fascinating work using Neural Fields (NeFs) as continuous data surrogates, this paper introduces Equivariant Neural Fields (ENFs); a novel NeF parameterization that re-introduces inductive biases (locality, equivariance) into NeF-based representations. ENF uses a geometry-grounded conditioning variable—a latent attributed point cloud—to achieve an equivariant decoding process, ensuring that transformations in an input field are preserved in the latent space and enabling steering of the latent to transform the output signal. This steerability property allows for the accurate representation of geometric information, and for efficient weight-sharing over spatially similar patterns, significantly improving learning efficiency and generalization, as validated on a range of experiments with varying data modalities and objectives.

## 6 Reproducibility

All datasets can be downloaded via their cited references without any effort except for the ShapeNet dataset - which requires registration and approval. The pre-processing steps are described in appendix C.3. For all model parameter settings for the ENFs, downstream models or Functa we refer to the appendix C. As supplementary material we added a codebase containing code to reproduce results for the CIFAR10 and OMBRIA experiments. Code for all other experiments will be released during the rebuttal phase of the review process, containing all settings to reproduce the experiments in config files.

## Appendix A Appendix

### A.1 Autodecoding and Meta-Learning

#### A.1.1 Meta-learning

When fitting samples with (Conditional) NeFs using autodecoding (gradient-descent based optimisation of the latent at test-time) (Park et al., ), two key challenges emerge: (1) optimising the sample-specific parameters/latent $z$ for a novel sample can be time-consuming -taking e.g. up to 500 gradient updates (Yin et al., )- and (2) more gradient updates to NeF weights may impede downstream performance through a phenomenon known as overtraining (Papa et al., ) -where the relationship between field $f$ and $z$ is obscured by oversensitivity to high-frequency details. To address the first point, Sitzmann et al. (); Tancik et al. (); Dupont et al. () propose a Model-Agnostic Meta-Learning (MAML) (Finn et al., ) based optimisation method, enabling the network or latent initialisation to be learned such that each sample can be fitted with just a few gradient steps. More specifically, Dupont et al. () proposes an inner-outer loop structure where the modulations are updated in the inner loop, while the base network weights are updated in the outer loop. This method corresponds to an instance of learning a subset of weights with MAML, also known as Contextual Variable Interaction Analysis (CAVIA) Zintgraf et al. (). More recently, Knigge et al. () note that this meta-learning approach also improves downstream performance by imposing structure on the NeF’s latent-space. We provide pseudocode for this approach in Alg. 1.

Figure: Algorithm 1 Meta-learning ENF

#### A.1.2 Autodecoding

During our experiments, we found that not all types of signals lend themselves easily to this encoding approach when using ENFs (specifically SDFs and occupancy functions). Although it saves time in inference and adds structure to the latent space, (Dupont et al., ) also remark on the limited expressivity of Meta-Learning due to the small number of gradient descent steps used to optimize a latent $z$. As such, for all shape experiments we instead opt for autodecoding (Park et al., ), in which latents and backbone are optimized simultaneously. We provide pseudocode for this approach in Alg. 2.

Figure: Algorithm 2 Autodecoding ENF

#### A.1.3 A note on pose initialization

We noted during our experiments that initialization of the latent poses-i.e. their initial position/orientation in the inner loop-has a significant impact on the reconstruction capacity and stability of the ENF. We found that a good way to initialize the latents is to space them as equidistantly as possible and then adding small Gaussian noise ($~{}N(0,1e-3)$), e.g. for 2D images on a perturbed 2D grid. Any orientations are initialized canonically, i.e. all latents are initialized with the same orientation. When defining an equidistantly spaced grid is hard, for example on point clouds or data defined on a sphere, we propose using Farthest Point Sampling on a training grid to initialize positions for the latents.

## Appendix B Bi-invariant function parameterizations 𝐚 m , i subscript 𝐚 𝑚 𝑖 \mathbf{a}_{m,i} bold_a start_POSTSUBSCRIPT italic_m , italic_i end_POSTSUBSCRIPT

The bi-invariants attributes that are used in the experiments section are listed here.

Translational symmetries $\mathbb{R}^{n}$ In this setting, poses correspond to translations $\mathbf{t}_{i}\in\mathbb{R}^{n}$:

$$ $\mathbf{a}^{\mathbb{R}^{n}}_{m,i}=x_{m}-\mathbf{t}_{i}$ (6) $$

Roto-translational symmetries ${\rm SE(2)}$. In this setting, poses $p_{i}$ correspond to group elements $g=(\theta_{i},\mathbf{t}_{i})\in{\rm SE(2)}$. We adopt the invariant attribute introduced by (Bekkers et al., ):

$$ $\mathbf{a}^{\rm SE(2)}_{m,i}=\mathbf{R}_{\theta_{i}}(x_{m}-\mathbf{t}_{i})$ (7) $$

No transformation symmetries. A simple ”bi-invariant” for this setting that preserves all geometric information is given by simply concatenating coordinates $p$ with coordinates $x$:

$$ $\mathbf{a}^{\emptyset}_{i,m}=p_{i}\oplus x_{m}$ (8) $$

Parameterizing the cross-attention operation in Eq. 3.1 as function of this bi-invariant results in a framework without any equivariance constraints. We use this in experiments to ablate over equivariance constraints and its impact on performance.

## Appendix C Experimental details

We provide hyperparameters per experiment. We optimize the weights of the neural field $f_{\theta}$ in all experiments with Adam (Kingma & Ba, ) with a learning rate of 1e-4, and an inner step size of 30.0 for $\mathbf{c}_{i}$ and 1.0 for $p_{i}$ (increasing inner step size in general speeds up convergence and improves reconstruction - but may also lead to instabilities). For downstream classification we train an equivariant MPNN $F_{\psi}$, using 3 message passing layers in the architecture specified in Bekkers et al. () conditioned on the same bi-invariant which was used to fit the ENF, with a hidden dimensionality of 256, always trained with learning rate 1e-4. The std parameters $\sigma_{q},\sigma_{v}$ of the RFF embedding functions $\varphi_{q},\varphi_{v}$ are chosen per experiment based on an ablation. In general, increasing both values leads to increased frequency response of the ENF, though generally the model is more susceptible to small change in $\sigma_{q}$. as well as hidden dim size and the number of attention heads are chosen per experiment, detailed below. We run all experiments on a single H100.

### C.1 Image reconstruction, classification, segmentation, forecasting

##### Cifar10 reconstruction and classification

For Cifar10 (Krizhevsky et al., ) reconstruction and classification we use a hidden dim of 128 with 3 heads, 25 latents of size 64, a batch size of 32 and restrict the cross-attention operator to k=4 nearest latents for each input coordinate $x$. For $\sigma_{q},\sigma_{v}$ we choose 1.0 and 3.0 respectively. We train the model for 100 epochs.

##### CelebA64 × \times × 64

For CelebA Liu et al. () we use a hidden dim of 256, 36 latents of size 64, a batch size of 2 and restrict the cross-attention operator to k=4 nearest latents for each input coordinate $x$. For $\sigma_{q},\sigma_{v}$ we choose 2.0 and 10.0. We train the model for 30 epochs.

##### ImageNet1K reconstruction

For ImageNet1K (Deng et al., ) reconstruction we use a hidden dim of 128 with 3 heads, 169 latents of size 64, a batch size of 2, restricting the cross-attention operator to k=4 nearest latent for each input coordinate $x$. For $\sigma_{q},\sigma_{v}$ we choose 2.0 and 10.0 respectively. We train the model for 2 epochs.

##### Ombria

For OMBRIA Drakonakis et al. () we trained a reconstruction model $f_{\theta_{\text{recon}}}$ with, 256 hidden dim, 4 heads, 169 latents of size 128, a batch size 8. Restricting the cross-attention operator to k=1 nearest latent for each input coordinate $x$. For $\sigma_{q},\sigma_{v}$ we choose 2.0 and 10.0 respectively. The segmentation model $f_{\theta_{\text{seg}}}$ has hidden size of 128, 8 heads, with cross-attention restricted to k=4 nearest latents, trained with batch size 16. For $\sigma_{q},\sigma_{v}$ we choose 2.0 and 3.0 respectively. We train both models, sequentially, for 500 epochs.

##### ERA5 forecasting

For ERA5 forecasting (Hersbach et al., ) we train a reconstruction model $f_{\theta_{\text{recon}}}$ with 128 hidden dim, 3 heads, 36 latents of size 64, a batch size of 32. Restricting the cross-attention operator to k=4 nearest latent for each input coordinate $x$. For $\sigma_{q},\sigma_{v}$ we choose 2.0 and 8.0 respectively. Inputs are defined over a latitude longitude $\theta,\phi$ grid, which we map to 3D euclidean coordinates per $\mathbf{x}=\left[\cos\theta\cos\phi,\cos\theta\sin\phi,\sin\theta\right]$. We first train the ENF for 800 epochs. As forecasting model, we train a P$\Theta$NITA MPNN of 3 layers with 256 hidden dim for 1000 epochs. Both models are trained with a batch size of 32.

As objective, since we don’t want to overfit the reconstruction error incurred by fitting a latent $z_{t}$ to the initial state, we supervise the forecasting model with $L_{2}$ loss between the decoded output for predicted latent $\hat{z}_{t+1}$ per:

$$ $L_{\text{forecast}}=||(f_{\theta_{\text{recon}}}(\cdot;z_{t})+\Delta T)-(f_{ \theta_{\text{recon}}}(\cdot;\hat{z}_{t+1})||_{2}^{2}$ $$

$\Delta T$ being the ground truth change in temperature, and $\hat{z}_{t+1}=F_{\psi_{\text{forecast}}}(z_{t})$.

### C.2 Functa baseline models

For the Functa baselines (Dupont et al., ), we try to keep as close as possible to the setup defined by the original authors. However, we found that training deeper models (¿8 layers) lead to very unstable training. Instead, we opted to go for shallower models, up to 6 layers, with a hidden dim of 512 and latent modulation size of 512 (1024 for ImageNet1K reconstruction). We would like to note here that in all experiments, the Functa baseline has larger parameter count than the ENF models applied to each task (e.g. for CIFAR10, ENF has 522K params where Functa has 2.6M params). Although we did not explore this in-depth, it seems that the proposed ENF representation is more parameter efficient compared to the deep SIREN model defined in (Dupont et al., ).

### C.3 Shape reconstruction and classification

##### Voxel Dataset and Segmentation

The voxels are given with the ShapeNet dataset where the segmentation labels are given as point clouds. However, the coordinate frames of the voxels are different, so to align them we mapped both between -1 and 1.

We trained a model $f_{\theta_{\text{recon}}}$ with a hidden dim of 128, 3 heads, 27 latents of size 32. We set $\sigma_{q},\sigma_{v}$ for the RFF embedding functions $\phi_{q},\phi_{v}$ to 2 and 10 respectively.
For Functa we used, a latent dim of 864 to have the same latent parameters as the conditioning variable used for ENF. As NeF we used a 5-layer Siren with a hidden-dim of 512, $w_{0}$ is set to 10. The modulation network is a two-layer MLP of hidden sizes 256 and 512.

As is customary for ShapeNet-part segmentation, we condition on the object class and supervise over all segmentation classes with a cross-entropy loss, but only calculate test IoU based on segmentation classes that correspond to the object class. We chose the class-emb dim to be 32 for all settings.

The segmentation NeFs are all trained for 500 epochs with the same parameters as the reconstruction NeF. However, for ENF, we chose $\sigma_{q},\sigma_{v}$ to be 1, 1 for extra stability.

##### SDF Dataset

To create the signed distance functions from ShapeNetCore V2 Chang et al. () objects, we took their meshes and made them water-tight using Point Cloud Utils (Williams, ). Afterwards, we sampled a point cloud of 150,000 points from the surface. To create the actual SDF, we perturbed the points with Gaussian noise along the mesh normals, and recalculate the signed distances to the surface. Finally, the dataset consisted of 55 classes with a total number of 42.472 and 5.000 samples for the train and test set respectively.

For ENF we used a latent point cloud of 27 points with context vectors of dimension 32. The std parameters $\sigma_{q},\sigma_{v}$ for the RFF embedding functions $\phi_{q},\phi_{v}$ are 2 and 10 respectively. The hidden dim of the ENF was set to 128 and we used 3 attention-heads. The

For Functa (Dupont et al., ) we used a latent modulation of 864 which corresponds to ENFs chosen 27*32 parameters for the conditioning variable. As a NeF we used a 5 layer Siren with an hidden-dim of 512 and a $w_{0}$ parameter of 15. As a modulation network we used a two-layer MLP with 256,512 hidden-dim.

## Appendix D Additional results

### D.1 Latent point-cloud ablation

In this section, we delve deeper into the hyper-parameters of the latent point clouds used as conditioning variables. Equivariant neural fields can increase the number of parameters used to represent a signal in two ways: by increasing the number of latent points or by expanding the dimensionality of the context vectors. Intuitively, we can either enhance the representational capability of a single region in the input domain or create more, smaller regions with lower representational dimensions.

To gain further insights into how these conditioning variables behave, we train multiple CNFs using different latent point-cloud configurations. In these experiments, the chosen latent dimension or number of latent points is adjusted to keep the total number of parameters as close as possible across configurations. We trained each model for 400 epochs, as only small improvements occurred beyond this point and the overall trend was already clear. After fitting the ENF, we used the meta-learned representation to perform classification tasks. We employed a simple message-passing GNN, which we trained for 20 epochs, after which performance improvements began to degrade. Below, we present the ablation results for these different approaches to increasing representational capabilities.

**Table 7: Reconstruction PSNR (db$\uparrow$) and ACC (%$\uparrow$) on Cifar10 for different parametrisations of the latent point-clouds.**
| # Latent points | Context Dim | # params | PSNR | ACC (%) |
| --- | --- | --- | --- | --- |
| 1 | 1600 | 1600 | 22.69 | 53.21 |
| 4 | 400 | 1600 | 29.14 | 64.98 |
| 9 | 178 | 1602 | 35.49 | 73.54 |
| 16 | 100 | 1600 | 39.93 | 77.09 |

### D.2 ShapeNet-Part segmentation without shape information

Further investigating the results obtained in the ShapeNet Part classification task, we train an ENF $f_{\theta_{\text{seg}}}$ without conditioning on $z^{\text{recon}}$ - i.e. without any shape-specific conditioning but instead only conditioning on the object class. This model obtains class and instance mIoU of 64.3 and 69.2 respectively, indicating that a lot of points in this dataset can be correctly segmented purely based on their absolute position, and as such the backbone NeF model does not need to capture to perform decently on this dataset - though we would expect additional geometric to help with performance.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. Does it make sense to compare against Inr2Array [1]?
2. Should NF2vec also be a baseline for the shape classification task (Table 2)?
3. Can Functa be trained with a cross-attention-based architecture, similar to that proposed for ENF?
4. For Functa baselines on downstream tasks such as classification, what was the architecture of the decoders used?

[1]: Zhou, Allan, et al. "Neural functional transformers." Advances in neural information processing systems 36 (2024).
soundness: 2
strengths: This paper creates a novel equivariant neural field based on the notion of steerability from equivariant networks, which has the advantages of weight-sharing, locality, and geometric interpretability.
summary: This paper proposes equivariant conditional neural fields based on steerable networks. Architecture-wise, this paper proposes equivariant  cross-attention layers with Gaussian windowing as the basis of their Equivariant Neural Fields (ENF). The ENF is trained with a two-stage process: in the first stage, the ENF backbone takes in an input signal and outputs a latent point cloud of (pose, context) pairs. Downstream tasks can be accomplished by training a decoder which takes the latent point cloud as input. Experiments are performed on 2D image reconstruction and classification, 3D reconstruction, classification, and part-segmentation, flood map segmentation, and climate forecasting.
weaknesses: 1. The original Functa paper uses a SIREN neural field architecture but this paper uses an attention-based neural network architecture. This seems like a potentially unfair comparison. 
2. Another weakness of this paper is that there is no way to decide ahead of time whether to train the latent point cloud using MAML or autodecoding. 
3. The only baseline is Functa for most experiments. Is it possible to use NF2vec in Table 2 and Inr2Array [1] for any of the experiments involving downstream tasks? For tasks involving generalization, 
4. ENF performs only comparably to to the baselines on part-segmentation (Table 3), and some experiments (Table 2) don't show the effectiveness of using equivariance.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to the weakness section
soundness: 3
strengths: ***Clear and Professional Presentation***: The paper is well-written, structured effectively, and easy to follow. Its clear motivation, logical organization, and high-quality visualizations contribute to a polished and professional presentation, making the methodology accessible and engaging.

***Introduction of Equivariant Neural Fields Model***: The authors propose a novel model, Equivariant Neural Fields, which combines conditional neural fields with point cloud conditioning and equivariant decoding from latent space to field. This approach creatively integrates Neural Fields with equivariant models designed for point clouds, expanding on existing techniques. Additionally, the paper introduces specialized attention layers and engineering optimizations that enhance the model's efficiency, showcasing an innovative blend of established methods.

***Comprehensive Experimental Validation***: The method is rigorously tested across a wide range of use cases and downstream tasks spanning various domains. This extensive evaluation demonstrates the versatility and potential real-world applicability of the proposed approach, supporting its robustness and utility across diverse applications.
summary: The paper introduces the Equivariant Neural Fields, a variant of conditional neural field that uses a geometry-informed cross-attention to condition the NeF using geometrical point cloud representation. The method was validated using a variety of applications, including classification, segmentation, forecasting, and reconstruction.
weaknesses: ***High Time Complexity***: The proposed approach appears to be computationally intensive. It would be beneficial for the authors to compare the training time and memory usage of their method against a reference model, such as the Functa method, to provide a clearer assessment of its efficiency.

***Lack of Ablation Studies***: The paper would benefit from ablation studies to clarify the contributions of key components, such as Gaussian spatial windowing and the k-nearest neighbors (kNN) efficiency trick. These studies would help demonstrate how each element impacts the model’s training efficiency and overall performance.

***Suboptimal Segmentation Performance****: The segmentation results are weaker than those of traditional point cloud segmentation baselines. A deeper investigation and discussion of these performance differences would help in understanding and potentially addressing the gaps in segmentation accuracy.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 4
questions: - I'm particularly curious about the use of these equivariant neural fields as a general backbone for any neural field based task? Are there any situations where it's not helpful to enforce equivariance especially for vision / PDE-based applications?
- Have you considered using this methodology in a generative context? I think the localized latent point clouds are a particularly interesting property that could lead to more structured creation.
- Did you study the sample efficiency of ENFs against other CNF methodologies in tasks such as classification? One would assume that enforcing equivariance should lead to a better sample efficiency throughout all truncations of the training dataset
- I'm curious about the computational cost of your experiments. Does it have a similar run time to the other baselines that were discussed?

Additionally, I believe there are a couple of typos that I may have spotted:
- In the abstract: faitfhully -> faithfulll
- Also, on line 103, posses needs to be possess?
soundness: 4
strengths: - The paper introduces a novel and mathematically sound method to incorporating geometric structure to neural fields through the equivariant cross attention. The steerability property is well formulated with proven bi-invariant constraints.
- The experimental details demonstrate an advantage over methods that do not incorporate such geometry informed structure in the latent space. Additionally, the locality and weight-sharing properties discussed are clearly demonstrated.
- The paper is well-written providing clear background on the neural fields, and the motivation for the need for enforcing equivariance in neural fields. The diagrams are informative and highlight the key components of the methodology. Highlighing geometry attributes in Section 3 with a blue text color was particularly helpful in aiding understanding
summary: The paper introduces a novel class of Conditional Neural Fields (CNFs) called Equivariant Neural Fields (ENFs) which aim to address the limitations of CNFs in tasks requiring geometric reasoning. The authors propose a geometry-informed cross-attention mechanism that conditions on a latent point cloud of features, enabling equivariant decoding from the latents to the field of interest. This approach possess a steerability property where transformations in the field and mirrored in the latent space. Further, this approach ensures that the cross-attention attention operators respond similarly regardless of pose allowing for weight sharing over similar local patterns leading to more efficient learning. These claims are backed with experiments that demonstrate the advantages posed by the formulation and show a clear advantage over the baselines that have a geometry-free latent space.
weaknesses: - While the motivation to compare against other CNF based approaches is clear, the methodology seems to be restricted to a discussion and comparison to the results reported in functa (Dupont et al.) and other CNF-based methods but do not provide a thorough comparison against other equivariant methods or other state of the art methods. Perhaps a comparison of ENFs against more comparisons would strengthen the paper.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: I would appreciate a response regarding the weaknesses and questions mentioned above.
soundness: 2
strengths: The paper is well-written and organized, with a clearly defined method supported by formal definitions.

The proposed solution is simple and intuitive for enhancing CNFs with local equivariant features.

I appreciate the variety of dataset types used in the experiments.
summary: The paper presents a method for conditioning a neural field using a set of $SE(n)$ equivariant local latents. The aim is to enhance downstream task performance by operating on the neural field’s learned latent representation, rather than on discrete samples from the continuous signal as in conventional approaches. It outlines the necessary conditions for an equivariant latent representation in neural fields and adapts a cross-attention architecture to support these conditions. The approach is evaluated across a wide variety of tasks.
weaknesses: Lack of motivation for using CNF latent encodings in downstream tasks

The paper does not explicitly discuss the motivation for using latent encodings of CNFs for downstream tasks. It seems that one advantage might be the ability to utilize more data for training since the reconstruction training stage does not require labeled data. This raises a follow-up question:
What benefit do latent features learned through continuous reconstruction (decoder) have over latent features learned through reconstructing a discrete sample?  It seems that a continuous decoder could enable the learning of discretization-agnostic features.  Is there another motivation for using latent encodings of CNFs for downstream tasks?  The purpose behind their use in enhancing downstream tasks remains somewhat unclear.

The motivation for using local CNF latent encodings could be framed more clearly.

The paper states that a notable limitation for conventional CNF (ln 51): “each field is encoded by a global variable”. However, this statement about cnfs limitations seems to be partially accurate. In fact, this approach for latent space modeling also has some clear advantages. For example, interpolating between two latents to generate novel signals is far more natural with a global latent structure, whereas a local latent structure requires solving the complex problem of finding correspondences between latent points. Thus, the characterization of a tradeoff rather than a limitation may be more appropriate.
Additionally, to address the limitations of a global latent, why not employ an encoder-decoder architecture with gradually decreasing spatial dependency in the latent representation (similar to a UNet)? This approach would provide a final latent that incorporates both local and global information. The rationale for restricting the model to an auto-decoder-style architecture remains unclear.

Overclaiming on Geometry-Appearance Separation in Neural Fields

The paper claims that the proposed method “separates geometry from appearance” in its representation. My understanding is that this refers to the structure of pose-appearance tuples in the latent space. However, how does the method ensure that only appearance information is captured in  $c_i$ ? This seems to rely solely on  $c_i$  being an $SE(n)$ invariant feature. Yet, some relevant geometric features are also invariant (e.g., shape volume), while some equivariant features can relate to appearance (e.g., how an object’s appearance changes are affected by material reflectance features under rotation). Consequently, enforcing a latent structure of invariant and equivariant features may not be sufficient to achieve true separation. Is there empirical evidence to support the above claim about separation?

Unclear reconstruction results

The paper claims: “Results show that ours as well as the baseline models struggle with accurately reconstructing the underlying shape from the SDF point clouds”. Given the inaccuracies in reconstruction, how can the learned features be effectively used for downstream tasks? Additionally, it’s unclear why this model underperforms in reconstruction compared to [3]. Both architectures appear similar (apart from the equivariant features), yet [3] reports more accurate reconstruction results.

Unclear segmentation results

The choice of ShapeNet as the dataset for segmentation evaluation is questionable, as it is an aligned dataset (line 468). A better alternative might be to use non-aligned datasets, such as those used for human-body segmentation in [4] and [5]. Another option would be to unalign ShapeNet by applying a random $SE(3)$ transformation to each data point. Additionally, it’s unclear if the point cloud-specific architectures were also trained with a reconstruction pretext stage. 

Additional comments.

Figure 8 is uninformative on its own without comparison to other methods, showcasing some of the proposed method qualitative benefits/limitations. 

Conditioning with k-nearest neighbors appears to restrict the smoothness of the modeled field to be at most continuous, while the data signals are at least differentiable.

The Steerability property for CNFs has also been defined and utilized in prior works, such as [1] and [2].

[1] Frame Averaging for Equivariant Shape Space Learning. Matan Atzmon, Koki Nagano, Sanja Fidler, Sameh Khamis, Yaron Lipman.

[2] Vector Neurons: A General Framework for SO(3)-Equivariant Networks. Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, Leonidas Guibas.

[3] Biao Zhang, Jiapeng Tang, Matthias Niessner, and Peter Wonka. 3dshape2vecset: A 3d shape representation for neural fields and generative diffusion models.

[4] Approximately Piecewise E(3) Equivariant Point Networks. Matan Atzmon, Jiahui Huang, Francis Williams, Or Litany.

[5] Generalizing neural human fitting to unseen poses with articulated se (3) equivariance. Haiwen Feng, Peter Kulits, Shichen Liu, Michael J Black, and Victoria Fernandez Abrevaya

Meta-review: This paper proposes a variant of conditional neural fields by utilizing a latent point cloud rather than a global latent for a neural field. The authors claimed locality and steerability for the resulting Equivariant Neural Fields and demonstrated the effectiveness of latent for many tasks including perception and generative modeling. This paper is well-written as acknowledged by multiple reviewers. The implementation of equivariance through cross-attention between queries and latents is simple and elegant. Experiments clearly demonstrated the usefulness of geometry equivariance for representation learning.
The downside of the proposed method is the extra computational complexity of quadratic cross-attention between queries and multiple latents in a neural field. The authors have proposed a based approximation to full attention operation.
All four reviewers gave positive ratings for this paper. The AC recommends acceptance of this paper due to the novelty in conditioning neural fields on latent point cloud.

Decision: Accept (Poster)