Grounding Continuous Representations in Geometry: Equivariant Neural Fields

Authors: David Wessels, David M Knigge, Riccardo Valperga, Samuele Papa, Sharvaree Vadgama, Efstratios Gavves, Erik J Bekkers
Primary area: learning on graphs and other geometries & topologies
Keywords: Geometric Deep Learning, Neural Fields, Equivariance, Representation Learning, Latent Point Clouds

Abstract:
Conditional Neural Fields (CNFs) are increasingly being leveraged as continuous signal representations, by associating each data-sample with a latent variable that conditions a shared backbone Neural Field (NeF) to reconstruct the sample. However, existing CNF architectures face limitations when using this latent downstream in tasks requiring fine-grained geometric reasoning, such as classification and segmentation. We posit that this results from lack of explicit modelling of geometric information (e.g. locality in the signal or the orientation of a feature) in the latent space of CNFs. As such, we propose Equivariant Neural Fields (ENFs), a novel CNF architecture which uses a geometry-informed cross-attention to condition the NeF on a geometric variable—a latent point cloud of features—that enables an equivariant decoding from latent to field. We show that this approach induces a steerability property by which both field and latent are grounded in geometry and amenable to transformation laws: if the field transforms, the latent representation transforms accordingly—and vice versa. Crucially, this equivariance relation ensures that the latent is capable of (1) representing geometric patterns faitfhully, allowing for geometric reasoning in latent space, (2) weight-sharing over similar local patterns, allowing for efficient learning of datasets of fields. We validate these main properties in a range of tasks including classification, segmentation, forecasting, reconstruction and generative modelling, showing clear improvement over baselines with a geometry-free latent space.

Decision: accept (poster)
Reviewer scores: 6, 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1730324681918

thread_type:
official_review

<thread_object>
creator: reviewer_9x4v

<content>
summary:
The paper presents a method for conditioning a neural field using a set of $SE(n)$ equivariant local latents. The aim is to enhance downstream task performance by operating on the neural field’s learned latent representation, rather than on discrete samples from the continuous signal as in conventional approaches. It outlines the necessary conditions for an equivariant latent representation in neural fields and adapts a cross-attention architecture to support these conditions. The approach is evaluated across a wide variety of tasks.

strengths:
The paper is well-written and organized, with a clearly defined method supported by formal definitions.

The proposed solution is simple and intuitive for enhancing CNFs with local equivariant features.

I appreciate the variety of dataset types used in the experiments.

weaknesses:
Lack of motivation for using CNF latent encodings in downstream tasks

The paper does not explicitly discuss the motivation for using latent encodings of CNFs for downstream tasks. It seems that one advantage might be the ability to utilize more data for training since the reconstruction training stage does not require labeled data. This raises a follow-up question:
What benefit do latent features learned through continuous reconstruction (decoder) have over latent features learned through reconstructing a discrete sample?  It seems that a continuous decoder could enable the learning of discretization-agnostic features.  Is there another motivation for using latent encodings of CNFs for downstream tasks?  The purpose behind their use in enhancing downstream tasks remains somewhat unclear.

The motivation for using local CNF latent encodings could be framed more clearly.

The paper states that a notable limitation for conventional CNF (ln 51): “each field is encoded by a global variable”. However, this statement about cnfs limitations seems to be partially accurate. In fact, this approach for latent space modeling also has some clear advantages. For example, interpolating between two latents to generate novel signals is far more natural with a global latent structure, whereas a local latent structure requires solving the complex problem of finding correspondences between latent points. Thus, the characterization of a tradeoff rather than a limitation may be more appropriate.
Additionally, to address the limitations of a global latent, why not employ an encoder-decoder architecture with gradually decreasing spatial dependency in the latent representation (similar to a UNet)? This approach would provide a final latent that incorporates both local and global information. The rationale for restricting the model to an auto-decoder-style architecture remains unclear.

Overclaiming on Geometry-Appearance Separation in Neural Fields

The paper claims that the proposed method “separates geometry from appearance” in its representation. My understanding is that this refers to the structure of pose-appearance tuples in the latent space. However, how does the method ensure that only appearance information is captured in  $c_i$ ? This seems to rely solely on  $c_i$  being an $SE(n)$ invariant feature. Yet, some relevant geometric features are also invariant (e.g., shape volume), while some equivariant features can relate to appearance (e.g., how an object’s appearance changes are affected by material reflectance features under rotation). Consequently, enforcing a latent structure of invariant and equivariant features may not be sufficient to achieve true separation. Is there empirical evidence to support the above claim about separation?

Unclear reconstruction results

The paper claims: “Results show that ours as well as the baseline models struggle with accurately reconstructing the underlying shape from the SDF point clouds”. Given the inaccuracies in reconstruction, how can the learned features be effectively used for downstream tasks? Additionally, it’s unclear why this model underperforms in reconstruction compared to [3]. Both architectures appear similar (apart from the equivariant features), yet [3] reports more accurate reconstruction results.

Unclear segmentation results

The choice of ShapeNet as the dataset for segmentation evaluation is questionable, as it is an aligned dataset (line 468). A better alternative might be to use non-aligned datasets, such as those used for human-body segmentation in [4] and [5]. Another option would be to unalign ShapeNet by applying a random $SE(3)$ transformation to each data point. Additionally, it’s unclear if the point cloud-specific architectures were also trained with a reconstruction pretext stage. 

Additional comments.

Figure 8 is uninformative on its own without comparison to other methods, showcasing some of the proposed method qualitative benefits/limitations. 

Conditioning with k-nearest neighbors appears to restrict the smoothness of the modeled field to be at most continuous, while the data signals are at least differentiable.

The Steerability property for CNFs has also been defined and utilized in prior works, such as [1] and [2].

[1] Frame Averaging for Equivariant Shape Space Learning. Matan Atzmon, Koki Nagano, Sanja Fidler, Sameh Khamis, Yaron Lipman.

[2] Vector Neurons: A General Framework for SO(3)-Equivariant Networks. Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, Leonidas Guibas.

[3] Biao Zhang, Jiapeng Tang, Matthias Niessner, and Peter Wonka. 3dshape2vecset: A 3d shape representation for neural fields and generative diffusion models.

[4] Approximately Piecewise E(3) Equivariant Point Networks. Matan Atzmon, Jiahui Huang, Francis Williams, Or Litany.

[5] Generalizing neural human fitting to unseen poses with articulated se (3) equivariance. Haiwen Feng, Peter Kulits, Shichen Liu, Michael J Black, and Victoria Fernandez Abrevaya

questions:
I would appreciate a response regarding the weaknesses and questions mentioned above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730597904025

thread_type:
official_review

<thread_object>
creator: reviewer_yjcq

<content>
summary:
The paper introduces a novel class of Conditional Neural Fields (CNFs) called Equivariant Neural Fields (ENFs) which aim to address the limitations of CNFs in tasks requiring geometric reasoning. The authors propose a geometry-informed cross-attention mechanism that conditions on a latent point cloud of features, enabling equivariant decoding from the latents to the field of interest. This approach possess a steerability property where transformations in the field and mirrored in the latent space. Further, this approach ensures that the cross-attention attention operators respond similarly regardless of pose allowing for weight sharing over similar local patterns leading to more efficient learning. These claims are backed with experiments that demonstrate the advantages posed by the formulation and show a clear advantage over the baselines that have a geometry-free latent space.

strengths:
- The paper introduces a novel and mathematically sound method to incorporating geometric structure to neural fields through the equivariant cross attention. The steerability property is well formulated with proven bi-invariant constraints.
- The experimental details demonstrate an advantage over methods that do not incorporate such geometry informed structure in the latent space. Additionally, the locality and weight-sharing properties discussed are clearly demonstrated.
- The paper is well-written providing clear background on the neural fields, and the motivation for the need for enforcing equivariance in neural fields. The diagrams are informative and highlight the key components of the methodology. Highlighing geometry attributes in Section 3 with a blue text color was particularly helpful in aiding understanding

weaknesses:
- While the motivation to compare against other CNF based approaches is clear, the methodology seems to be restricted to a discussion and comparison to the results reported in functa (Dupont et al.) and other CNF-based methods but do not provide a thorough comparison against other equivariant methods or other state of the art methods. Perhaps a comparison of ENFs against more comparisons would strengthen the paper.

questions:
- I'm particularly curious about the use of these equivariant neural fields as a general backbone for any neural field based task? Are there any situations where it's not helpful to enforce equivariance especially for vision / PDE-based applications?
- Have you considered using this methodology in a generative context? I think the localized latent point clouds are a particularly interesting property that could lead to more structured creation.
- Did you study the sample efficiency of ENFs against other CNF methodologies in tasks such as classification? One would assume that enforcing equivariance should lead to a better sample efficiency throughout all truncations of the training dataset
- I'm curious about the computational cost of your experiments. Does it have a similar run time to the other baselines that were discussed?

Additionally, I believe there are a couple of typos that I may have spotted:
- In the abstract: faitfhully -> faithfulll
- Also, on line 103, posses needs to be possess?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730664174965

thread_type:
official_review

<thread_object>
creator: reviewer_akxe

<content>
summary:
The paper introduces the Equivariant Neural Fields, a variant of conditional neural field that uses a geometry-informed cross-attention to condition the NeF using geometrical point cloud representation. The method was validated using a variety of applications, including classification, segmentation, forecasting, and reconstruction.

strengths:
***Clear and Professional Presentation***: The paper is well-written, structured effectively, and easy to follow. Its clear motivation, logical organization, and high-quality visualizations contribute to a polished and professional presentation, making the methodology accessible and engaging.

***Introduction of Equivariant Neural Fields Model***: The authors propose a novel model, Equivariant Neural Fields, which combines conditional neural fields with point cloud conditioning and equivariant decoding from latent space to field. This approach creatively integrates Neural Fields with equivariant models designed for point clouds, expanding on existing techniques. Additionally, the paper introduces specialized attention layers and engineering optimizations that enhance the model's efficiency, showcasing an innovative blend of established methods.

***Comprehensive Experimental Validation***: The method is rigorously tested across a wide range of use cases and downstream tasks spanning various domains. This extensive evaluation demonstrates the versatility and potential real-world applicability of the proposed approach, supporting its robustness and utility across diverse applications.

weaknesses:
***High Time Complexity***: The proposed approach appears to be computationally intensive. It would be beneficial for the authors to compare the training time and memory usage of their method against a reference model, such as the Functa method, to provide a clearer assessment of its efficiency.

***Lack of Ablation Studies***: The paper would benefit from ablation studies to clarify the contributions of key components, such as Gaussian spatial windowing and the k-nearest neighbors (kNN) efficiency trick. These studies would help demonstrate how each element impacts the model’s training efficiency and overall performance.

***Suboptimal Segmentation Performance****: The segmentation results are weaker than those of traditional point cloud segmentation baselines. A deeper investigation and discussion of these performance differences would help in understanding and potentially addressing the gaps in segmentation accuracy.

questions:
Please refer to the weakness section
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687416433

thread_type:
official_review

<thread_object>
creator: reviewer_4ovu

<content>
summary:
This paper proposes equivariant conditional neural fields based on steerable networks. Architecture-wise, this paper proposes equivariant  cross-attention layers with Gaussian windowing as the basis of their Equivariant Neural Fields (ENF). The ENF is trained with a two-stage process: in the first stage, the ENF backbone takes in an input signal and outputs a latent point cloud of (pose, context) pairs. Downstream tasks can be accomplished by training a decoder which takes the latent point cloud as input. Experiments are performed on 2D image reconstruction and classification, 3D reconstruction, classification, and part-segmentation, flood map segmentation, and climate forecasting.

strengths:
This paper creates a novel equivariant neural field based on the notion of steerability from equivariant networks, which has the advantages of weight-sharing, locality, and geometric interpretability.

weaknesses:
1. The original Functa paper uses a SIREN neural field architecture but this paper uses an attention-based neural network architecture. This seems like a potentially unfair comparison. 
2. Another weakness of this paper is that there is no way to decide ahead of time whether to train the latent point cloud using MAML or autodecoding. 
3. The only baseline is Functa for most experiments. Is it possible to use NF2vec in Table 2 and Inr2Array [1] for any of the experiments involving downstream tasks? For tasks involving generalization, 
4. ENF performs only comparably to to the baselines on part-segmentation (Table 3), and some experiments (Table 2) don't show the effectiveness of using equivariance.

questions:
1. Does it make sense to compare against Inr2Array [1]?
2. Should NF2vec also be a baseline for the shape classification task (Table 2)?
3. Can Functa be trained with a cross-attention-based architecture, similar to that proposed for ENF?
4. For Functa baselines on downstream tasks such as classification, what was the architecture of the decoders used?

[1]: Zhou, Allan, et al. "Neural functional transformers." Advances in neural information processing systems 36 (2024).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732541506729

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Invitation to participate in further discussion

comment:
Dear Reviewers,

Thank you once again for your thoughtful reviews and constructive feedback on our manuscript. We appreciate the time and effort you’ve put into evaluating our work.

As we approach the end of the discussion period, we would like to kindly invite you to participate in the ongoing discussion. We feel we have addressed the points raised in your reviews, including additional experiments (Rev. yjcq), ablations (Rev. 4oVU), information and comparison on computational efficiency (Rev. yjcq, AKXe), and clarifications and revisions to our manuscript to clarify motivation and claims made in our work (Rev. 9X4V). Your insights have significantly improved the quality and rigor of our work, and we want to ensure that any remaining concerns are addressed thoroughly.

We would greatly value your engagement in this discussion to confirm that we’ve adequately addressed your comments, and invite you to amend your recommendations as you see fit. Additionally, we would like to explore any additional suggestions you might have; if there are specific areas where you feel further elaboration or revision is needed, please do not hesitate to share your thoughts.

Thank you for your continued involvement. We look forward to your response.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a variant of conditional neural fields by utilizing a latent point cloud rather than a global latent for a neural field. The authors claimed locality and steerability for the resulting Equivariant Neural Fields and demonstrated the effectiveness of latent for many tasks including perception and generative modeling. This paper is well-written as acknowledged by multiple reviewers. The implementation of equivariance through cross-attention between queries and latents is simple and elegant. Experiments clearly demonstrated the usefulness of geometry equivariance for representation learning.
The downside of the proposed method is the extra computational complexity of quadratic cross-attention between queries and multiple latents in a neural field. The authors have proposed a based approximation to full attention operation.
All four reviewers gave positive ratings for this paper. The AC recommends acceptance of this paper due to the novelty in conditioning neural fields on latent point cloud.