Neural Processing of Tri-Plane Hybrid Neural Fields
Authors: Adriano Cardace
Pierluigi Zama Ramirez
Francesco Ballerini
Allan Zhou
Samuele Salti
Luigi di Stefano
Venue: ICLR 2024 poster (2024)
Keywords: Neural Fields
3D classification
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Driven by the appealing properties of neural fields for storing and communicating 3D data, the problem of directly processing them to address tasks such as classification and part segmentation has emerged and has been investigated in recent works. 
Early approaches employ neural fields parameterized by shared networks trained on the whole dataset, achieving good task performance but sacrificing reconstruction quality.
To improve the latter, later methods focus on individual neural fields parameterized as large Multi-Layer Perceptrons (MLPs), which are, however, challenging to process due to the high dimensionality of the weight space, intrinsic weight space symmetries, and sensitivity to random initialization. Hence, results turn out significantly inferior to those achieved by processing explicit representations, e.g., point clouds or meshes.
In the meantime, hybrid representations, in particular based on tri-planes, have emerged as a more effective and efficient alternative to realize neural fields, but their direct processing has not been investigated yet.
In this paper, we show that the tri-plane discrete data structure encodes rich information, which can be effectively processed by standard deep-learning machinery. We define an extensive benchmark covering a diverse set of fields such as occupancy, signed/unsigned distance, and, for the first time,  radiance fields. While processing a field with the same reconstruction quality, we achieve task performance far superior to frameworks that process large MLPs and, for the first time, almost on par with architectures handling explicit representations.

Published as a conference paper at ICLR 2024
NEURAL PROCESSING OF TRI-PLANE HYBRID NEURAL FIELDS
Adriano Cardace' Pierluigi Zama Ramirez'
Francesco Ballerini' Allan Zhou?
Samuele Salti" Luigi Di Stefano
University of Bologna 2 Stanford University
adriano car www.aaaaaeaeannaaaaaaaa
ABSTRACT
Driven by the appealing properties of neural fields for storing and communicating 3D data, the problem of directly processing them to address tasks such as classi- fication and part segmentation has emerged and has been investigated works. Early approaches employ neural fields parameterized by shared networks trained on the whole dataset, achieving good task performance but sacrificing reconstruction quality. To improve the latter, later methods focus on individual neural fields parameterized as large Multi-Layer Perceptrons (MLPs), which are, however, challenging to process due to the high dimensionality of the weight space, intrinsic weight space symmetries, and sensitivity to random initialization. Hence, results out significantly inferior those achieved by processing explieit repre- sentations, e.g., point clouds or meshes. In the meantime, hybrid representations, in particular based on tri-planes, have emerged as a more effective and efficient alternative to realize neural fields, but their direct processing has not been investi- gated yet. In this paper, we show that the tri- tri-plane discrete data structure encodes rich information, which can be effectively processed by standard deep-learnin machinery. We define an extensive benchmark covering a diverse set of fields such as occupancy, signed/unsigned distance, and, for the first time, radiance fields. While processing a field with the same reconstruction quality, we achieve task performance far superior to frameworks that process large MLPs and, for the first time, almost on par with architectures handling explicit representations.
recent
INTRODUCTION
world of neural fields. Neural fields (Xie et al.] 2021) are functions defined at all spatial coordinates, parameterized by neural network such Multi-Layer Perceptron (MLP). They have been used to represent different kinds data, like image intensitics, scene radiances, 3D shapes, etc. In the context of 3D world representation, various types of neural fields have been explored, such as the signed/unsigned distance field (SDF/UDF) (Park et al. 2019 Chibane et al. 2020 Gropp et al.) 2020 Takikawa et al., 2021), the occupancy field (OF) (Mescheder et 2019 Peng al. (020). and the radiance field (RF) (Mildenhall et al. 2020b). Their main advantage is the ability to obtain continuous representation of the world, thereby providing information every point in space, unlike discrete counterparts like voxels, meshes, or point clouds. Moreover, neural fields allow for encoding a 3D geometry arbitrary resolution while using finite number of parameters, i.e., the weights of the MLP. Thus, the memory cost of the representation and its spatial resolution are decoupled.
Recently, hybrid neural fields (Xie et al., 202Tp, which combine continuous neural elements (i.e., MLPs) with discrete spatial structures (e.g., voxel grids (Peng et al.] 2020), point clouds (Tretschk) ct al. 2020), etc.) that encode local information, are gaining popularity due to faster inference (Reiser al. 2021 I), better use of network capacity (Rebain et al. 2021] and suitability to editing tasks (Liu 2020). In particular, the community has recently investigated tri-planes (Chan et al. 2022 ) a et al.
type of bybrid representation whose discrete components are three feature planes (zzy, y2, 222 due to its regular grid structure and compactness. Tri-planes have been deployed for RF (Hu et al.
2023
and SDF (Wang et al. 2023).

---

Published as a conference paper at ICLR 2024
MI Parly
Ad Nawval
Explicit Worl
Cosiiruet Acunner
Standed Nawala
PIEVIOUS
Prinicinass
Raconsrrudion Lere (Ctamiee Dritenca)
Figure I: Left: Neural processing of hybrid neural fields allows us to employ well-established architectures tackle deep learning tasks while avoiding problems related to processing MLPs, such as the high-dimensional weight space and the random initialization. Right: We achieve performance better than other works on this topic, close to methods that operate directly on explicit representations. without sacrificing the reconstruction quality of neural fields.
Neural processing of neural fields. As conjectured in De Luigi al. (2023), due to their advantages and increasing adoption in recent years, neural fields may become one of the standard methods for storing and communicating 3D information, i.e., repositories of digital twins of real objects stored as neural networks will become available. In such a scenario, developing strategies to solve tasks such as classification or segmentation by directly processing neural fields becomes relevant utilize these representations in practical applications. For instance, given a NeRF of chair, classifying the weights of the MLP without rendering and processing images would be faster, less computationally demanding, and more straightforward, e.g., there is no need to understand where to sample the 3D space as there is no sampling at all.
Farlier methods on the topic, such as Functa (Dupont et al. 2022), approached this scenario with shared networks trained on the whole dataset conditioned on a different global embedding for each object. In this case, neural field is realized by the shared network plus the embedding, which is then processed for downstream tasks. However, representing a whole dataset with a shared network is difficult, and the reconstruction quality of neural fields inevitably drops (see the plot in Fig. 1. For this reason, later approaches such as inr2vec (De Luigi et al. 2023). NEN (Zhou et al. 2023b NFT (Zhou et al., 2023b), and DWSNet NNvon al.] 2023) propose process neural fields consisting of a single large MIP, such as SIREN (Sitzmann et al. 2020), for each object. Although this strategy effectively maintains the reconstruction capabilities of neural fields, task performance suffers due to the challenges introduced by the need to handle MLPs, such as the large number of weights and the difficulty of embedding inductive biases into neural networks aimed processing MLPs, Moreover, randomly initialized MLPs trained on the same input data can converge to drastically different regions of the weight space due to the non-convex optimization problem and the symmetries of neural weight spaces (Entezari et al. [222] Ainsworth et al., 2023). Thus, identifying a model capable of processing MLPs and generalizing among all possible initializations is not straightforward. Previous works partially address these problems: inr2vec proposes an efficient and scalable architecture, and bypasses the initialization problem by fixing across MLPs; NFN, NFT, and DWSNet design networks that equivariant to weight symmetries. Nonetheless, all previous methods processing neural fields realized as single MI.Ps achieve unsatisfying performance, far from established architectures that operate on explicit representations, e.g., point clouds or meshes, as shown Fig. right.
Neural processing of tri-plane neural fields. To overcome the limitations of previous approaches and given the appealing properties of hybrid representations, in this paper, we explore the new research problem of tackling common 3D tasks by directly processing tri-plane neural fields. To this end, we analyze the information stored in the two components of this representation, which comprises a discrete feature space alongside a small MLP, and find out that the former contains rich semantic and geometric information. Based on this finding, we propose to process tri-plane neural fields by seamlessly applying, directly on the discrete feature space, standard neural architectures that have been developed and engineered over many years of research, such as CNNs (He et al.. 2016) or, thanks to tri-plane compactness, even Transformers (Vaswani et al. (017) (Fig. [ left). Moreover, we note empirically that the same geometric structures are encoded in tri- tri-planes fitted on the same shape from different initializations up to a permutation of the channels. Thus, we exploit this property to achieve robustness to the random initialization problem by processing tri-planes with standard architectures that are made invariant permutation of the channels. We achieve much better
2

---

Published as a conference paper at ICLR 2024
performance than all previous methods in classifying and segmenting objects represented as neural fields, almost on par with established architectures that operate on explicit representations, without sacrificing the representation quality (Fig. D
Summary of our contributions. Code available at
com/CVLBB-Unbbbo
triplane wwteeeeroossssiseeee
We set forth the new research problem of solving tasks by directly processing tri-plane neural fields. We show that the discrete features encode rich semantic and geometric information, which can be elaborated by applying well-established architectures. Moreover, we note how similar information is stored in tri-planes with different initializations of the same shape. Yet, the information is organized with different channel orders.
We show that applying well-established architectures on tri-planes achieves much better results than processing neural fields realized as a large MLP. Moreover, we reveal that employing architectures made invariant to the channel order improves performance in the challenging but more realistic scenario of randomly initialized neural fields. In this way, we almost close the gap between methods that operate on explicit representations and those working directly on neural representations.
To validate our results, we build a comprehensive benchmark for tri-plane neural field classification. We test our method by classifying neural fields that model various fields (UDF, SDF, OF, RF). In particular, to the best of our knowledge, we are the first to classify NeRFs without explicitly reconstructing the represented signal.
Finally, as the tri-plane structure is independent of the represented field, we train a single network to classify diverse tri-plane neural fields. Specifically, we show promising preliminary results of a unique model capable of classifying UDF, SDF, and OF.
2 RELATED WORK
Neural fields. Recent approaches have shown the ability of MLPs to parameterize fields representing any physical quantity of interest (Xie al. 2021). The works focusing on representing 3D data with MLPs rely on fitting functions such as the unsigned distance (Chibane et al. 2020), the signed distance (Park et al. 2019 Gropp et al. 22200; Sitzmann al., 2019 Jiang et al. 2020: Peng et al. 2020)) the occupancy (Mescheder et al. 2019 Chen & Zhang] 2019). or the scene radiance (Mildenhall et al.)
2020a). Among these approaches. SIREN (Sitzmann al. 2020) uses periodic activation functions to capture high-frequenee details. Recently, hybrid representations, in which the MLP is paired with a discrete data structure, have been introduced within the vision and graphic communities motivated by faster inference (Reiser et al., [202]), better use of network capacity (Rebain et al.) [222]) and suitability to editing tasks (Liu et (200)). These data structures decompose the input coordinate
space, either regularly, such as for voxel grids (Reiser et al. 2021 Fridovich-Ker Keil et al. 2022; Liu
jet al. 2020), tri- tri-planes (Wang et al. 2023 Chan et al. 2022; Wu & Zheng 2022; Hu et al. 2023), and 4D tensors (Chen et al. (2022), or irregularly, such as for point clouds (Tretschk et al. 2020).
and meshes (Peng et al.) 2021). Unlike these works, we do not focus on designing a neural field representation, but investigate how to directly process hybrid fields to solve tasks such as shape classification and 3D part segmentation. We focus on tri-planes due to their regular grid structure and compactness, which enable standard neural networks to process them seamlessly and effectively.
Neural functionals. Several recent approaches aim at processing functions parameterized as MLPs by employing other neural networks. MLPs are known exhibit weight space symmetries (Hecht- Nielsen 1990), i.e., hidden neurons can be permuted across layers without changing the function represented by the network. Works such as DWSNet (Navon et al.) 2023). NFN (Zhou al. 2023a), and NFT (Zhou et al. 2023b) leverage weight space symmetries as an inductive bias to develop novel architectures designed to process MIPps. Both DWSNet and NFN devise neural layers equivariant to the permutations arising in MLPs. In contrast, NFT builds upon the intuition of achieving permutation equivariance by removing positional encoding from a Transformer architecture. Among the works processing MLPs, inr2vec (De Luigi et al.) 2023) is the first that focuses specifically on MLPs representing 3D neural fields. It proposes a representation learning framework that compresses neural fields of 3D shapes into embeddings, which can then be used input for downstream tasks. In the scenario addressed by inr2vec, DWSNet, NEN, and NFT, each neural field is parameterized by its own ML.. Differently, the framework proposed in Functa (Dupont et al. 2022) relies on
3

---

Published as a conference paper at ICLR 2024
Ossenations
sertusrior
#:
Th-plare deteve
1ehop
x2
3B world
(p:
P...
PB
M
Figure 2: Left: Tri-plane representation and learning of each neural field. Right: Datasets
composed of many independent tri-plane hybrid neural fields, each representing a 3D object.
learning priors on the whole dataset with a shared network and then encoding each sample in a compact embedding. In this case, each neural field is parameterized by the shared network plus the emhedding. In particular, Functa (Dupont et al. 2022) leverages meta-learning techniques to learn the shared network, which is modulated with latent vectors to represent each data point. These vectors are then used to address both discriminative and generative tasks. It is worth pointing out that, though not originally proposed as a framework to process neural fields, DeepSDF (Park et al.) 2019) learns dataset priors by optimizing reconstruction objective through a shared auto-decoder network conditioned on thape-specifis embedding. Thus, as investigated in De Luigi al. (2023). ) the embeddings learnt by DeepSDF may be used for neural processing tasks similar to Functa s. However, as noted in De Luigi et al. (2023). shared network frameworks are problematic, as they cannot reconstruct the underlying signal with high fidelity and need whole dataset to learn the neural field of an object. Thus, akin to inr2vec, DWSNet, NFN, and NFT, we adopt the setting in which individual network represents each sample in dataset, as it is casier to deploy in the wild and thus more likely to become the standard practice in neural field processing. Unlike all previous works, however, process hybrid neural fields that combine MLP with a discrete spatial data structure. By only processing the discrete component, we circumvent the issues arising from directly processing MLP weights and obtain remarkable performance.
TRI-PLANE HYBRID NEURAL FIELDS
3.1 PRELIMINARIES
Neural fields. A field is physical quantity defined for all domain coordinates. We focus on fields describing the 3D world, and thus on RR coordinates p = (a, y. 2). We consider the 3D fields commonly used in computer vision and graphics, i.e., the SDF (Park et al.) 2019) and UDF (Chibane ett al. 2020), which map coordinates to the signed an unsigned distance from the closest surface, respectively, the OF (Mescheder et al., 2019), which computes the occupancy probability, and the RF (Mildenhall et al. 2020b), that outputs (R, G, B) colors and density a. A field can be modelled by function, 4. parameterized by Thus, for any point p. the field is given by q = (p: #). If parameters 0 are the weights of a neural network, Φ is said to be a neural field. On the other hand, if some of the parameters are the weights a neural network, whereas the rest encode local information within a discrete spatial structure, & is hybrid neural field (Xie et al. 2021).
Tri-plane representation. A special case of hybrid neural fields, originally proposed in Chan et al. (2022). is parameterized by discrete tri-plane feature map, T. and small MLP network, M (Fig. 2 left). T consists of three orthogonal 2D feature maps, T (Fay F.Z, , F F,z), with Fzg, Pzz, Føz E C where C' is the number of channels and W, H are the spatial dimensions of the feature maps. The feature vector associated with a 3D point, is computed by projecting the point onto the three orthogonal planes so to get the 2D coordinates, Pay. Pas. and Paz. relative to each plane. Then, the four feature vectors corresponding to the nearest neighbours in each plane are bi-leny interpolated to calculate three feature vectors, fey. faz, and fyz. which are summed up element- forment-wise&aee to obtain = fry + f,= + fyz. f E RC Finally, we concatenate with a positional encoding (Mildenhall et al. 2020b). PE, of the 3D point p and feed it to the MLP, which in turn outputs the field value at p q
4 ((p; f) = M (lf, PE).. We implement M with sin activation
functions (Sizzmann al. 2020) to better capture high-frequency details.
4

---

Published as a conference paper at ICLR 2024
Mesh from SDF
Point Cloud from UDF
Method
Type
Params (K)
CD (mm)
F-score (%)
CD (mm)
F-score (%)
inr2vec (De Luigi et al. 2023
Single
800
0.26
69.7
0.21
65.5
Tri-plane
Single
68.6
0.24
Tri-planc
Shared
64
1.57
42.9
3.45
33.3
DeepSDFF Park al. 2019
Shared
2400
6.6
25.1
5.6
5.7
Functa DDupott
Shared
7091
2.85
21.3
12.8
5,8
Table 1: Results of mesh and point cloud reconstruction on the Manifold40 test set. "Single" and "Shared" indicate neural fields trained on each shape independently or on the whole dataset.
Learning tri-planes. To learn field, we optimize a (T, M) pair for each 3D object, starting from randomly initialized parameters, 0, for both M and T. We sample points p; and feed them to T and M to compute the corresponding field quantities q. (p:;; 0). Then, loss, L. capturing the discrepancy between the predicted fields qi and the ground truth y.. applying an optional mapping between the output and the available supervision if needed (e.g., volumetric rendering in case of RF). An overview of this procedure is shown on the left of Fig. 2] described in detail in Appendix A We repeat this process for each 3D shape of dataset, thereby creating dataset of tri-plane hybrid neural fields (Fig. 2 right). We set C' to 16 and both H and W to 32. We use MLPs with three hidden layers, each having 64 neurons. We note that our proposal is independent of the learning procedure, and, in scenario in which neural fields are standard 3D data representation. we would already have datasets available.
optimize 0 with
3.2 TRI-PLANE ANALYSIS
We investigate here the benefits of tri-planes for 3D data representation and neural processing. Firstly, we assess their reconstruction capability, which is crucial in a world where neural fields may be used standard way to represent 3D assets. Secondly, we analyze the information learned in the 2D planes and how to be robust to the random initialization problem when handling tri-planes.
Reconstruction quality. We assess the tri-plane reconstruction performance by following the benchmark introduced in De Luigi et al. (2023). In Table we present the quantitative outcomes obtained by fitting SDFs and UDFs from meshes and point clouds of the Manifold40 dataset (Hu et al.) 2022). We compare with neural fields employed in inr2vec (De Luigi et al. 2023) and alternatives based on a shared architecture, such as DeepSDF (Park et al.) 2019) and Functa (Dupont et al.) 2022). Given the SDF and UDF fields learned by each framework, we reconstruct the explicit meshes and point clouds as described in Appendix B.T and evaluate them against the ground-truths. To conduct this evaluation, we sample dense point clouds of 16,384 points from both the reconstructed and ground- -oound-truthaa shapes. We employ the Chamfer Distance (Fan et al. (017)) and the F-Score (Tatarchenko) et al. 2019) to evaluate fidelity to ground-truths. As for meshes, the tri-planes representation stands out with the lowest Chamfer Distance (CD) (0. 18 mm), indicating its excellent reconstruction quality despite the relatively small number of parameters (only 64K). For point clouds, tri tri-planes produce reconstructions slightly worse than inr2vec but still comparable, i.c., 0.2 vs 0.24mm CD. In Appendix B.2 (Fig. 10 we show reconstructions attained from tri-plane representations for various types of fields. Moreover, in agreement with the findings of De Luigi et al. (2023). Table [ that shared network frameworks such as DeepSDF and Functa yield significantly worse performance in terms of reconstruction quality. We finally point out how sharing the MLP for all tri-planes is not as effective as learning individual neural fields (third vs second row). These results support our intuition that reconstruction quality mandates hybrid neural fields optimized individually on each data sample and highlight the importance of investigating the direct neural processing of these representations.
In Appendix B.3 (Fig. 5 Fig. 6) we show the reconstructions obtained by tri-planes and the other approaches considered in our evaluation.
Tri-plane content. To investigate how to directly process tri-plane neural fields, we inspected the content of their discrete spatial structure by visualizing the features stored in a plane alongside the view of the object rendered from the vantage point corresponding to the plane. Examples of these visualizations are depicted in Fig. 3 (left) for various objects such as a car, an airplane, and a bottle. To visualize features single image, displayed by a viridis colormap, feature channels at each spatial location. These visualizations show clearly that the tri-plane spatial structure learns the object shape, i.e., it contains information about its geometry. For this reason
take sum across the
5

---

Published as a conference paper at ICLR 2024
View
Tri-plane
(Ta, Ma)
(Tp, Ma)
(Ta, Ma)
(Perouuted Ta. Ma)
BDF
Figure 3: Left: For three different hybrid neural fields (from top bottom: SDF, UDF, RF) render a view of the reconstructed 3D object alongside the corresponding tri-plane feature map. Right: From left to right, reconstructions of two (tri-plane, MLP) pairs with different initializations, namely (TA, MA) and (Tp, n); the mixed pair (TA, RR;; a channel permutation of Ta and M
and further investigations reported in Appendix D we conjecture, and demonstrate empirically in subsequent sections, that to tackle tasks such as classification and segmentation we can discard the MLPs and process only the tri-plane structure of the neural fields. Remarkably, the regular grid structure of tri-planes allows to deploy popular and effective neural architectures, such as CNNs and Transformers. On the contrary, direct ingestion of MLPs for neural processing is problematic and leads to sub-optimal task performance.
Random initialization. Furthermore, we investigate the effect of random initializations on tri plane neural fields. We note here that by "random initialization" we mean that cach (tri-plane, MLP) pair adheres to the same initialization scheme but has a different random seed (see Appendix E.5). We find out empirically that the main difference between tri-plane structures learnt from different optimizations of the same shape lies in the channel order within feature plane. Indeed, we conducted experiments where we fit the same 3D shape twice (see Fig. (right)), starting from two random initializations of both the tri-plane structure and the MLP weights. Although the geometric content of the two tri-planes is similar, due the different initialization, the tri-plane learnt in the first run cannot be used with the MLP obtained in the second (third column of Fig, 3] right side), and vice-versa. However,
always possible to find suitable permutation of the channels of the first tri-plane such that the second MIP can correctly decode its features (fourth column of Fig 3 right side), and vice-versa. We found the right permutation by a brute-force search based on maximizing reconstruction quality. To make the search feasible, we used smaller number of channels, i.e., C' 8 rather than C 16. Still, the experimental results in Section 4.1 support our belief that the main source of variance across randomly initialized tri- tri-plane optimizations of the same shape consists of a permutation of the channel order. Thus, unlike neural fields realized as MLPs, with tri- ri-planes, it is straightforward counteract the nuisances due random initialization by adopting standard architectures made invariant to the channel order.
3.3 ARCHITECTURES FOR NEURAL PROCESSING OF TRI-PLANE NEURAL FIELDS
Based on the above analysis, we propose to process tripllnes with Transformers (Vaswani al.) 2017). In particular, we propose to rely on Transformer encoder without positional encoding, which equivariant token positions. By tokenizing tri-planes so that each token represents channel of plane, such architecture seamlessly computes representations equivariant to the order of the channels. Specifically, we unroll each channel of size x W, to obtain a token of dimension / HW within sequence of length 3C' tokens. These tokens then linearly projected and fed into the Transformer. The output the encoder once again a sequence of 3C tokens.
For global tasks like classification, the output sequence is subsequently subjected to pool operator to obtain a global embedding that characterizes the input shape. our experiments, this embedding is then processed through a stack of fully connected layers to compute the logits. The

---

Published as a conference paper at ICLR 2024
BDF
SDF
Mothod
Type
Input
ModelNer10
ShapeNettO
ScunNetLO
Manifold
ShapeNettO
ShapeNetRender
DeepSDF Park ct al_
2019
Shared
Latent vector
41.2
76.9
512
64.9
Functa
Dogonrer
20c4
Shared
Modulation
56.4
85.9
Inr2vec De Luigi et al. 2025
Single
MLP
10.6
42.0
40.9
13.1
38.6
MLP
Single
MLP
3.7
28.8
36.7
4.2
29.6
22.0
NPN Zoou
20124
Single
MI.P
9.0
45.3
4.1
87.0
NET
Zin mi A0730
Single
MLP
6.9
45.3
4.1
33.8
85.3
DW SNar Niiverr of al 2023
Single
MLP
56.3
78.4
62.2
47.9
79.1
83.1
Single
Tri-planc
87.0
94,1
69.1
86.8
91.8
92.6
Table 2: Test set accuracy for shape classification across neural fields. frameworks capable of processing neural fields.
We compare several
way the tokens are defined, the absence of positional encoding, and the final max pool operator allow for achieving invariance to the channel order. For dense tasks like part segmentation, we also utilize treat the coordinates queries to segment
the decoder part of Transformers. More specifically, sequence of input tokens to the decoder. Each point with coordinates (x, y. z) undergoes positional encoding (Mildenhall al. 2020b) and is then projected to a higher-dimensional space using linear layer. By leveraging the cross- attenntoonnion mechanisms within the decoder, each imput token representing a query point can globally attend to the most relevant parts of the tri- tri-planes processed by the encoder to produce its logits. Additional details on the architectures are reported in Appendix E.3
4 TASKS ON NEURAL FIELDS
4.1 NEURAL FIELD CLASSIFICATION
Benchmark. We perform extensive tests to validate our approach. In so doing, we build the first neural field classification benchmark, where we compare all the existing proposals for neural field processing on the task of predicting the category of the objects represented within the field without recreating the explicit signal. Specifically, we test all methods on UDF fields obtained from point clouds of ModeINet40 (Wu et al. 2015). ShapeNet10 (Qin et al., 2019). and ScanNet10 (Qin et al. 2019); SDF fields learned from meshes of Manifold40 (Hu et al. [002];; ; OF fields obtained from voxels grids of Ne110 In addition, we provide for the first time classification results on neural radiance fields (RF), learned from ShapenetRender (Xu et al. 2019). See Appendix B. for more details on the benchmark. Besides a simple MIP baseline, we compare with frameworks designed to process neural fields realized as MLPs, i.c., inr2vec (De Luigi al. 2023). NEN (Zhou et al. 2023a), NFT (Zhou et al. 2023b), and DWSNet (Navon et al. (023). These methods process single MLP neural fields, which we implement as SIREN networks (Sitzmann et al. 2020). Differently from De Luigi et a (202)) the MLPs in our benchmark are randomly initialized to simulate real-world scenarios. Unlike all previous methods, ours processes individual tri- tri-plane neural fields, which are also randomly initialized. Moreover, we compare with frameworks where neural fields are realized by a shared network and a small latent vector or modulation, i.e., DeepSDF (Park et al., 20T9) and Functa (Dupont et al. 2022). Whenever possible, we use the official code released by the authors to run the experiments. Note that not all frameworks can be easily extended to all fields. Therefore, only test each framework in the settings that are compatible with our resources and that do not require fundamental changes to the original implementations (see Appendix E.2 for more details).
Results. As we can observe in Table 2 overall, shared architecture frameworks (DeepSDF and Functa) outperform previous methods that directly operate on neural fields represented as a single neural network. However, we point out again that the reconstruction capability of such frameworks is poor, as shown in Section 3.2 Conversely, previous methods that utilize individual neural fields demonstrate superior reconstruction quality but struggle to perform effectively in real-world scenarios where shapes need to be fitted starting from arbitrary initialization points. inr2vec makes the assumption of learning all MLPs starting from the same initialization, and does not work when this initialization schema is not applied. Among the family of methods that adopt layers equivariant and invariant to permutations of the neurons, only DWSNet works on the large MLPs constituting our benchmark, though performance tends to be worse than shared network approaches. Our method delivers the best of both worlds: it ingests tri-planes neural fields, which exhibit excellent reconstruction quality while achieving the best performance overall, often surpassing by large margin all other methods, including those relying on a shared neural field, e.g., the accuracy on ScanNet10 is 56.4 for Functa vs 69.1 for our method. Hence, we can state confidently that approach achieves the best trade-off
7

---

Published as a conference paper at ICLR 2024
Method
Inpat
ShapeNer10 Model
ScamNetto
Manifold40)
ShupeNert0
ShupeNerRender
our
Tri-plane
87.0
94.1
69.1
86.8
91.8
92.6
PoiniNet (Q)
2017a
Point Cloud
88.8
94.3
72.7
Waskiaalerr Lane & Jsl 2020
Mesh
90.0
Con
Mituranh & Soneeer
2015
Vouel
92.1
ResNer50
Images
94.0
Table 3: Comparison with explicit representations. Top: Test set accuracy of our neural field processing method. Bottom: Standard networks trained and tested on explicit representations.
between classification accuracy and reconstruction quality. Finally, we highlight that proposal is effective with all the datasets and kinds of fields addressed in the experiments.
Comparison with explicit representations. In Table 3 we compare our method against established architectures specifically designed to process explicit representations. For a fair comparison, we reconstruct the explicit data from each field so that exactly the same shapes are used in each experiment. Practically, we reconstruct point clouds, mesh, and voxel grids from UDF, SDF, and OF, respectively. Then, we process them with specialized architectures, i.c., PointNet (Qi al. 2017a for point clouds, MeshWalker (Lahav & Tal 2020) for meshes, and Conv (Maturana & Scherer 2015) for voxel grids. As for RF, we render a multi- dataset with 36 views for each object. Then, we train 36 per-view ResNet50 (He et al.) 2016) so as to ensemble the predictions at test time. We highlight how our proposal, which can classify every neural field with the same standard architecture, almost closes the performance gap with respect to specialized architectures designed to process explicit representations. Noticeably, we show that NeRFs can be classified accurately from the features stored in a tri-plane structure without rendering any images.
Towards universal tri-plane classification. Finally, to the best of our knowledge, we implement for the first time a universal tri- pripplan classifier, i.e., a model which can be trained and tested with any kind of tri-plane hybrid neural field. Indeed, since the tri-plane structure, as well as the neural processing architecture, are just the same, regardless of the kind of field, we can seamlessly learn a unified model able to classify a variety of fields. For example, we start from the meshes of the Manifold40 dataset and obtain the corresponding point clouds and voxel grids so as to three different fields (SDF, UDF, and OF). Accordingly, we build training, validation, and test sets with samples drawn from all three fields. More precisely, shape appears in set represented as SDF, it also appears in that set UDF and OF. Then, as reported in Table 4 we run classification experiments by training models on each of the individual fields well as on all three of them jointly. The results show that when classifier is trained only one field, it may not generalize well to others. On the other hand, a single model trained jointly on all fields not only works well with test samples coming from each one, but it also outperforms the models trained individually on a single kind of field.
Train
Test
UDF
SDF
OF
UDF
SDF
OF
84.7
78.4
15.6
67.3
86.8
11.9
49.3
46.9
77.7
87.4
87.8
80.3
Table 4: Universal tri-plane classifier.
Test set accuracy on Manifold40.
4.2 NEURAL FIELD 3D PART SEGMENTATION
We explore here the potential of our method in tackling dense prediction tasks like part segmentation, where the goal is to predict the correct part label for any given 3D point. In Table we compare our method inr2vec (De Luigi et al. (023). which was trained on fields generated from random initialization and is the only competitor capable of addressing the part segmentation task. Our experiments were conducted by fitting UDF fields from point clouds of 2048 points from the ShapeNetPart dataset (Yi et al.] 2016). As reference, we present the results obtained using specialized architectures commonly used for point cloud segmentation, like PointNet, PointNet++, and DGCNN. Akin De Luigi et al. (2023), all models are trained on the point clouds reconstructed from the fitted fields. We observe that our proposal outperforms inr2vec by a large margin, with improvements of 20% and 16.7% for instance and class mloU, respectively. Moreover, Table 5 demonstrates once again that tri- tri-planes are effective in substantially reducing the performance gap between processing neural fields and explicit representations.
8

---

Published as a conference paper at ICLR 2024
an stoU
mlo!
Metledd
inz3vx: De La al. 2023
MEPP
64.2
64.5
57.9
72.9
678
56.4
67.6
68.4
81.6
70.6
55.5
58.8
51.5
67.2
64.7
40.1
58.4
62.5
Dusg
To: glane
84.2
81.5
830
30.2
57.4
76.6
V0.3
68.3
41.6
85.9
82.1
95.0
76.7
94.
61.9
59.0
73.4
30.9
PannNa
stal Ear
load
78 95
11
89
67.0
30.0
90.6
57.8
71.6
81.5
PainNet
Cloud
52.73
72.2
94.7
81.3
61.1
74.1
82.3
DOUNN
Wang 2019
Cloud
80.85
890
51.3
34.0
59.3
92.8
77.8
625
71.6 81.1
Table 5: Part segmentation results. Top: Implicit frameworks. Bottom: Methods on explicit representation. In bold, best results among frameworks processing neural fields.
D22
SE3 Manifold40
OF
Method
Inpux
ModelNetIO
StapeNettU
ScaaNet10
ShapeNett0
MLP
Tri-phan
41.6
$4.2
55,8
40.2
79.1
CNN
Tri-plane
82.2
82.5
Net
plane
93.4
91.5
Spetial PointNet
Tripplate
52.3
51.3
54.7
Transfrmer
Tri Triphine
87.0
94.1
69.1
86.8
91.8
Table 6: Ablation study of architectures for tri-plane neural field classification
4.3 DIFFERENT ARCHITECTURES FOR TRI-PLANE PROCESSING
In T we compare several plausible alternatives to Transformers for processing tri-planes, which have roughly the same number of parameters and have been trained with the same hyperparameters. As discussed previously, since tri- planes contain an informative and regular discrete data structure and are compact, they can be processed with standard architectures. Hence, we test an MLP, a ResNet50 (He et al. 2016), and two variants of PointNet all with roughly the same parameters. A simple MLP that processes the flattened tri-planes (row 1) severely under-performs with respect to the alternatives, likely due to its inability to capture the spatial structures present in the input as well its sensitivity to the channel permutation caused by random initializations. A standard CNN like ResNet50, processing tri tri-planes stacked together and treated as a multi-channed image of resolution W H, is instead equipped with the inductive biases needed to effectively process the spatial information contained in the tri-planes (Fig- 3 and already delivers promising performance, although cannot cope with channel permutations. The two variants of PointNer show the i importance of invariance to channel order. In the first variant (row 3), each channel is flattened to create set of vectors in TRW x with 3C' elements, and then the max pool operator is applied to extract global embedding invariant to the channel order that is fed to a classifier. We observe here better performance across all fields than those attained by CNN. If we instead unroll tri- planes along the channel dimension to create set of vectors in 3-3 with W x H elements (row 4), the results are poor, as this arrangement does not make the network invariant to the channel order but to the spatial position of the features. Finally, we report (row 5) results for the Transformer architecture adopted in this paper, which, similarly to the previous PointNet, invariant to the channel order thanks to the max-pool operator and yields slightly better performance, probably due to the attention mechanism that better captures inter-channel correlations.
5 CONCLUDING REMARKS AND LIMITATIONS
We have shown that trane hybrid neural fields particularly amenable direct neural processing without sacrificing representation quality. Indeed, by feeding only the tri-plane structure into standard architectures, such Transformers, we achieve better classification and segmentation performance compared to previous frameworks aimed at processing neural fields and dramatically shrink the gap with respect to specialized architectures designed to process 3D data represented explicitly. To validate our intuitions, we propose the first benchmark for neural processing of neural fields, which includes the main kinds of fields used to model the 3D world as well all the published methods that tackle this very novel research problem. Within our experimental evaluation, we show for the first time that NeRFs can be effectively classified without rendering any images. A major limitation of our work is that tri-plane neural fields are specific to 3D world modeling. Thus, we plan to address other kinds of hybrid neural fields, like those relying on sparse feature sets (Li et al. (2022) as well as other kinds of signals, such time-varying radiance fields (Sara Fridovich- Keil and Giacomo Meanti et al.) 2023). Other research directions deal with processing hybrid neural fields capturing large scenes featuring multiple objects address tasks like 3D object detection semantic segmentation.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
We wish to thank lacopo Curti for the results produced during his master's thesis.
REFERENCES
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re- basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Representations, 2023.
Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini de Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-aware 3d generative adversarial networks. 2022 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2022. doi: 10.1 Ree.1//0Vvr22288222222222 doi
URL http::://dx.dd..oo///00000 11t.:...111111.11111
2022. 01565
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint 1rXiv:1512.03012, 2015.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision (ECCV), 2022.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5939-5948, 2019.
Julian Chibane, Gerard Pons-Moll, et al. Neural unsigned distance fields for implicit function learning. Advances in Neural Information Processing Systems, 33:21638-2166 2020.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias
NicBner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5828-5839, 2017.
Luca De Luigi, Adriano Cardace, Riccardo Spezialetti, Pierluigi Zama Ramirez, Samuele Salti, and Luigi Di Stefano. Deep learning on implicit neural representations of shapes. In International Conference on Learning Representations (ICLR), 2023.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13142-13153 13153, 2023.
Emilien Dupont, Hyunjik Kim, SM Ali Eslami, Danilo Jimenez Rezende, and Dan Rosenbaum. From data to functa: Your data point is a function and you can treat it like one. In International Conference on Machine Learning, pp. 5694-5725. PMLR, 2022.
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. In International Conference Learning Representations, 2021.
Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 605-613, 2017.
Sara -ridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angioo Kanazawa. Plenoxels: Radiance fields without neural networks. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Jun 2022. doi: 10. 11111///vpp226822222220000 URL http: doi wttp:///......1/.//....ee..111./ 00028888.202 2022.02222 00542
Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regular- ization for learning shapes. In International Conference on Machine Learning, pp. 3789-3799. PMLR, 2020.
10

---

Published as a conference paper at ICLR 2024
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 770-778, 2016. doi: 10.1109/CVPR.2016..90
Robert Hecht-Nielsen. On the algebraic structure of feedforward network weight spaces. In Advanced Neural Computers, pp. 129- 135. Elsevier, 1990.
Shi-Min Hu, Zheng-Ning Liu, Meng-Haa Heng Guo, Jun-Xiong Cai, Jiahui Huang, Tai-Jiang Mu, and Ralph R Martin. Subdivision-based mesh convolution networks. ACM Transactions on Graphics (TOG), -163):1-16, 2022.
Wenbo Hu, Yuling Wang, Lin Ma, Bangbang Yang, Lin Gao, Xiao Liu, and Yuewen Ma. Tri-miprf: Tri-mip representation for efficient anti-aliasing neural radiance fields. In ICCV, 2023.
Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, Thomas Funkhouser, et al. Local implicit grid representations for 3d scenes. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 6001-6010, 2020.
Alon Lahav and Ayellet Tal. Meshwalker: Deep mesh understanding by random walks. ACM Transactions on Graphies (TOG), 39(6):1-13, 2020.
Ruilong Li, Hang Gao, Matthew Tancik, and Angjoo Kanazawa. Nerface: Efficient sampling accelerates nerfs. arXiv preprint htt.:2305004966.. 2023.
Tianyang Li, Xin Wen, Yu-Shen I Liu, Hua Su, and Zhizhong Han. Learning deep implicit functions for 3d shapes with dynamic code clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12840-12850, 2022.
Tsung- Yi Lin, Priya Goyal, Ross Girshick. Kaiming He, and Piotr Dollár, Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Larochelle, M. Ranzato, Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 15651-15663 Curran Asso- ciates, Inc., 2020. URL htt.ttttss///rcceeiiiiii neurips.........ss http://eapppppaeeeeppppeeeeeee files/paper) pdd
William Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics, 21(4):163-169, 1987.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1111.0101, 2017.
Daniel Maturana and Sebastian Scherer. Voxnet: 3d convolutional neural network for real-time object recognition. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 922-928. IEEE, 2015.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction function space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4460-4477 4470, 2019.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Taneik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020a.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In European conference on computer vision, pp. 405-421 Springer, 2020b.
Aviv Navon, Aviv Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik, and Haggai Maron. Equiv- ariant architectures for learning in deep weight spaces. In International Conference on Machine Learning, 2023.
11

---

Published as a conference paper at ICLR 2024
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 165-174. 2019.
Sida Peng, Yuanqing Zhang. Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9054-9063. 2021.
Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolu- tional occupancy networks. In European Conference on Computer Vision, pp. 523-540. Springer, 2020.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-660, 2017a.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. Advances in neural information processing systems, 30, 2017b.
Can Qin, Haoxuan You, Lichen Wang, C-C Jay Kuo, and Yun Fu, Pointdan: A multi-scale 3d domain adaption network for point cloud representation. Advances in Neural Information Processing Systems, 32, 2019.
Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke Li, Kwang Moo Yi, and Andrea Tagliasacchi. Derf: Decomposed radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14153-14161, 2021.
Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Oct 2021. doi: 10. (11119/ccv4992222221..0022
URL htLLhhtpp///x.ooo doi:::/xxxxodoooo...oooooo org/
00.1109/1CCV48990 . 2021. C1407
Sara Kridovich-Keil and Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In CVPR, 2023.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Contin- uous 3d-structure-aware neural scene representations. Advances in Neural Information Processing Systems, 32, 2019.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-7473, 2020.
Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for multi-domain operations applications, volume 11006, pp. 1100612. International Society for Optics and Photonics, 2019.
Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11358-11367, 2021.
Maxim Tatarchenko, Stephan R Richter, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox. What do niegle-view 3d reconstruction networks learn? In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 3405-3414, 2019.
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Carsten Stoll, and Christian Theobalt. Patchnets: Patch-based based generalizable deep implicit 3d shape representations. In 2020: 16th European Conference, Glasgow, UK August 23-28, 2020, Proceedings, Part XVI 16, pp. 293-309. Springer, 2020.
Computer
12

---

Published as a conference paper at ICLR 2024
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gareett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., wwwrinu.rrpccmm 2017. URL https:://proceeddinne i wwtpurippcccpppprrrrreeepeppeeee tamper_fflle wrrfffeessee www.fees/pppp//211/... Tlle://fDoe22BSSTeeeee1oooooLLod Papeeeaaaaaaappppr.... Paper.pppppp....
Yiqun Wang, Ivan Skorokhodov, and Peter Wonka. Pet-neus: Positional encoding tri-planes for neural surfaces, 2023.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph enn for learning point clouds. Acm Transactions On Graphies (tog), 38(5):1-12, 2019.
Rundi Wu and Changxi Zheng. Learning to generate 3d shapes from single example. ACM Transactions Graphics (TOG), (1)6), 2022.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In 2015 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 1912-1920, 2015.
Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual computing and beyond. arXiv preprint arXiv:2111.11422 2021.
Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann. Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.. 2019. URL httpss//pprooeeeinsss..eute nwwrips. hwtp://wweeeeeeeeeeeeee....oooo. file/ F1103330072441559999111111166663 Papeee ppff
Li Yi, Vladimir G. Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections. SIGGRAPH Asia, 2016.
Allan Zhou, Kaien Yang, Kaylee Burns, Adriano Cardace, Yiding Jiang, Samuel Sokota, Zico Kolter, and Chelsea Finn. Permutation equivariant neural functionals. Advances in neural information processing systems, 37, 2023a.
Allan Zhou, Kaien Yang. Yiding Jiang, Kaylee Burns, Winnie Xu, Samuel Sokota, J Zico Kolter, and Chelsea Finn. Neural functional transformers. Advances in neural information processing systems, 37, 2023b.
13

---

Published as a conference paper at ICLR 2024
A LEARNING TRI-PLANE NEURAL FIELDS
In this section, we outline the procedure used to learn a single (tri-plane, MLP) pair, denoted as (I', (M), to create the datasets of hybrid neural fields. We note that, nonetheless, our proposal is agnostic to how neural fields were trained, as, in the scenario we consider, neural fields would be available as a standard data representation and ready to be processed.
To learn a field, we optimize the parameters f of both TT' and M with a loss between the reconstructed field qi and the sensor measurement yt. The optimized weights 6* are obtained as follows:
N
0" arg min Σ 0 N
((ai(yi),
where Tb R33 -> Rdd represents the field function defined by both the tri-plane and the MLP, and C is a function that computes the error between predicted and ground-truth values, with d' 1 when supervising the fitting process of UDF, SDF, or OF, while d for RF. oi represents the mapping from the sensor domain to the field, e.g., from a mesh to its SDF, which does not need to be differentiable. On the other hand, a2 represents a forward map between the output of the field and the domain of the available supervisory signal and must be a differentiable function (c.g., 02 models volumetrie rendering for RF). 01 and ag can also be identity functions, c.g., this is the case for ag when learning an SDF from mesh, supervise directly with the field values. In the remainder of this section, we describe the steps required to create our neural field datasets.
UDF from point cloud. Given point p E R3, UDF(p) is defined as min l|p- r11 namely the Euclidean distance from p to the closest point r of the point cloud. For each shape, we first sample 600K points and compute the corresponding ground truth UDF values g:- Between these points, 100K are sampled on the surface, 250K close to the surface, 200K points at a medium distance from the surface, 25K far from the surface, and an additional 25K scattered uniformly in the volume. More precisely, close points are computed by corrupting the point on the surface with noise sampled from the normal distribution (0, 0.001), medium-distance ones with noise from A 0.01), and far-away ones with noise from (O, 0.1). Then, during the optimization process, we randomly select N = 50K points for each batch, apply the interpolation scheme explained in Section 3.I retrieve the correct feature vector from the tri-plane, concatenate it with the positional encoding of p. and feed it to the MLP to compute the loss. This procedure is repeated 1000 times. As for the training objective, we follow De Luigi et al. (2023). i.e., we scale the UDF ground truth labels into the [0, 1] range, with and representing the maximum and minimum distance from the surface, respectively, We then constrain predictions to be in that range through final sigmoid activation function. Finally, we optimize the weights of & using the binary cross entropy between the scaled ground truth labels q. and the predicted field values d::
Line
log(()) + (1 - q:) log(1 4.)
N
SDF from mesh. Given a point P E R3, SDF(p) is defined as the Euclidean distance from to the closest point of the surface, with positive sign p is outside the shape and negative sign otherwise. For watertight meshes, we can easily understand whether a point lies inside or outside the surface by analyzing normals. We compute SDF ground truth values s; for 600K sampled points. Then, we compute the binary cross entropy loss of Eq ① on N = 50K points. We found 600 optimization steps to be enough to achieve satisfying reconstruction quality. Similarly to the UDF, the SDF values s, are scaled into the [0, 1] range, with 0 and representing the maximum and minimum distance from the surface, respectively and 0.5 representing the surface level set.
OF from voxel grid. Given a point p R3, OF(p) is defined as the probability o, of p being occupied. For voxel grids, the fitting process is straightforward, as each cube c; can either be occupied directly apply the binary cross-entropy loss. We use the same protocol applied for point clouds and optimize for 1000 steps while sampling 50K points at each step. However, due to the high imbalance between empty and full cells, we follow De Luigi
(value 1) or empty (value 0). Thus,
et al. (2023) and employ focal loss (Lin et al. 2017 7
N
Liocal = N
Σ
- 8(1-o)) log(o,) + (1 - -3))0 - c.) log(1 - oz)
14

---

Published as a conference paper at ICLR 2024
with 3 and y representing the balancing and focusing parameters, respectively.
RF from images. Given a point p E ER RF(P) (Mildenhall et al. 2020b) is a 4-dimensional vector containing the (R, G, B) color channels and density a of the point. To fit RF, we rely the NerfAce library (Li et al. 2023). Specifically, we implement a simplified version of NeRF that does not take into account the viewing direction and consists of (tri-plane, MLP) pair that predicts the four field values. To render RGB images, we leverage the volumetric rendering implementation of Li al. (2023). In partieular, at each iteration, for given camera pose, batch of rays (128 in our case) is selected with the corresponding RGB ground truth values and 3D points are sampled along these rays. The coordinates of these points are then interpolated, as described in Section 3.1, to extract features from the plane and fed to the MLP, which in turn predicts the (R, G, B, σ) values for each of them. Finally, by means of volumetric rendering, final RGB vector color is obtained and compared to the ground truth via smooth L.1 loss. Each RF is trained for 1500 steps.
B EXPLICIT RECONSTRUCTION FROM NEURAL FIELDS
In this section, we discuss how to sample 3D explicit representations from neural fields and show some examples of such reconstructions.
B.1 SAMPLING EXPLICIT REPRESENTATIONS
Point cloud from UDF We generate point cloud based on the corresponding UDF using slightly adapted version of the algorithm introduced by [Chbbane et al.] (2020). The fundamental concept involves querying the UDF with points distributed throughout the specific region of the 3D space under consideration. These points are then projected onto the isosurface based on their predicted UDF values. For given point p R its updated position Pnew is determined as follows:
V O(P())))
Pnew = P - -pp; 0)
(p:
where ((p; 8) represents the field value at point p which is approximated using the (T, M) pair with parameters 8. It is important to note that the negative gradient of the UDF indicates the direction of the steepest decrease in distance from the surface, effectively pointing towards the nearest point on the isosurface. Eq. (2) can thus be interpreted as shifting point along the direction of maximum UDF decrease, ultimately arriving at point Pnew on the surface. However, is crucial to observe that 4 serves as an approximation of the true UDF. This raises two key considerations: (i) the gradient of must be normalized, as illustrated in Eq. (2). whereas the gradient of the actual UDF maintains
norm 1 everywhere except on the surface; (ii) the predicted UDF value can be imprecise, potentially
resulting in point p remaining distant from the surface even after applying Eq. (2). To address the
second point, we refine Pnew by iteratively repeating the update described in Eq. (2). With each iteration, the point gradually approaches the surface, where the values approximated by & become more accurate, eventually placing the point precisely on the isosurface. The whole algorithm for sampling a dense point cloud from a given UDF entails the following steps: (i) generate a set of points uniformly scattered within the specified region of 3D space and predict their UDF values using the provided (T, .) pair; (ti) discard points with predicted UDF values exceeding a fixed threshold (0.05 in our experiments). For the remaining points, update their coordinates iteratively using Eq. (2). typically requiring 5 updates for satisfactory results; (iii) repeat the entire procedure until the reconstructed point cloud reaches the desired number of points.
Triangle mesh from SDF. We employ the Marching Cubes algorithm (Lorensen & Cline 1987) to construct a mesh based on the corresponding SDF. The Marching Cubes process involves system- atically traversing the 3D space by evaluating the SDF at 8 locations simultancously. forming the vertices of a tiny virtual cube. This traversal continues until the entirety of the desired 3D region has been covered. For each cube, the algorithm identifies the triangles necessary to represent the portion of the isosurface passing through it. These triangles from all cubes are then integrated to create the final reconstructed surface. To determine the number and placement of triangles for an individual cube, the algorithm examines the SDF values at pairs of neighbouring vertices within the cube. A triangle vertex is inserted between two vertices with opposing SDF signs. Because the possible combinations of SDF signs at cube vertices are limited, a lookup table is generated to retrieve the triangle configuration for a given cube. This configuration is derived from the SDF signs at the eight
15

---

Published as a conference paper at ICLR 2024
Point clouds from UDF
Meshes from SDF
Voxels from OF
Images from RF
Figure 4: Tri-plane reconstruction examples of point clouds from UDF, meshes from SDF,
voxels from OF, and images from RF (from top to bottom)
16

---

Published as a conference paper at ICLR 2024
Tri-Plane
Inr2vec
DeepSDF
Functa
Figure 5: Reconstruction comparison for Manifold40 meshes obtained from SDF
Tri-Plane
Inr2vec
DeepSDF
Functa
Figure 6: Reconstruction comparison for ModelNet40 point clouds obtained from UDF
vertices of the cube, which are combined into an 8-bit integer and used as a lookup table key. Once the triangle configuration for cube is retrieved, the vertices of the triangles are positioned along the edges connecting the cube vertices. This positioning is accomplished through linear interpolation of the two SDF values associated with each edge.
Voxel grid from OF, To generate voxel grid from its OF, we employ straightforward procedure. Each neural field trained to estimate the likelihood a specific voxel being full when provided with the 3D coordinates of the voxel center: Consequently, the initial step in reconstructing the fitted voxels involves constructing a grid with the desired resolution, denoted as V. Then, the field queried using the 3 centroids of this grid, and it produces Eventually, we designate voxels as occupied only their predicted probability surpasses predefined threshold, which we have empirically set to 0.4. This threshold selection has been determined through
estimated probability of occupancy for each centroid.
17

---

Published as a conference paper at ICLR 2024
Mesh from SDF
Method
Type
# Params (K)
CD (mm)
F-score (%)
Voxel
Single
554
0.19
69.1
Tri-plane
Single
0.18
68.6
Table 7: Mesh reconstruction results the Manifold40 test set (voxel grid tri-plane). We
compare hybrid representations employing tri-planes or voxel grids as discrete data structures.
UDF
SDF
OF
Method
Input
ModelNet40
ShapeNet10
ScanNet10
Manifold40
ShapeNet
CNN
Tri-plane
82.2
92.1
63.4
82.5
88.4
3D CNN
Voxel
83.7
91.6
68.1
81.1
85.2
Transformer
Tri-plane
87.0
94.1
69.1
86.8
91.8
Table Neural field classification results (voxel grid tri-plane). We compare hybrid representa- tions employing tri-planes voxel grids as discrete data structures.
experimentation, as it strikes suitable balance between creating reconstructions that are neither overly sparse nor excessively dense.
Images from RF. Given camera pose and intrinsic parameters, to render images from a RF, we employ the same volumetric rendering scheme as [i et al. (2023), outlined Appendix A overview of the procedure is the following: for each pixel location, from the camera and sample 3D points along the ray. We feed these coordinates to a (T) pair,
cast the corresponding ray
obtaining the corresponding (R, G. B, or) field values. Then, compute the volumetric rendering equation to calculate the final RGB value of the image pixel.
B.2 EXAMPLES RECONSTRUCTIONS BY TRI-PLANES
In Fig. 4 we report some examples of point clouds, meshes, voxel grids and images reconstructed from tri-plane neural fields fitted to UDFs, SDFs, OF's and RFs, respectively. For all fields, we can observe very good reconstruction quality.
B.3 COMPARISON BETWEEN RECONSTRUCTIONS BY NEURAL FIELD PROCESSING FRAMEWORKS
In Fig. 5 and Fig. 6 we show reconstructions of point clouds and meshes obtained by different frameworks used process neural fields. We can notice that neural fields in which the neural component is a shared network trained on the whole dataset, i.c. Functa and DeepSDF, cannot properly reconstruct the original explicit data. Conversely, methods relying on fitting an individual network, either large MLP (inr2vec) or tri-plane and a small MLP (ours), provide high-quality reconstructions.
C VOXEL GRID HYBRID NEURAL FIELDS
In this paper, we use tri-plane neural fields. However, other kinds of hybrid neural fields may be considered plausible alternatives. Thus, we investigate the employment of voxel grid hybrid neural fields. First, analyze their reconstruction quality and memory footprint. We report results on the Manifold40 test set in Table 7 We observe that, compared to tri-planes, voxel-based hybrid fields achieve comparable reconstruction accuracy but at the cost of much larger number of parameters, ie., 554K vs 64K.
Then, we also try to classify voxel grid hybrid neural fields. With resources (an RTX 3090 GPU), we were not able to train the Transformer architecture used for tri tri-planes. Indeed, using a voxel grid would require tokens of size 323 in our formulation (Transformer invariant the channel
18

---

Published as a conference paper at ICLR 2024
Figure 7: Tri-plane channel shuffling. Left: Mesh reconstructed from SDF tri-plane neural field. Right: Mesh reconstructed by spatially shuffling the features of the tri-plane while keeping the original MLP.
order), leading prohibitive training resources. Thus, employ 3D CNN (ResNet5O-style), Results are reported Table g We can see that although processing voxels with a 3D CNN is more expansive in terms of both storage and computation than processing tri-planes with 2D CNN, they achieve comparable classification performance (row I vs 2) without a clear winner between the two approaches. Yet, thanks to their high compactness. we can process tri-planes by Transformers, achieving significantly better performance (last row). Finally, we highlight that though the performances yielded in this experiment by hybrid voxel-based approach are inferior to tri-planes, they are still much better than those achievable by previous proposals (see Table 2 which vouches for the effectiveness of hybrid approaches when it comes to processing neural fields without compromising reconstruction accuracy.
D DEEPER INVESTIGATION ON THE TRI-PLANE AND MLP CONTENT
In this section, our objective gain a deeper understanding of the information stored in the tri tri-plane and the MIP.
D. IS THE MIP ALONE ENOUGH RECONSTRUCTION?
The first question we aim to address is whether the MI. alone can serve neural field capable of representing 3D shapes without relying on the features provided by the tri-plane. As explained in Section we utilize the 3D coordinates of point to retrieve the corresponding feature vector from the tri-planes. This latter is then concatenated with the positional encoding of the coordinates. To examine whether the 3D coordinates and the MLP alone sufficient obtain accurate outputs, conduct an experiment where we shuffle the tri-plane features along the spatial dimensions while preserving the channel orders. This means that each point p will be associated with different yet meaningful feature vector from another 3D point. the MIP is still capable of providing the correct output in this scenario, it implies that all the geometric information of the underlying 3D shapes is likely contained within the MLP, and the tri-plane is actually not necessary. Thus, we conducted reconstruction experiment of meshes from SDF, similar to that reported in Table We compared the reconstructed mesh with the ground truth by calculating the Chamfer Distance and the F-score, using 16,384 points sampled from the surface. We note that when utilizing the MLP with features shuffled spatially, we get significantly worse values for both the Chamfer Distance and the F-score (2.9mm vs 0. 6mm), which indicates poor reconstruction quality. This clearly visible in Fig. 7 This outcome highlights that the MLP alone, without the tri-plane features, is not capable of accurately reconstructing the original shape. Thus, we believe that the tri tri-plane provides essential geometric information about the represented 3D shape.
19

---

Published as a conference paper at ICLR 2024
SDF
Method
Input
Manifold40
MLP
MLP (Tri-plane)
4.3
NFN (Zhou al.
2023a
MLP (Tri-plane)
4.1
NFT Zhou ct al.
2023b
MLP (Tri-plane)
4.1
inr2vec
(De Luigi al.
2023)
MLP (Tri-plane)
7.4
Ours
Tri-plane
86.8
Table 9: Classification of MLPs of tri-plane neural fields on the Manifold40 test set. Neural fields were randomly initialized.
Figure 8: Channel visualizations. We select one of the tri-planes and visualize its 16 channels learned for an airplane.
D.2 IS THE MLP ALONE ENOUGH FOR CLASSIFICATION?
We conducted additional experiments where we treated each MLP associated with tri-plane as input classification pipeline. We utilize various methods, including simple MLP classifier, well as advanced frameworks such as NFT and inr2vec, which directly process MLPs. In this case, we completely disregarded the information provided by the tri-plane. We performed this experiment starting from random initialization. As shown in Table | even though MLPs used in tri- tri-planes are smaller than those used in other experiments, we note that directly classifying them leads to unsatisfying performances.
D.3 TRI-PLANE CHANNEL VISUALIZATIONS
To understand what information is contained in different channels of tri tri-plane feature map, try to visualize all of them for fixed shape. Fig. [ shows all channels of a SDF tri-plane feature map of SDF neural field representing airplane. We can that, although values can change across channels, the overall shape outlined and repeated each channel. This also motivates why a CNN network that not invariant the channel order can work such kind of input.
D.4 CHANNEL ORDER INVESTIGATION
In this section, we further analyze the tri-plane channel content highlight that the main difference between two training episodes concerning the same shape boils down to a permutation of the channel order. Thus, we conduct a similar experiment that illustrated in Fig. with tri-plane with only channels. We visualize results in Fig. [ also reporting the tri-plane channel visualization for both training episodes, A and B, as well as the permutation that aligns the channels found in to We discovered that permutation by minimizing the reconstruction error. We can appreciate that after the permutation, the corresponding channels contain similar features, resulting the possibility of reconstructing the shape with the Mg.
D.5 ADDITIONAL. VISTANISZTTIOSS
In Fig. 10 show additional tri-plane visualization, similar to those of Fig. (left).
20

---

Published as a conference paper at ICLR 2024
(A. MA)
(TB, MB)
(TA, MB)
(Permuted TA. MB)
Ta channels
TB channels
Permuted T' channels
Figure 9: Row 1: Reconstructions of different combinations of (tri (tri-plane, MLP) pairs with different initializations. Permuted Ta means that the channels of TA are permuted in order to minimize the reconstruction enror when combined with MR. Rows 2-4: Channel visualizations.
E IMPLEMENTATION DETAILS
E.1 DATASETS
Regarding single MLP neural fields based SIREN networks, we employ the same datasets shared by De Luigi et al. (2023) with the same train/val/test splits for SDF, UDF, and OF, When creating the tri-plane neural field datasets, we employ the original explicit datasets used create the inr2vec De Luigi et al. (223)) neural fields. We also follow the same offline augmentation protocol that employs non-unin uniform sealing along the three axes and then a re-normalization of the shape into the unit sphere to reach roughly 100K shapes in total. Thereby, the same explicit data is used to build the benchmark, while the only aspect that varies is the way to represent the neural field, i.c., MLP only vs (tri-plane, MLP). Finally, to learn RF, we adopt the dataset introduced in Xu et al. (2019). This dataset contains renderings from 13 classes of ShapeNet, and for each shape, 36 views are generated around the object and used to fit the RF. In this case, 40511 shapes with no prior augmentation. For the train/val/test splits, we randomly sample 80% of the objects for the training set and select 10% shapes for both the validation and test splits.
the original dataset that accounts for
21

---

Published as a conference paper at ICLR 2024
Figure 10: Additional visualizations. For each object, we visualize the explicit reconstruction obtained from the hybrid neural field and one tri-plane feature map for three different fields, SDF, UDF, and RF, We visualize features as normalized sum of all channels with viridis colormap.
We additionally note that the ShapeNet10 and ScanNer10 datasets mentioned in the main text are subsets of 10 (shared) classes of ShapeNet (Chang et al. 2015) and ScanNet (Dai et al. (017). respectively, originally proposed in Qin et al. 2019
E.2 BENCHMARK
In this section, we detail some implementation details and choices behind the results reported in Table by specifying how experiments were run for each one of our competitors.
DeepSDF. DeepSDF (Park al., 2019) was originally intended as a shared network architecture that learns an SDF from dataset of meshes. The results reported in Table for Manifold40 (Hu et al.) 2022) were therefore obtained by running the code from the official repository to train the framework Tor 100 epochs and compute the 1024-dimensional embeddings of training, validation, and test set. A classifier (3 fully connected layers with ReLUs) was then trained on those embeddings. For UDF results, instead, trained the framework point cloud datasets by replacing the DeepSDF loss with the binary cross entropy of Eq. (T). Extensions to OF and RF, however, are not as straightforward and were thus not included part of our work.
inr2vec. The inr2vec framework (De Luigi et al. 2023 was trained on each point cloud, mesh, and voxel dataset for 100 epochs via the official code and the resulting embeddings used to train classifier (3 fully connected layers with ReLUs). The original work by De Luigi et al. (2023) deals with UDF, SDF, and OF; an extension RF would not be trivial thus it not covered by our experiments.
DWSNet.
Navon al. 2023 test their DWSNet architecture on neural fields representing grayscale images. We modilied their official neural field classification code to work with UDF, SDF, OF, and RF, and trained their classifier (DWSNet classification head) for 100 epochs.
NFN and NFT. We modified the official code for NFN Zhou et al. 2023a) and NFT (Zhou et al. 2023b) work with UDF, SDF, OF, and RF, by using two hidden equivariant layers (16 channels each) followed by an MIPP classification head. We trained on each dataset for 80 epochs, by which time the validation loss had stopped improving.
Functa. To perform experiments with the Functa framework (Dupont et al. 2022) we start from the original code provided by authors and adapt it to fit UDF, SDF, and OF. In particular, we use latent modulation of size 512. At each iteration, we use 10k points to compute the loss. The inner loop of the meta-le leeanlee process optimized for three iterations, and in total, we optimize for 5e steps with batch size 2. After the training phase, the weights of the shared network are frozen, and we execute the inner loop of the meta-learnig protocol for three steps to obtain the latent modulation vector for each field.
22

---

Published as a conference paper at ICLR 2024
flatten
FC
max pool
Classification
Class logits for tri-plane
FC
Transformer Encoder
Tranformer Decoder
shared
Segmentation
Class logits for p
FC
H X W
class embedding
Figure 11: Architecture for tri-plane processing
E.3 ARCHITECTURES
We provide here some additional details regarding the Transformer architectures used for classification and segmentation depicted in Fig. 11 In both cases, the channels of the tri-planes are flattened and linearly projected to 512-dimensional vectors and fed to the Transformer encoder, which is the original Transformer encoder proposed in Vaswani al. (2017). In our case, the encoder consists of 4 heads with 8 layers each. The encoder produces a single embedding for each tri-plane channel, and for classification, use a max pool operator achieve invariance the channel order and obtain single global embedding. The classifier consists of a single fully connected layer that yields the predicted probability distribution for the input field. In total, the number of parameters is roughly 25M, which is very similar to the number of parameters of a Resnet50. As regards part segmentation, we attach an additional Transformer decoder composed of parallel heads with layers each. The decoder takes in input the sequence of tokens from the encoder and the sequence of tokens obtained starting from the coordinates query to be segmented. More precisely, each point is first encoded using positional encoding from Mildenhall et al. (2020b) that gives 63-dimensional vector, and then linearly project it into a 512-dimensional vector. Finally, given an object to segment, we also attach The decoder output is sequence of transformed tokens that are given in input to a single layer to predict the final probability distribution. Note that we do not the decoder rugo-regressively in the original Transformer. We just compute a forward pass on batch of query points to segment all of them simultancously. Hence, we also do not use masked attention at training time.
the one-hot encoding of its class (standard practice in
E.4 TRAINING
We provide here additional training details. For classification, we use the same hyperparameters for all experiments. We adopt the AdamW optimizer (Loshchilov & Hutter 2017). with the OneCycle scheduler (Smith & Topin 2019) with maximum learning rate of Te-4, and train for 150 epochs with batch size set to 256 using the cross entropy loss. We use single NVIDIA RTX 3090 for all experiments. At training time, we apply a random crop of size 30 x 30 to the tri-planes that have a resolution of 32 x 32. As for part segmentation, we use the same configuration although we train for 75 epochs with batch size 32. For training and testing, we use the same protocol used by competitors. Indeed, training time, object parts, although at test time, metries are computed by selecting the highest scores among the correct object parts for given class.
for classification,
the cross entropy loss function with all the
23

---

Published as a conference paper at ICLR 2024
E.5 RANDOM INITIALIZATION
Throughout the main paper, we use the term "random initialization" to differentiate our setup from that considered in De Luigi et al. (2023), where all neural fields are trained starting from the same, albeit random, set of parameters, i.c. initial values of weights and biases are sampled once and then used as the starting point fit all MLPs, In our work, instead, we consider the more realistic scenario in which each individual neural field is trained starting from a random and different set of parameters. Nonetheless, when sampling initial values of weights and biases for each shape, we do indeed follow the specific initialization scheme suggested for SIRENs (Sitzmann et al.) 2020). In other words, "random initialization" means that a different random set of parameters, sampled according to the SIREN rules, is used to initialize each MLP. Tri-planes, instead, are initialized with values sampled from a Gaussian distribution.
F TRAINING AND INFERENCE TIME
In this section, we provide additional details on training and inference times, computed on a single NVIDIA RTX 3090.
SDF
Method
Time (hours)
DeepSDF (Park et al. 2019)
61
Functa (Dupont et al. MLP
2022
120
80
Tri-plane (Ours)
Table 10: Training time comparison. Time required to fit approximately 100K shapes the
Mandifold40 training set. Times are computed on a single NVIDIA RTX 3090.
UDF
Time (seconds)
Method
2048 points
16K points
32K points
64K points
PointNet QQ et al.
2017a)
0.002
0.004
0.006
0.012
* Qi al. 2017a)
0.099
0.764
1.388
2.793
Ours
0.003
0.003
0.003
0.003
Table 11:
Inference time classify an input shape. indicates that the time reconstruct the point cloud from the neural field is included. Times are computed on a single NVIDIA RTX 3090.
Table TO shows comparison between the times required fit approximately 100K shapes the Manifold40 (Hu et al. 2022) training set for different methods. For neural fields consisting of an MLP only and (tri-plane, MLP) pair, denoted by "MLP" and "Tri-plane" respectively, we fit each shape for 600 steps. Functa was trained with the original hyperparameters for 500K iterations for the meta-learning outer loop. DeepSDF was trained for 100 epochs. Notice how tri-plane hybrid neural fields are the ones requiring the least amount of time to fit the dataset.
Table II compares the inference time required to classify an input shape with different strategies. The first row is the classification time of PointNet processing point cloud. In the second row, we report the inference time of PointNet, assuming we do not have an explicit point cloud available. namely PointNet*. In this case, the time includes reconstructing the point cloud from the tri-plane neural field, which would be the only data available in our scenario. Finally, the last row reports the time required by our method, i.e., Transformer processing input hybrid tri-plane neural field. We show times for different point cloud resolutions (2048, 16K, 32K, 64K points). We notice that the inference time of our method is constant along the resolution axis; in particular, comparable to the PointNet at low resolution, whereas becomes increasingly faster as the resolution grows. Moreover, we highlight that method inference times w.r.t. to those of PointNet directly
method is
24

---

Published as a conference paper at ICLR 2024
processing explicit point clouds comparable lower resolutions and even better at higher ones (e.g., 0.003 Ours vs 0.012 PointNet for 16K points).
G TRI-PLANE ABLATIONS
This section provides additional ablations concerning the tri-plane structure and hyperparameters. Table [2 shows that sharing the MLP across tri-planes leads to inferior results in both classification and reconstruction on Manifold40 (Hu al. 2022). It also worth highlighting how sharing the MLP requires its availability at test time create neural fields, i.c., to create new test data, limiting the deployment scenarios of our methodology, which are instead equivalent to those of discrete data structures when using (MLP, triplane) pair for cach sample. Table 13 provides comparative analysis classification and reconstruction results on Manifold40 (Hu et al. 2022) between tri- planes with different resolution and/or number of channels. Interestingly, the classification accuracy and reconstruction егrоr are quite robust to the number of channels and tri-plane resolution, which therefore are not critical design hyperparameters.
SDF
Mesh from SDF
Classification
Reconstruction
Method
Type
Accuracy (%)
CD (mm) +
F-score (%)
Tri-plane
Shared
84.7
1.57
42.9
Tri-plane (Ours)
Single
86.8
0.18
68.6
Table 12: Shared vs individual MLP. Comparison of classification and reconstruction results of tri-planes sharing all the same MLP vs when each tri-plane has its individual MLP. Results were computed on the Manifold40 test set.
SDF
Mesh from SDF
Resolution
Channels
Accuracy (%) f
CD (mm) +
F-score (%) f
32 x 32
32
86.3
0.18
68.6
32 32
16
86.8
0.18
68.8
32 x 32
8
86.4
0.18
69.2
24 x 24
16
86.6
0.18
68.9
40 x 40
16
86.4
0.18
69.0
Table 13: Ablation study of tri-plane resolution and number channels. Second row is our choice of tri-plane size. Results were obtained the Manifold40 test set.
Accuracy (%)
Method
Input
ModelNee-0
ShapeNet10
ScanNet10
Manitold40
ShapeNet10
Tri-plane
87.0
94.1
69.1
86.8
91.8
PointNet al 2017a
Point Cloud
88.8
94.3
72.7
PointNet
11
Cloud
McshWalker Liaay & Tal
2020
Mesh
90.0
MeshWalker*
Mesh
90.6
Cons3DNet
Maturana & Scherer 2013
Voxel
92.1
Conv3DNet
Voxel
92.5
Table 14: Explicit methods on reconstructed data original discrete data. Classification accuracy
on the test set is reported. Top: our method, that processes (tri-planc) neural fields. Bottom: Methods that process explicit discrete 3D data, either reconstructed from the corresponding neural field (without *) or by operating directly on the original data (with *).
25

---

Published as a conference paper at ICLR 2024
H EVALUATING ON THE ORIGINAL DISCRETE 3D REPRESENTATIONS
In Table 3 of the main paper, evaluate the competitors on the data reconstructed from the tri-plane neural fields fitted on the test sets of the considered datasets since these would be the only data available at test time in the scenario described in Section [ originally proposed by De Luigi et al. (2023), where neural fields are used to store and communicate 3D data. Nonetheless, we compare classification performance of methods operating and when it is, instead, reconstructed from the corresponding neural field. Table shows slightly better results when the original data is used. However, the difference is small enough to prove that no significant loss of information occurs when storing data with tri-plane neural fields.
discrete data both when the original data is used
I STUDY ON THE MEMORY OCCUPATION OF NEURAL FIELDS
Dataset
Representation
Num Shapes
Memory (GB)
ModeINet40
Point Cloud
110054
2.52
ModelNet40
Siren MLP (UDF)
110054
327.99
ModelNet40
Tri-plane (UDF)
110054
20.15
ShapeNet10
Point Cloud
103230
2.36
ShapeNet10
Siren MLP (UDF)
103230
307.65
ShapeNet10
Tri-plane (UDF)
103230
18.90
ScanNet10
Point Cloud
108360
2.48
ScanNet10
Siren MLP (UDF)
108360
322.94
ScanNer10
Tri-plane (UDF)
108360
19.84
Manifold40
Mesh
100864
10.79
Manifold40
Siren MLP (SDF)
100864
300.60
Manifold40
Tri-plane (SDF)
100864
18.47
ShapeNet10
Voxel
103230
25.20
ShapeNet10
Siren MLP (OF)
103230
307.65
ShapeNet10
Tri-plane (OF)
103230
18.90
ShapeNetRender
Training Images Siren MLP (NeRF)
4051
204.45
ShapeNetRender
40511
120.73
ShapeNetRender
Tri-plane (NeRF)
40511
7.42
Table 15: Training dataset memory occupancy
Nowadays, datasets typically made of data represented explicitly, e.g., a point cloud dataset, multi-view image dataset, and so on. However, in our vision, these datasets might be replaced by their neural field counterparts, i.e., each element would be represented neural field. Thus, this section investigates the trade-off of using these novel representations regarding memory occupancy.
First, we investigate the memory in GB required by each raw dataset employed in our paper. We report the results in Table [5] by their original explicit representation or with neural fields. As neural field representation,
show the SIREN MLP adopted by our competitors (e.g.. De Luigi et al. (2023) or our tri tri-plane representation. We note that the tri-plane representation requires more memory than point clouds and meshes (e.g.. ModelNet40 and Manifold40). Yet, the advantages of using tri-plane representations stand out when compared with either voxels (ShapeNet ()) or images (ShapeNetRender)) Notably, tri-planes take significantly less memory than Siren MI.Ps.
However, we argue that the real advantages of using neural fields are related to the memory occupied by the data being decoupled from its spatial resolution. Thus, in Fig. 12 we study the number of parameters required for different explicit representations compared to those of neural fields by varying the spatial resolution of the data. We include all variables required by an explicit representation in their parameter number, e.g., each point of point cloud has 3 parameters: its 3 coordinates æ, y. and z. The blue, red, and green lines represent the parameters the explicit, the SIREN MLP, and the tri-plane representation when changing the resolution. Regarding meshes and point clouds,
26

---

Published as a conference paper at ICLR 2024
Point cloud vs Neural Field
Triangle Mesh vs Neural Field
800000
Number of Parameters
Number Parameters
Point cloud Tri plane
Mesh SIREN Tri-pland
64000
64000 26710
6144
21333
266666
1,2222
177777
2048
6380
(ShapeNet10)
(Manifoid40)
Number of 3D Points
Number of Triangles
Voxel Grid vs Neural Field
NeRF Training Images vs Neural Field
Images
SIREN MLP
Tri-plane
Nurrbee Parameterss
Number of Parameters
Voxel SIREN MLP
Tri-plane
800000
61000
0
64000
86 x 86
224 x 224 x [ShapeNetRender)
(ShapeNet10) Voxel Grid Size
x 24
x W Numimaage ages
Figure 12: Number of parameters relation to spatial resolution: neural fields vs explicit representations
notice that the space occupied by tri-plane neural fields is slightly larger for the resolutions used in the datasets. However, even with a point cloud of 21,333 points and mesh with 14,222 faces, using tri- tri-plane neural fields to represent the data becomes ad advantageous. This is of utmost importance, considering that real datasets could contain point clouds or meshes with many more points or faces e.g., Objaverse (Deitke et al. 2023 features meshes with more than 10" polygons. The advantages are even more significant in the case of voxel grids, which the memory occupancy scales cubically with the spatial resolution NeRFs in which many training images per sample are required. Finally, we point out that the representation with tri-plane neural fields is advantageous compared to that with SIREN MLPs.
27


Reviews:
[{"review_id": "NRPlwPikYA", "reviewer": "Reviewer_3J4Y", "summary": "The paper presents a method to encode discrete 3D information into the effective continuous triplane, allowing for the larger vision transformer to perceive in 3D. The paper validates their methods in both classification and segmentation tasks -- much better then previous version that used MLP.", "questions": "Please address the question above", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "I like the topics this paper explores. Instead of directly consuming raw and messy 3D data, we could represent that data using neural representation, which will make the network design much easier. \n\nThe presented method achieves the significantly improved results compared with baselines. Although the proposed method is not novel -- simply replacing MLP with the more effective triplane, it shows better performance than the network that takes in raw discrete 3D dataset, which suggests a new paradigm to deal with 3D data.", "weaknesses": "Parameter and time efficiency comparison is missing. We know that triplanes work better than global MLP. However, there was no free lunch. Triplane-based is usually parameter-intensive. So I’m concerned that the triplane based representation would consume lots of space compared with the original dataset. And the paper doesn’t report any comparison. \n\nAlso I notice that the paper uses the explicit extracted from the learned triplane in the classification tasks of Table3. I’m not very sure if it makes sense. Although the triplane is very effective, there's still information loss compared with original data. I would suggest the author justify it a bit. \n\nThe random initialization of triplane and MLP  is concerning. The reason is that the method uses the Sine/Cosine as the activation function whereas random initialization is not preferred as stated in other works. Instead, a specific way of random initialization was suggested in other papers like the SIREN paper. Also, the channel order with respect to the initialized value is not very clear. \n\nRegarding the dataset, I’m not very sure what's ScanNet10 used in the paper. It seems unexplained. \n\nIn short, I tend to hold this paper and vote for weakly reject. I'm really looking forward to hearing back from authors during the rebuttal and clarify about my concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JWMpGfJKlB", "reviewer": "Reviewer_ue4V", "summary": "The paper investigates the utilization of learned neural fields as a means of representing objects for classification and part segmentation tasks. The authors propose a hybrid NeRF (Neural Radiance Field) that combines a tri-planes data structure with an MLP (Multi-Layer Perceptron) for object encoding. This hybrid approach encompasses various fields representations, including Sign/Unsigned Distance Functions (SDF/UDF), Occupancy, and Radiance fields.\n\nThe experiments conducted in the study reveal that the representation learned through tri-plane parameters remains nearly identical (up to channel permutation) even when the NeRF is trained on the same data but with different random initializations. This robustness greatly simplifies the deployment of this method in comparison to previous approaches. Additionally, the experiments demonstrate that the classification and part segmentation performance achieved through tri-plane features is on par with specialized models designed for processing explicit object representations, such as point clouds or meshes.", "questions": "- How do you explain the relatively lower performance of model when is trained and tested on the same neural field, like UDF and SDF as shown in Table 4? This seems to be counterintuitive.\n\n- As experiments show (e.g. Fig.3 right), the object representation mostly is encoded by the tri-plane parameters, rather than the MLP. This raises a question of whether the MLP can be shared across data samples for efficiency?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed approach introduces a versatile method for encoding object representations across various neural fields. \n\n- Classification using learned tri-plane features demonstrates superior performance compared to other existing NeRF encoding methods that rely solely on MLP parameters. \n\n- The authors also explore different techniques for reshaping tri-plane feature tensors to ensure that predictions remain invariant to channel permutations.", "weaknesses": "- The proposed method necessitates per-object optimization to acquire individual tri-plane features. The author conducted a comparative analysis of object reconstruction using this technique against other solutions, which employed a shared network trained on the entire dataset, like Functa (Dupont et al.). They reported the performance and the number of parameters (see Table 1). However, it is worth noting that the required computational resources, particularly in terms of training time, have not been explicitly reported or discussed. \n\n- In the ablation study, as presented in Table 6, the primary focus is on investigating various architectural aspects related to the classification of the learned tri-plane representations. While this provides valuable insights, it's important to highlight that certain variations within the proposed method, such as utilizing a shared MLP with distinct tri-planes across data, adjusting spatial resolution, or varying the number of channels within tri-plane structures, have not been subjected to ablation analysis. Addressing these aspects could provide a more comprehensive understanding of the method's performance and potential optimizations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KqNhZWMhcl", "reviewer": "Reviewer_fdy7", "summary": "The authors propose to use optimized triplanes for downstream tasks on neural fields, such as 3D object classification or part segmentation on various neural field types, such as unsigned distance fields, signed distance fields, occupancy fields, or radiance fields. It is shown that using triplanes in this scenario leads to a clearly better trade-off between reconstruction quality and downstream task accuracy, in comparison to previous works that utilize a latent code from a shared MLP or MLP weights as descriptors.\n\nFurther, the authors expose that architectures that are invariant to channel order in the fitted representations are important to achieve optimal accuracy in downstream tasks.\n\nAlso, the authors provide a benchmark for downstream tasks on triplane representations, which they plan to make publicy available.", "questions": "The use of instance-specific MLPs as decoders for triplanes makes sense in terms of reconstruction quality. However, this also leads to some information being represented in the MLP, instead of the triplane. I wonder how the downstream task quality is changing when a shared MLP is used. This experiment seems to be missing in the paper and I would be interested to see the comparison.\n\n---------\nI thank the authors for the replies to my concerns and also for elaborating on the general motivation of the research direction.\nThe main critique of other reviewers seem to go into a similar direction - questioning the usefulness of the proposed approach. I still agree to some degree but I also see that there might be potential applications in the future. That aside, I think this paper does something novel and analyses it well, which is why I will keep my score.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The idea of using fitted triplanes as embeddings for downstream tasks is simple but seems effective and has not been analyzed before to my knowledge.\n- The results clearly show the better trade-off compared to previous methods.\n- The method is evaluated on a diverse set of tasks and function representations.\n- The insight regarding channel invariance is interesting and leads to the conclusion that transformers are better than CNNs, which seem unintuitive at first.\n- The authors provide a benchmark datasets for the community to test architectures on.\n- The paper is well written and easy to understand.", "weaknesses": "- The idea of using fitted triplanes for downstream tasks like classification is \"obvious\" in a sense.\n- There is the general question of what the relevant application of the proposed approach might be. This is a problem for all methods that aim to solve downstream tasks on optimized neural field representations. Usually, data (images or point clouds, etc) was used to obtain the neural field in the first place. Solving the downstream tasks on this input representation instead of the neural field usually leads to better results, which is also confirmed by an experiment in the paper. The gap is reduced a lot though, in comparison to previous works.\n- I think the term \"universal neural field classifier\" (which the authors claim their method to be) is misleading, as the method is not for all neural fields but only for those represented as triplanes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wggsssnKaW", "reviewer": "Reviewer_hFtS", "summary": "This paper defines a benchmark that covers a set of fields including occupancy, signed and unsigned, and radiance fields. The authors show that applying well established archtieures on triplanes achieves better results that processing neural fields realized as a large MLP.", "questions": "See the questions above.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. A benchmark for triplane neural field classification.\n\n2. The motivation of creating this benchmark is interesting and makes sense.", "weaknesses": "1. The paper is difficult to follow.\n\n2. The presentation has room to improve.\n\n3. The proposed method performs worse than existing point cloud methods as shown in Table 5.\n\n4. It is not obvious on the advantage of the proposed method over methods working with point clouds and / or meshes.\n\n5. How well does the proposed method generalize to unseen scenes?\n\n6. Does the proposed method have a faster run time compared to mesh / point cloud methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)