Deep Learning on Implicit Neural Representations of Shapes
Authors: Luca De Luigi
Adriano Cardace
Riccardo Spezialetti
Pierluigi Zama Ramirez
Samuele Salti
Luigi di Stefano
Venue: ICLR 2023 poster (2023)

Abstract: Implicit Neural Representations (INRs) have emerged in the last few years as a powerful tool to encode continuously  a variety of different signals like images, videos, audio and 3D shapes. When applied to 3D shapes, INRs allow to overcome the fragmentation and shortcomings of the popular discrete representations used so far. Yet, considering that INRs consist in neural networks, it is not clear whether and how it may be possible to feed them into deep learning pipelines aimed at solving a downstream task. In this paper, we put forward this research problem and propose inr2vec, a framework that can compute a compact latent representation for an input INR in a single inference pass. We verify that inr2vec can embed effectively the 3D shapes represented by the input INRs and show how the produced embeddings can be fed into deep learning pipelines to solve several tasks by processing exclusively INRs.

Published as a conference paper at ICLR 2023
DEEP LEARNING ON IMPLICIT NEURAL REPRESENTATIONS OF SHAPES
Luca De Luigi", Adriano Cardace", Riccardo Spezialetti University of Bologna
(luca.deluigi.. adriano. edriane..arrdace/uuiioo. it, riccardo spezia lett! 1 bb.
Pierluigi Zama Ramirez, Samuele Salti, Luigi Di Stefano University of Bologna
ABSTRACT
Implicit Neural Representations (INRs) have emerged in the last few years as a powerful tool to encode continuously variety of different signals like images, videos, audio and 3D shapes. When applied to 3D shapes, INRs allow to overcome the fragmentation and shortcomings the popular discrete representations used so far. Yet. considering that INRs consist in neural networks, it is not clear whether and how it may be possible to feed them into deep learning pipelines aimed at solving downstream task. In this paper, put forward this research problem and propose inr2vec, a framework that can compute a compact latent representation for input INR in single inference pass. We verify that i inr2ves can embed effectively the 3D shapes represented by the input INRs and show how the produced embeddings can be fed into deep learning pipelines to solve several tasks by processing exclusively INRs.
INTRODUCTION
Since the early days of computer vision, researchers have been processing images stored two- dimensional grids of pixels carrying intensity or color measurements. But the world that surrounds us is three dimensional, motivating researchers try to process also 3D data sensed from surfaces. Unfortunately, representation of 3D surfaces in computers does not enjoy the same uniformity as digital images, with a variety of discrete representations, such as voxel grids, point clouds and meshes, coexisting today. Besides, when it comes to processing by deep neural networks, all these kinds of representations are affected by peculiar shortcomings, requiring complex ad-hoe machinery (Qi et al.
2017b Wang et al. 2019b; Hu et al. 2022) and/or large memory resources (Maturana & Scherer 2015
Hence, no standard way to store and process 3D surfaces has yet emerged.
Recently, new kind of representation has been proposed, which leverages on the possibility of deploying a Multi-Layer Perceptron (MLP) to fit a continuous function that represents implicitly a signal of interest (Xie al. 2021). These representations, usually referred to as Implicit Neural Representations (INRs), have been proven capable of encoding effectively 3D shapes by fitting signed distance functions (sdf) (Park al. 2019 Takikawa et al.) [220 Gropp al.) 2020), unsigned distance functions (udf) (Chibane et al.) 2020) and occupancy fields (occ) (Mescheder et al., 2019 Peng al. 2020). Encoding 3D shape with continuous function parameterized as an MLP decouples the memory cost of the representation from the actual spatial resolution, i.e., surface with arbitrarily fine resolution can be reconstructed from a fixed number of parameters. Moreover, the same neural network architecture can be used to fit different implicit functions, holding the potential to provide a unified framework to represent 3D shapes.
Due to their effectiveness and potential advantages over traditional representations, INRs are gathering ever-increasing attention from the scientific community, with novel and striking results published more and more frequently (Müller et al. 2022 Martel et al., 2021 Takikawa al. 202 1 Liu et al. 2022). This lead us to conjecture that, in the forthcoming future, INRs might emerge as a standard
"Jiint first authorship. We thank also Francesco Ballerini for the results produced during his master thesis.

---

Published as a conference paper at ICLR 2023
representation to store and communicate 3D shapes, with repositories hosting digital twins of 3D objects realized only as MLPs becoming commonly available.
An intriguing research question does arise from the above scenario: beyond storage and communi- cation, would it be possible to process directly INRs of 3D shapes with deep learning pipelines to solve downstream tasks is routinely done today with discrete representations like point clouds or meshes? other words, would it be possible to process an INR of a 3D shape to solve a downstream task, e.g., shape classification, without reconstructing discrete representation of the surface?
Since INRs are neural networks, there is no straightforward way to process them. Earlier work in the field, namely OecupancyNetworks (Mescheder al., (0T9) and DeepSDF (Park al., (0T9). fit the whole dataset with shared network conditioned on a different embedding for each shape. In such formulation, the natural solution to the above mentioned research problem could be to use such embeddings as representations of the shapes in downstream tasks. This is indeed the approach followed by contemporary work (Dupont et al.) 2022). which addresses such research problem by using embedding latent modulation vector applied to shared base network. However, representing a whole dataset by a shared network sets forth a difficult learning task, with the network struggling in fitting accurately the totality of the samples (as we show in Appendix A). Conversely, several recent works, like SIREN (Sitzmann et al. 2020b) and others (Sitzmann et al. [2020a] Dupont et al. 202 SSrümpper et al. 2021 Zhang et al. 2021, Tancik et al., 2020) have shown that, by fitting an individual network to each input sample, one can get high quality reconstructions even when dealing with very complex 3D shapes or images. Moreover, constructing individual INR for each shape is easier to deploy in the wild, as availability of the whole dataset is not required to fit an individual shape. Such works are gaining ever-increasing popularity and we led believe that fitting an individual network is likely to become the common practice in learning INRs.
Thus, in this paper, we investigate how perform downstream tasks with deep learning pipelines on shapes represented as individual INRs. However, single INR can easily count hundreds of thousands of parameters. though is well known that the weights of deep model provide vastly redundant parametrization of the underlying function (Frankle & Carbin 2018 Choudhary et al.) 2020). Hence, we settle on investigating whether and how an answer to the above research question may be provided by representation learning framework that learns to squeeze individual INRs into compact and meaningful embeddings amenable to pursuing a variety of downstream tasks.
Our framework, dubbed inr2vec and shown in Fig. T has at its core an encoder designed to produce a task-agnostic embedding representing the input INR by processing only the INR weights. These embeddings can be seamlessly used in downstream deep learning pipelines, as validate experimentally for a variety of tasks, like classification, retrieval, part segmentation, unconditioned generation, surface reconstruction and completion. Interestingly, since embeddings obtained from INRs live in low-dimensional vector spaces regardless of the underlying implicit function, the last Iwo tasks can be solved by learning a simple mapping between the embeddings produced with our framework, e.g., by transforming the INR of a udf into the INR of an sdf. Moreover, inr2ves vec can learn a smooth latent space conducive to interpolating INRs representing unseen 3D objects. Additional details and code can be found at pntttss: http::::///vaabuuiniiii Our contributions can be summarised as follows:
qithub. 1o/inr2ved
we propose and investigate the novel research problem of applying deep learning directly on individual INRs representing 3D shapes;
to address the above problem, we introduce inr2vec, framework that can be used to obtain meaningful compact representation of input INR by processing only its weights, without sampling the underlying implicit function;
we show that a variety of tasks, usually addressed with representation-speciff and complex frameworks, can indeed be performed by deploying simple deep learning machinery INRs embedded by inr2vec, the same machinery regardless of the INRs underlying signal.
RELATED WORK
Deep learning on 3D shapes. Due to their regular structure, voxel grids have always been appealing representations for 3D shapes and several works proposed to use 3D convolutions to perform both discriminative (Maturana & Scherer 2015 Qi
al. 2016 Song & Xiao 2016) and generative tasks
2

---

Published as a conference paper at ICLR 2023
INRs provide an unified representation for 3D shapes
inr2vec: Embedding INRs into compact latent codes
2nr2vec embeddings can be fed to downstream tasks
Phint COLLd
Trianglo Keen
Houl
INR.
CLASSIIGATON
GRd
URCOROIT OONE
INR
DENUMA KWW
AFTSEVA
INR
FAXT PEMMENIAAIO
INR
SURFICE
IN
INCOINTTLACTION
COMPLETION
INR,
INR
INR
INR
INRs hold the potential to provide an unified representation for 3D shapes. Center: Our framework, dubbed inr2vec, produces compact representation for an input INR by looking only at its weights. Right: 2nr2vec embeddings can be used with standard deep learning machinery to solve a variety of downstream tasks.
Figure 1: Overview of our framework. Left:
(Choy et al. 2016
Girdhar et al. 2016 Jimenez Rezende et al. 2016 Stutz & Geiger 2018 Wu
et al.] 2016 2015 The huge memory requirements of voxel-based representations, though, led researchers to look for less demanding alternatives, such as point clouds. Processing point clouds, however, is far from straightforward because of their unorganized nature. As a possible solution, some works projected the original point clouds to intermediate regular grid structures such as voxels (Shi et al.) (220) or images You et al.) 2018 Li et al., 2020a). Alternatively. PointNet (Qi et al., 2017ab proposed to operate directly on raw coordinates by means of shared multi-layer perceptrons followed by max pooling to aggregate point features. PointNet++ (Qi et al. (017b) extended PointNet with hierarchical feature learning paradigm to capture the local geometric structures. Following PointNet++, many works focused on designing new local aggregation operators (Liu et al. 2020).
resulting in wide spectrum of specialized methods based on convolution (Hua et al., [2188] Xu
et al. 2018: Atzmon et al. 2018 Wu et al. 2019 Fan et al. 2021: Xu et al. [021 Thomas et al. 2019), , graph (Li et al. 2019 Wang et al. 2019a b), and attention (Guo et al. 2022 Zhao et al. 2021) operators. Yet another completely unrelated set of deep learning methods have been developed to
process surfaces represented as meshes, which differ in the way they exploit vertices, edges and faces
as input data (Hu et al. 2022). Vertex-based approaches leverage the availability of regular domain to encode the knowledge about points neighborhoods through convolution or kernel functions (Masci)
et al. 2015; Boscaini et al. 2016, Huang et al. 2019 Yang et al.) 2020. Haim et al., 2019 Schult eet al. 2020 Monti et al. (mminnov Solomon 2021). Edge-based methods take advantages of
these connections to define an ordering invariant convolution (Hanocka et al., (019). to construct
a graph on the input meshes (Milano et al. 2020) or to navigate the shape structure (Lahav & Tal 2020). Finally. Face-based works extract information from neighboring faces (Lian et al. [0019]
Feng et al. 2019 Li et al., 20206] Hertz et al. 2020). In this work, we explore INRs as a unified representation for 3D shapes and propose framework that enables the use of the same standard deep learning machinery to process them, independently of the INRs underlying signal.
3D Implicit neural representations (INRs). Recent approaches have shown the ability of MLPs to represent implicitly diverse kinds of data such as objects (Genova et al. 2019 2020 Park et al. 2019
Atzmon Lipman| 2020 Michalkiewicz al. |019 Gropp et al. (020) and scenes (Sitzmann et al.
2019 Jiang et al.) 2020; Peng et al. [2020] Chabra et al.) 2020). The works focused on representing
3D data by MLPs rely on litting implicit functions such as unsigned distance (for point clouds), signed
distance (for meshes) (Atzmon & Lipman 2020 Park et al. 2019; Gropp et al. 2020; Stizmann
al. Diang et al. [2020; Peng et al., (020) or occupancy (for voxel grids) (Mescheder et al.
2019 Chen & Zhang, 2019). In addition to representing shapes, some of these models have been
extended to encode also object appearance (Saito et al., 2019 Mildenhall et al. 2020) Sitzmann et al. 2019; Occhsle et al. 2019 Niemeyer et al. 2020). to include temporal information (Niemeyer ct al. 2019). Among these approaches, sinusoidal representation networks (SIREN) (Sitzmann et al. 2020b) periodical activation functions capture the high frequency details of the input data. In our work, we focus on implicit neural representations of 3D data represented as voxel grids, meshes and point clouds and we use SIREN as main architecture.
Deep learning on neural networks. Few works attempted to process neural networks by means of other neural networks. For instance, (Unterthiner al. 2020) takes as input the weights of network
3

---

Published as a conference paper at ICLR 2023
inr2vee ENCODER
w; ge
L 1 hidden layers
g
1. linear transforms: Weights W. & Rire
gdsa
Mox pooling
Blases b, R
by c gatt
shared
embedding
Stack weights and biases P g (8((
W, ga00
IMaAR BATCH NOW
by
Figure 2: inr2vec encoder architecture.
and predicts its classífication accuracy. (Schürholt et al. 2021) learns a network representation with self-supervised learning strategy applied on the N N-dimensional weight array, and then uses the learned representations to predict various characteristics of the input classifier. (Knyazev et al. [022 Jacckle & Kumar 202T; Lu & Kumar 2020) represent neural networks computational graphs, which are processed by a GNN to predict optimal parameters, adversarial examples, or branching strategies for neural network verification. All these works see neural networks algorithms and focus on predicting properties such as their accuracy. On the contrary, we process networks that represent implicitly 3D shapes perform variety of tasks directly from their weights, i.e., we neural networks as input/output data. To the best of our knowledge, processing 3D shapes represented as INRs has been attempted only contemporary work (Dupont al. 2022). However, they rely on shared network conditioned on shape-speciffe embeddings while we process the weights of individual INRs, that better capture the underlying signal and are easier deploy in the wild.
3 LEARNING TO REPRESENT INRS
The research question we address in this paper is whether and how can we process directly INRs to perform downstream tasks. For instance, can we classify a 3D shape that is implicitly encoded in an INR? And how? As anticipated in Section [ we propose to rely on a representation learning framework to squeeze the redundant information contained in the weights of INRs into compact latent codes that could be conveniently processed with standard deep learning pipelines.
Our framework, dubbed inr2vec, is composed of an encoder and decoder. The encoder, detailed in Fig. [ is designed to take as input the weights of an INR and produce a compact embedding that encodes all the relevant information of the input INR. A first challenge in designing an encoder INRs consists in defining how the encoder should ingest the weights as input, since processing naively all the weights would require huge amount of memory (see Appendix E Following standard practice (Sitzmann et al. 2020bba a Dupont et al. 2021a Strümpler et al. 2021]] Zhang consider INRs composed of several hidden layers, each
et al. 2021), the linear transformation between two consecutive layers is parameterized by a matrix of weights W, E # and a vector of biases b; E I Thus, stacking W; and b," T.T the mapping between two consecutive layers can be represented by a single matrix 2 E (I++1)) For an INR composed consider the L linear transformations between them. Hence, stacking all
with 77 nodes, iee,
of L + hidden layers, the L matrices P; IR. (
I,..,, L, between the hidden layers obtain a single matrix P E IR ())+ that we use to represent the INR in input to inr2vec encoder. We discard the input and output layers in our formulation as they feature different dimensionality and their use does not change inr2vec performance, as shown in Appendix
The inevvec encoder designed with a simple architecture, consisting of series of linear layers with batch norm and ReLU non-linearity followed by final max pooling. At each stage, the input matrix is transformed by one línear layer, that applies the same weights to each row the matrix. The final max pooling compresses all the rows into a single one, obtaining the desired embedding. It worth observing that the randomness involved in fitting an individual INR (weights initialization, data shuffling. etc.) causes the weights in the same position in the INR architecture not to share the same role across INRs. Thus, inr2vec encoder would have to deal with input vectors whose elements capture different information across the different data samples, making impossible train
4

---

Published as a conference paper at ICLR 2023
Pi E f: unsigned distonce Junction
inr2vea: training
inr2vec: inference
(Pss)
W
W
PN.)(pw)
INR
((.)
W2
W
W
W
Jnccas Miax Pool
Implict Decoder
Linears
c(f(p.)
(unseen)
Figure 3: Training and inference of our framework. Left: We consider shapes represented as INRs. As example, show INR fitting the udf of surface. Center: inr2vee encoder is trained together with an implicit decoder to replicate the underlying 3D signal of the input INR. Right: At inference time, the learned encoder can be used obtain compact embedding from unseen INRs.
INPUT
mn2veo
INFUT
INPUT
T22/20
interpolation factor
INR
OUTPUT
OUTPUT
OUTPUT
PONH
MESH
S
VOKEES
(a) inr2vec reconstructions.
(b) inr2vec interpolations.
Figure Properties of inr2vec latent space.
the framework. However, the use of a shared, pre-computed initialization has been advocated as good practice when fitting INRs, e.g., reduce training time by means of meta-learned initialization vectors, as done in MetaSDF (Sitzmann et al. 2020a) and in the contemporary work exploring processing of INRs (Dupont al. 2022). or to obtain desirable geometric properties (Gropp et al.) 2020). We empirically found that following such a practice, i.e., initializing all INRs with the same random vector, favours alignment of weights across INRs and enables convergence of our framework.
In order to guide the inr2vec encoder to produce meaningful embeddings, we first note that we are interested in encoding the values of the input weights in the embeddings produced by our framework, but, rather, in storing information about the 3D shape represented by the input INR. For this reason, we supervise the decoder replicate the function approximated by the input INR instead of directly reproducing its weights. In particular, during training, we adopt an implicit decoder inspired by (Park et al.) 2019), which takes in input the embeddings produced by the encoder and decodes the input INRs from them (see Fig. center). More specifically, when the inr encoder processes given INR, we use the underlying signal create set of 3D queries p.. paired with the values f(p) of the function approximated by the input INR at those locations (the type of function depends on the underlying case of
would be the case in standard auto-encoder formulation.
signal modality, can be udf in case of point clouds, sdf in case of triangle meshes or voxel grids). The decoder takes in input the embedding produced by the encoder concatenated with the 3D coordinates of query P; and the whole encoder-decode (p.).. After the overall framework has been trained end to end, the frozen encoder can be used to compute embeddings of unseen INRs with single forward pass (see Fig. 3 right) while the implicit decoder can be used, if needed, to reconstruct the discrete representation given embedding.
supervised regress the value
In Fig. 4a we compare 3D shapes reconstructed from INRs unseen during training with those reconstructed by the inr2vec decoder starting from the latent codes yielded by the encoder. We visualize point clouds with 8192 points, meshes reconstructed by marching cubes (Lorensen & Cline 1987) from grid with resolution 128a and voxels with resolution 643. We note that. though embedding is dramatically more compact than the original INR, the reconstructed shape resembles the ground-truth with good level of detail. Moreover, in Fig. 4h we linearly interpolate between the
5

---

Published as a conference paper at ICLR 2023
ModelNer40
ShapeNettO m
ScanNer1D mAPes
Method
mAFe
mAPg5
IAP
mAPe]
mAFe TO
mAPe
mAPe 10
NoiniNer 0 ct 201
801]
91.7
94.4
90.6
96.6
98.1
65.7
86.2
92.6
PointNet++ 0 el al 20176
85.1
93.9
96.0
92.2
97.5
98.6
71.6
59.3
93.7
DOUNN [Wang 20196
83.2
92.7
95.1
91.0
96.7
98.2
66.1
88.0
93.1
95.1
90.6
96.7
98.1
Table 1: Point cloud retrieval quantitative results.
Point Cloud ShapeNet10
Mesh
Voxels
Method
ModelNet4D
ScanNet10
Manifold40
ShapeNet10
PointNet (Q) el al. 24
88.8
94.3
72.7
PrintNates Wr et
70177b
89.7
94.6
76.4
DGCNN Wing al
20196
89.9
94.3
76.2
MeshWalker Lahav
lai 20240
90.0
Cone3DNer (Maturana & Schorer 2015
92.1
inE2vec
87.0
93.3
72.1
86.3
Table 2: Results on shape classification across representations.
embeddings produced by n 2vec from two input shapes and show the shapes reconstructed from the interpolated embeddings. Results highlight that the latent space learned by inr2vec enables smooth interpolations between shapes represented as INRs. Additional details on inr2vec training and the procedure reconstruct the discrete representations from the decoder are in the Appendices.
4 DEEP LEARNING ON INRs
In this section, we first present the set-up of our experiments. Then, we show how several tasks dealing with 3D shapes can be tackled by working only with inr2vec embeddings as input and/or output. Additional details on the architectures and on the experimental settings are in Appendix [
General settings. In all the experiments reported in this section, we convert 3D discrete representa- tions into INRs featuring 4 hidden layers with 512 nodes each, using the SIREN activation function (Sitzmann et al. 20206). We train inr2vec using an encoder composed of four linear layers with respectively 512, 512, 1024 and 1024 features, embeddings with 1024 values and implicit decoder with hidden layers with 512 features. In all the experiments, the baselines are trained using standard data augmentation (random scaling and point-wise jittering), while we train both v and the downstream task-specific networks on datasets augmented offline with the same transformations.
Point cloud retrieval. We first evaluate the feasibility of using inr2vec embeddings of INRs to solve tasks usually tackled by representation learning, and we select 3D retrieval as a benchmark. We follow the procedure introduced in (Chang et al. 2015). using the euclidean distance to measure the similarity between embeddings of unseen point clouds from the test sets of ModelNet40 (Wu) al..112.. 2015) and ShapeNet10 (a subset of 10 classes of the popular ShapeNet dataset (Chang et al. 2015)). ). For each embedded shape, we select its k-nearest-neighhouss and compute a Precision Score comparing the classes of the query and the retrieved shapes, reporting the mean Average Precision consider three baselines to embed point clouds, which are obtained by training the PointNet (Qi et al., 20177), PointNet++ (Qi et al. 2017b) and DGCNN (Wang et al. 2019b) encoders in combination with fully connected decoder similar to that proposed in (Fan et al. (017) to reconstruct the input cloud. Quantitative results, reported in Table I show that, while there is an average gap of 1.8 mAP with PointNet++, inr2vec is able to match, and in some cases even surpass, the performance of the other baselines. Moreover, it is possible to appreciate in Fig. 55 that the retrieved shapes not only belong the same class the query but present also the same coarse structure. This finding highlights how the pretext task used to learn inr2vec embeddings can summarise relevant shape information effectively.
for different l (mAP@@ k). Beside inr2vec.
Shape classification. We then address the problem of classifying point clouds, meshes and voxel grids. For point clouds we use three datasets: ShapeNet10, ModelNet40 and ScanNer10 (Dai et al.
2017). When dealing with meshes, we conduct our experiments on the Manifold40 dataset (Hu et al.
2022). Finally, for voxel grids, we use again ShapeNet10, quantizing clouds to grids with resolution
64 Despite the different nature of the discrete representations taken into account, inr2vec allows us to perform shape classification on INRs embeddings, augmented online with E-Stitchup (Wolfe & Lundgaard| 2019), by the very same downstream network architecture, i.e.. simple fully connected classifier consisting of three layers with 1024, 512 and 128 features. We consider as baselines well-known architectures that optimized work on the specific input representations of each

---

Published as a conference paper at ICLR 2023
QUERY
2-NN
3-NN
Inference time (sec)
10
ShapoNett0 ModeiNesd0
40g stale
Painther
10
Painthetee
DGCNN
109
inreved
10a
2043
10k
Number of points
Figure 5: Point cloud retrieval qualitative results. Given the inrevec embedding of a query shape, we show the shapes reconstructed from the closest embeddings (L2 distance).
Figure 6: Time required to classify INRs en- coding udf. For point cloud classifiers, the time to reconstruct the discrete point cloud from the INR is included in the chart.
cappone
shair
bafee
kmmp
Upplpp
10000
medett
Mbll
Method
Pointer Oreral 7D
78.36
29.4
67.0
2013
20.0
23.1
21.3
57.8
73.0
Net
84.9
82.2
88.8
76.0
50.4
80.6
91.8
84.9
94,9
72.2
94.7
61
82.3
DGCNN
Wang & 2019
83.6
30.86
80.7
84.3
82.8
74.8
39.0
81.2
90.1
36.4
34.0
95.4
19.3
91.8
77.8
62.5
71.6
51.1
$1.3
76.91
80.2
76.2
70.3
70.1
38.0
65.0
90.5
82.1
77.4
94.4
61.4
92.7
79.0
56.2
68.6
78.5
Table 3: Part segmentation quantitative results. We report the loU for each class, the mean IoU over all the classes (class mloU) and the mean IoU over all the instances (instance mloU).
dataset. For point clouds, we consider PointNet (Qi et al. 2017a), PointNet++ (Qi et al. 2017b)
and DGCNN (Wang et al., 2019b). For meshes, we consider a recent and competitive baseline that processes directly triangle meshes (Lahav & Tal 2020). As for voxel grids, we train 3D CNN classifier that we implemented following (Maturana & Scherer 2015) (Conv3DNet from now on). Since only the train and test splits are released for all the datasets, we created validation splits from the training sets in order to follow a proper train/val protocol for both the baselines and inr2ved As for the test shapes, INRs fitted on the original test sets, as these would be the only data available at test time in a scenario where INRs used to store and communicate 3D data. We report the results of the baselines tested on the original discrete representations available in the original datasets in Appendix [ they are in line with those provided here. The results Table 2 show that inr2vec embeddings deliver classification accuracy close to the specialized baselines across all the considered datasets, regardless of the original discrete representation of the shapes in each dataset. Remarkably, our framework allows us to apply the same simple classification architecture on all the considered input modalities, in stark contrast with all the baselines that are highly specialized for each modality, exploit inductive biases specific to each such modality and cannot be deployed on representations different from those they were designed for. Furthermore, while presenting a gap of some accuracy points w.r.f. the most recent architectures, like DGCNN and Mesh Walker, the simple fully connected classifier that applied on inr2vec embeddings obtains scores comparable to standard baselines like PointNet and Conv We also highlight that, should 3D shapes be stored INRs, classifying them with the considered specialized baselines would require recovering the original discrete representations by the lengthy procedures described in Appendix C Thus, in F 6 we report the inference time of standard point cloud classification networks while including also the time needed to reconstruct the discrete point cloud from the input INR of the underlying udf at different resolutions. Even at the coarsest resolution (2048 points), all the baselines yield an inference time which is one order of magnitude higher than that required to classify directly the inr2vec embeddings. Increasing the resolution of the reconstructed clouds makes the inference time of the baselines prohibitive, while nr2vec, not requiring the explicit clouds, delivers a constant inference time of 0.001 seconds.
evaluated all the baselines on the discrete representations reconstructed from the
Point cloud part segmentation. While the tasks of classification and retrieval concern the possibility of using inr2vec embeddings as a compact proxy for the global information of the input shapes, with the task of point cloud part segmentation we aim at investigating whether inr2vec embeddings can be used also to assess upon local properties of shapes. The part segmentation task consists in predicting semantic (i.e., part) label for each point of given cloud. We tackle this problem by training a decoder similar to that used to train our framework (see F Such decoder is fed with the i n 2r2vec embedding of the INR representing the input cloud, concatenated with the coordinate
7

---

Published as a conference paper at ICLR 2023
SEGMENTATION DECODER
INR
Method.
(h) Qualitative results.
Figure Point cloud part segmentation.
TRAINING
Anc2vac LATENT SPACE
SP-GAN
I inráveo
2nFC
GENGRATOR
Anr2vec OCCNET
INFERENCE
NOISE
2xFC
GENERATOR
(a) Method.
(b) Qualitative results.
Figure 8: Learning generate shapes from inr2vec latent space.
of 3D query, and it is trained to predict the label of the query point. We train well PointNer, PointNet++ and DGCNN, on the ShapeNet Part Segmentation dataset (Yi et al. 2016) with point clouds 2048 points, with the same train/val/test of the classification task. Quantitative results reported in Table show the possibility of performing also a local discriminative task as challenging as part segmentation based on the task-agnostic embeddings produced by inr2vec and, in so doing, to reach performance not far from that of specialized architectures. Additionally, in Fig. 77b we show point clouds reconstructed at 100K points from the input INRs and segmented with high precision thanks to our formulation based on semantic decoder conditioned by the inr2vec embedding.
Shape generation. With the experiments reported above we validated that, thanks inr2vec embeddings, INRs can be used as input in standard deep learning machinery. In this section, we address instead the task of shape generation in an adversarial setting to investigate whether the compact representations produced by framework can be adopted also medium for the output of deep learning pipelines. For this purpose, as depicted Fig. Sa we train a Latent-Gan GAN (Achlioptas) et al.) 2018) to generate embeddings indistinguishable from those produced by inr2vec starting from random noise. The generated embeddings can then be decoded into discrete representations with the implicit decoder exploited during i inr2vec training. Since w.r.l. the original discrete representation of shapes used to fit INRs, we can train Latent-GANs with embeddings representing point clouds or meshes based on the same identical protocol and architecture (two simple fully connected networks as generator and discriminator). For point clouds, we train one Latent-GAN on each class of ShapeNet10, while we use models of cars provided by (Mescheder et al., 2019) when dealing with meshes. In Fig. 8b) we show some samples generated with the described procedure, comparing them with SP-GAN (i et 2021 on the chair class for what concerns point clouds and Occupancy Networks (Mescheder al. 2019 (VAE formulation) for meshes. Generated examples of other classes for point clouds are shown in Appendix ! The shapes generated with our Latent-GAN trained only inr2ves embeddings seem comparable those produced by the considered baselines, in terms of both diversity and richness of details. Additionally, by generating embeddings that represent implicit functions, method enables sampling point
framework is agnostic
8

---

Published as a conference paper at ICLR 2023
TRAINING
INCOMPPETE (impur)
inz2wee LATER
Cure
COMPLLETE (ousput)
BxFC
TRANSFER
INFERENCE
(b) Point cloud completion.
MESH PCD (ousput) (impput)
(a) Method.
(c) Surface reconstruction.
Figure 9: Learning a mapping between inr2vec latent spaces.
clouds at any arbitrary resolution (e.g.. 8192 points in Fig. 8b) whilst SP-GAN would require a new training for each desired resolution since the number of generated points must be set at training time.
Learning a mapping between inr2vec embedding spaces. We have shown that inr 2vec embeddings be used proxy to feed INRs in input deep learning pipelines, and that they can also be obtained as output of generative frameworks. In this section we move a step further, considering the possibility of learning a mapping between two distinct latent spaces produced by our framework for two separate datasets of INRs, based on a transfer function designed to operate on inr2vec embeddings as both input and output data. Such transfer function can be realized by a simple fully connected network that maps the input embedding into the output one and is trained by standard MSE loss (see Fig. 9a). As inr2ved generates compact embeddings the same dimension regardless of the input INR modality, the transfer function described here can be applied seamlessly to great variety tasks, usually tackled with ad-hoc frameworks tailored to specific input/output modalities. Here, apply this idea to two tasks. Firstly, address point cloud completion on the dataset presented in (Pan et al.] (021) by learning a mapping from inr2vec embeddings of INRs that represent incomplete clouds to embeddings associated with complete clouds. Then, tackle the task of surface reconstruction on ShapeNet cars, training the transfer function to map inr vec embeddings representing point clouds into embeddings that can be decoded into meshes. As can appreciate from the samples in Fig. Pbland and Fig. 9c the transfer function can learn an effective mapping between inr2vec latent spaces. Indeed, by processing exclusively INRs embedding. we can obtain output shapes that are highly compatible with the input ones whilst preserving the distinctive details, like the pointy wing of the airplane Fig. ob the flap of the first car in Fig. 9c
5 CONCLUDING REMARKS
We have shown that it is possible to apply deep learning directly on individual INRs representing 3D shapes. Our formulation of this novel research problem leverages on a task-agnostic encoder which embeds INRs into compact and meaningful latent codes without accessing the underlying implicit function. Our framework ingests INRs obtained from different 3D discrete representations and performs various tasks through standard machinery. However, we point out two main limitations: i) Although INRs capture continuous geometric cues, inr2vec embeddings achieve results inferior to sate-of-the-art solutions ii) There is no obvious way to perform online data augmentation on shapes represented as INRs by directly altering their weights. In the future, we plan to investigate these shortcomings as well as applying inr2vec to other input modalities like images, audio or radiance fields. We will also investigate weight- space symmetries (Entezari et al., 2021 as a different path to favour alignment of weights across INRs despite the randomness of training. We reckon that our work may foster the adoption of INRs as a unified and standardized 3D representation, thereby overcoming the current fragmentation of discrete 3D data and associated processing architectures.

---

Published as a conference paper at ICLR 2023
REFERENCES
Panos Achlioptas, Olga Diamanti, loannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds. In International conference on machine learning, pp. 40-49. PMLR, 2018.
Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. arXiv preprint arXiv:2209.04833 04836, 2022.
Matan Atzmon and Yaron Lipman. Sal: Sign agnostic learning of shapes from raw data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 2565 2574,, 2020.
Matan Atzmon, Haggai Maron, and Yaron Lipman. Point convolutional neural networks by extension operators. ACM Transactions on Graphics, 37(4), 2018.
Pierre Baque, Edoardo Remelli, Francois Fleuret, and Pascal Fua. Geodesic convolutional shape optimization. In International Conference on Machine Learning, pp. 472-481. PMLR, 2018.
Davide Boscaini, Jonathan Masci, Emanuele Rodolà, and Michael Bronstein. Learning shape correspondence with anisotropic convolutional neural networks. Advances in neural information processing systems, 29, 2016.
Rohan Chabra, Jan E Lenssen, Eddy 1lg. Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe. Deep local shapes: Learning local sdf priors for detailed 3d reconstruction. In European Conference on Computer Vision, pp. 608-625 Springer, 2020.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv: 1512.03012. 2015.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5939-5948, 2019.
Julian Chibane, Gerard Pons-Moll, al. Neural unsigned distance fields for implicit function learning. Advances in Neural Information Processing Systems, 33:21638-21633 2020.
Tejalal Choudhary, Vipul Mishra, Anurag Goswami, and Jagannathan Sarangapani. comprehensive survey model compression and acceleration. Artificial Intelligence Review, 13375551 2020.
Christopher B Choy, Danfei Xu, Jun Yonnoung Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: unified approach for single and multi-view 3d object reconstruction. In European conference computer vision, pp. 628-644. Springer, 2016.
Angela Dai, Angel X Chang, Manolis Savva, Macicj Halber, Thomas Funkhouser, and Matthias NießBner. Scannet: Rinhty-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5828-5839, 2017. manifolds
Yilun Du, Katie Collins, Josh Tenenbaum, and Vincent Sitzmann. Learning of neural fields. Advances in Neural Information Processing Systems. 34:8320-88 1331, 2021.
Emilien Dupont, Adam Goliñski, Milad Alizadeh, Yee Whye Teh, and Arnaud Doucet. Coin: Compression with implicit neural representations. arXiv preprint arXiv:2103.03/22 2021a.
Emilien Dupont, Yee Whye Tch, and Arnaud Doucet. Generative models as distributions of functions. arXiv preprint arXiv:2102.0444 047776 2021b.
Emilien Dupont, Hyunjik Kim, SM Ali Eslami, Danilo Jimenez Rezende, and Dan Rosenbaum. From data to functa: Your data point a function and you can treat like one. In International Conference Machine Learning, pp. 5694-5725 PMLR, 2022.
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance linear mode connectivity neural networks. In International Conference on Learning Representations, 2021.
10

---

Published as a conference paper at ICLR 2023
Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 605-613, 2017.
Siqi Fan, Qiulei Dong, Fenghua Zhu, Yisheng Lv, Peijun Ye, and Fei- Yue Wang. Scf-net: Learn- ing spatial contextual features for large-scale point cloud segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14504-14513, 2021.
Yutong Feng, Yifan Feng, Haoxuan You, Xibin Zhao, and Yue Gao. Meshnet: Mesh neural network for 3d shape representation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 8279-8286, 8286, 2019.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: /803.0335, 2018.
Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T. Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Local deep implicit functions for 3d shape. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Rohit Girdhar, David F Fouhey, Mikel Rodriguez, and Abhinav Gupta. Learning a predictable and generative vector representation for objects. In European Conference on Computer Vision, pp. 484-499. Springer, 2016.
Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regular- ization for learning shapes. In International Conference on Machine Learning, pp. 3789-3799,
PMLR, 2020.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph Martin, and Shi-Min Hu. Pct: Point cloud transformer. Computational Visual Media, 7(2):187-199, 2021.
Niv Haim, Nimrod Segol, Heli Ben-Hamu, Haggai Maron, and Yaron Lipman. Surface networks via general covers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 632-641, 2019.
Rana Hanocka, Amir Hertz, Noa Fish, Raja Giryes, Shachar Fleishman, and Daniel Cohen-Or. Meshenn: a network with an edge. ACM Transactions on Graphics (TOG), 38(4):1-12, 2019.
Amir Hertz, Rana Hanocka, Raja Giryes, and Daniel Cohen-Or. Deep geometric texture synthesis. doi: 10.1455/38656633392477.. URL
ACM Trans. Graph 39(4), 2020. ISSN 0730-0301.
/tttps:/doi..rq// htt../ooo.g...1113333.663333333
Shi-Min Hu, Zheng-Ning Liu, Meng-Hao Guo, Jun Jun-Xiong Cai, Jiahui Huang, Tai-Jiang Mu, and Ralph R Martin. Subdivision-bass mesh convolution networks. ACM Transactions on Graphics (TOG), -1(3):1-16, 2022.
Binh-Son Hua, Minh-Khoi Tran, and Sii-Kit Yeung. Pointwise convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 984-99333 2018.
Jingwei Huang, Haotian Zhang, Li Yi, Thomas Funkhouser, Matthias Nießner, and Leonidas J Guibas. Texturenet: Consistent local parametrizations for learning from high-rcsolution signals on meshes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
1140-4449 2019.
Florian Jaeckle and M Pawan Kumar. Generating adversarial examples with graph neural networks. In Uncertainty in Artificial Intelligence, pp. 1556-1564. PMLR, 2021.
11

---

Published as a conference paper at ICLR 2023
Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, Thomas Funkhouser, et al. Local implicit grid representations for 3d scenes. In Proceedings of the IEEE/CVF Conference Computer Viston and Pattern Recognition, pp. 6001-6010, 2020.
Danilo Jimenez Rezende, SM Eslami, Shakir Mohamed, Peter Battaglia, Max Jaderberg, and Nicolas Heess. Unsupervised learning of 3d structure from images. Advances in neural information processing systems, 29, 2016.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprini arXiv: 1412.6980, 2014.
Boris Knyazev, Michal Drozdzal, Graham W. Taylor, and Adriana Romero. Parameter prediction for unseen deep architectures. In A. Beygelzimer, Y. Dauphin, Liang, and Wortman Vaughan (eds.), Advances Neural Information Processing Systems, 2021. URL httpppss//ppeeeeee///
Alon Lahay and Ayellet Tal. Meshwalker: Deep mesh understanding by random walks. ACM Transactions on Graphics (TOG), 1-13):1-13, 2020.
Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem. Deepgens: Can gens go as deep as cnns? In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9267-9276, 2019.
Lei Li. Siyu Zhu, Hongbo Fu, Ping Tan, and Chiew-Lan Tai. End-to-end learning local multi-view descriptors for 3d point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020a.
Ruihui Li, Xianzhi Li, Ka-Hei Hui, and Chi-Wing Fu. Sp-gan: Sphere-guided 3d shape generation and manipulation. ACM Transactions Graphics (TOG), 40(4):1-12, 2021.
Xianzhi Li, Ruihui Li, Lei Zhu, Chi-Wing Fu, and Pheng-Ann Heng. Dnf-net: deep normal filtering network for mesh denoising. IEEE Transactions Visualization and Computer Graphics, 27(10): 4060 4072, 2020b.
Chunfeng Lian, Li Wang, Tai-Hsien Wu, Mingxia Liu, Francisca Durán, Ching Srangg-Chngggan Ko, and Dinggang Shen. Meshsnet: Deep multi-scale mesh feature learning for end-to-end tooth labeling on 3d dental surfaces. In International Conference Medical Image Computing and Computer- Assisted Intervention, pp. 837-845. Springer, 2019.
Tsung-Y. Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Hsueh-Ti Derek Liu, Francis Williams, Alec Jacobson, Sanja Fidler, and Litany. Learning smooth neural functions via lipschitz regularization. arXiv preprint arXiv:2202.00 220.222222........ 2022.
Ze Liu, Han Hu, Yue Cao, Zheng Zhang, and Xin Tong. A closer look at local aggregation operators in point cloud analysis. In European Conference Computer Vision, pp. 326 342. Springer, 2020.
William Lorensen and Harvey E Cline. Marching cubes: A high resolution surface construction algorithm. ACM siggraph computer graphics. 21(4):163-169, 1987.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05111, 2017.
Jingyue Lu and M. Pawan Kumar. Neural network branching for neural network verification. In
International Conference Learning Representations, 2020. URL uttp::tts:///eeeeeerr
net/ htt.://ooommnndallfaatteeee
Julien N. P. Martel, David B. Lindell, Connor Z. Lin, Eric R. Chan, Marco Monteiro, and Gordon Wetzstein. Acom: Adaptive coordinate networks for neural scene representation. ACM Trans. Graph. (SIGGRAPH), 40(4), 2021.
12

---

Published as a conference paper at ICLR 2023
Jonathan Masci, Davide Boscaini, Michael Bronstein, and Pierre Vandergheynst. Geodesic con- volutional neural networks on riemannian manifolds. In Proceedings of the IEEE international conference on computer vision workshops, pp. 37-45, 2015.
Daniel Maturana and Sebastian Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In 2015 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 922-928. IEEE, 2015.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction function space. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 4460-4470, 2019.
Mateusz Michalkiewicz, Jhony K Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders Eriks- son. Implicit surface representations as layers in neural networks. Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 4743-4752, 2019.
Francesco Milano, Antonio Loquercio, Antoni Rosinol, Davide Scaramuzza, and Luca Carlone. Primal-dual mesh convolutional neural networks. Advances in Neural Information Processing Systems, 33:952-963, 2020.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pp. 405-421. Springer, 2020.
Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Jan Svoboda, and Michael M Bronstein. Geometric deep learning on graphs and manifolds using mixture model cnns. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5115-5124, 2017.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primi- tives with a multiresolution hash encoding. ACM Trans. Graph., 41(4)::02:11102::55 July 2022. doi: 10.1 00.1145/35282233335555 33530127.
3530127
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Occupancy flow: 4d reconstruction by learning particle dynamics. Proceedings of the IEEE/CVF international conference on computer vision, pp. 5379-5389, 2019.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 3504-3515 2020.
Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger. Texture fields: Learning texture representations in function space. Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 4531-4540. 2019.
Liang Pan, Xinyi Chen, Zhongang Cai, Junzhe Zhang. Haiyu Zhao, Shuai Yi, and Ziwei Liu. Variational relational point completion network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8524-88888 8533, 2021.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 165-174, 2019.
Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolu- tional occupancy networks. In European Conference on Computer Vision, pp. 523-540. Springer, 2020.
Charles R Qi. Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, and Leonidas J Guibas. Volumetric and multi-view cnns for object classification on 3d data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5648-5656, 2016.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-660, 20.
13

---

Published as a conference paper at ICLR 2023
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. Advances in neural information processing systems, 30, 2017b.
Nikhila Ravi, Jeremy Reizenstein, David Novotny. Taylor Gordon, Wan- Yen Lo. Justin Johnson, and Georgia Gkioxari. Accelerating 3d deep learning with pytorch3d. arXiv:2007.08500 2020.
Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF International Conference Computer Vision pp. 2304 2314, 2019.
Jonas Schult, Francis Engelmann, Theodora Kontogianni, and Bastian Leibe. Dualconvmesh-nee Joint geodesic and euclidean convolutions on 3d meshes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 8612 8622, 2020.
Konstantin Schürholt, Dimche Kostadinov, and Damian Borth. Self-super representation learning neural network weights for model characteristic prediction. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL
net www.ffoormiiiioooo 1DFFFF1X0111111
Shaoshuai Shi, Chaoxu Guo, Li Jiang. Zhe Wang. Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-renn: Poxnt feature set abstraction for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10529-10538, 2020.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Contin- uous 3d-structure-aware neural scene representations. Advances in Neural Information Processing Systems, 32, 2019.
Vincent Sitzmann, Eric Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein. Metasdf: Meta-learning signed distance functions. Advances in Neural Information Processing Systems, 33: 10136-10147, 2020a.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-7473, 2020b.
Dmitriy Smirnov and Justin Solomon. Hodgenet: learning spectral geometry on triangle meshes.
ACM Transactions on Graphics (TOG), 40(4): 2021.
Lestie N Smith and Nicholay Topin. Superr-convergence:
Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for multi-domain operations applications, volume 11066 pp. 1100612. International Society for Optics and Photonics, 2019.
Shuran Song and Jianxiong Xiao. Deep sliding shapes for amodal object detection in rgb-d images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 808- 108-816, 816, 2016.
Yannick Strümpler, Janis Postels, Ren Yang, Luc Van Gool, and Federico Tombari. Implicit neural representations for image compression. arXiv preprint arXiv:2112.04267, 2021.
David Stutz and Andreas Geiger. Learning 3d shape completion from laser sean data with weak supervision. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1955-1964, 2018
Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11358-11333 11367, 2021.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Kea Keil, Nithin Raghavan, Utkarsh Singhal. Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537-7547, 2020.
14

---

Published as a conference paper at ICLR 2023
Maxim Tatarchenko, Stephan R Richter, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox. What do single-view view 3d reconstruction networks lear?? In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9405-3414, 2019.
Hugues Thomas, Charles R Qi. Bean-mmmnuuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 641 1411-6420, 2019.
David JJ Toal and Andy J Keane. Efficient multipoint aerodynamic design optimization via cokriging.
Journal of Aircraft, 48(5):1685-1666 2011.
Nobuyuki Umetani and Bernd Bickel. Learning three-dimensional flow for interactive aerodynamic design. ACM Transactions Graphics (TOG), 37(4):1-10, 2018. 2018.
Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquut, and llya O. Tolstikhin. Pre- dicting neural network accuracy from weights. arXiv, abs/2002. 1448, 2020.
Lei Wang, Yuchun Huang, Yaolin Hou, Shenman Zhang, and Jie Shan. Graph attention convolution for point cloud semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 00296-10305 2019a.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. Acm Transactions On Graphics (tog), 38-5):1-12, 2019b.
Cameron R Wolfe and Keld T Lundgaard. E-stitchup: Data augmentation for pre-trained embeddings. arXiv preprint arXiv: 1912::0971000777 2019.
Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning a proba- bilistic latent space of object shapes via 3d senerative-adversaari modeling. Advances in neural information processing systems, 29, 2016.
Wenxuan Wu, Zhongang Qi. and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9621-9629611 2019.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1912-1920, 2015.
Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual computing and beyond. arXiv preprint arXiv:2111.11426, 2021.
Mutian Xu, Runyu Ding, Hengshuang Zhao, and Xiaojuan Qi. Paconv: Position adaptive convolution with dynamic kernel assembling on point clouds. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3173-3188 2021.
Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, and Yu Qiao. Spiderenn: Deep learning on point sets with parameterized convolutional filters. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 87-102, 2018.
Yuqi Yang. Shilin Liu, Hao Pan, Yang Liu, and Xin Tong. Pfenn: Convolutional neural networks on 3d surfaces using parallel frames. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13578-13587, 2020.
Li Yi, Vladimir G. Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. scalable active framework for region annotation in 3d shape collections. SIGGRAPH Asia, 2016.
Haoxuan You, Yifan Feng. Rongrong Ji, and Yue Gao. Pvnet: A joint convolutional network of point cloud and multi view for 3d shape recognition. In Proceedings of the 26th ACM international conference on Multimedia, pp. 1310-1318, 2018.
15

---

Published as a conference paper at ICLR 2023
Gizem Yüce, Guillermo Ortiz-Jiménez, Beril Besbinar, and Pascal Frossard. A structured dictionary perspective on implicit neural representations. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp.
2022.
Yunfan Zhang, Ties van Rozendaal, Johann Brehmer, Markus Nagel, and Taco Cohen. Implicit neural video compression. arXiv preprint arXiv:21 12.1112.111222 2021.
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16268. 2021.
Qian-Yr Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3D: A modern library for 3D data processing. arXiv: 2018.
16

---

Published as a conference paper at ICLR 2023
A INDIVIDUAL INRS VS. SHARED NETWORK FRAMEWORKS
In this section we aim at comparing the representation power of individual INRs (i.e., one network for each data point) with the one of frameworks adopting a single shared network for the whole dataset, like DeepSDF (Park et al. 2019), Oceupancy et al. 2022). The important difference between such approaches and our method relies in the fact that in shared network frameworks, the shared network and the set of latent codes are the implicit representation, whose reconstruction quality is negatively affected by using single network to represent the whole dataset, as shown below. In our framework, instead, we decouple the representations (INRs) from the embeddings used to process them in downstream tasks (yielded by inr2vec). The quality of the representation is then entrusted to the individual INRs and inr2vec does not influence it.
(Mescheder et al., 2019) or (Dupont
To compare the representation quality of individual INRs with the one of share network frameworks, we fitted the SDF of the meshes in the Manifold40 dataset with OccupancyNetworks (Mescheder (i.e.. the architecture used
it al., 2019). DeepSDF (Park et al. (2099) and LatentModulatedden by the confemporary work that addresses deep learning on INRs (Dupont al., 2022)). Then, we reconstructed the training discrete meshes from the three frameworks and we compared them with the ground-trune truth ones, performing the same comparison using the discrete shapes reconstructed from individual INRs. To perform the comparison, we first reconstructed meshes and then we sampled dense point clouds (16,384 points) from the reconstructed surfaces, doing the same for the ground- truth meshes. We report the quantitative comparisons in Table 4 using two metries: the Chamfer Distance as defined in (Fan et al. 2017) and the F-Score as defined in Tatarchenko et al. 2019).
train set
Method
Method
CD (mm)
F-Score
CD (mm) +
F-Score 1
OccupancyNetworks
0.8
0.44
Networss
0.39
Dee
11.1
0.14
DeepSDF
6.6
0.25
LatentModulatedSiienn
0.7
0.37
LatentModulattddSiee
0.28
Individual INRs
0.3
0.50
Individual INRs
0.3
0.49
(a) Train set.
(b) Test set.
Table 4: Individual INRs vs. shared network frameworks. Comparison between discrete meshes reconstructed from individual INRs and from shared network frameworks on Manifold40.
The comparison reported in the Table 4able|4asshoss that both OccupancyNetworks and Loduaatod Siren cannot represent the shapes of the training set with good fidelity, most likely because of the single shared network that struggles to fit a big number of shapes with high variability (~~10K shapes, 40 classes). At the same time, DeepSDF obtains really poor scores, highlighting the difficulty of training an auto-decoder framework on a large and varied dataset. Individual INRs, instead, produce reconstructions with good quality, even if we adopted fitting procedure with only 500 steps for each shape.
Moreover, the approaches based on conditioned shared network tend fail in representing unseen samples that are out of the training distribution. Hence, in the foreseen scenario where INRs become standard representation for 3D data hosted in public repositories, leveraging on single shared network may imply the need to frequently retrain the model upon uploading new samples which, in turn, would change the embeddings of all the previously stored data. On the contrary, uploading the repository with a new shape would not cause any sort of issue with individual INRs, where one learns a network for each data point.
To better support our statements. in Table 4b we report the comparison between shape reconstructed from OccupancyNetworkk DeepSDF LatenttoduulttedSieen and individual INRs, using shapes from the test set of Manifold40, i.e., shapes unseen at training time. The numbers show that both OetwooncyNetworks and LatentModulltedSiienn present a drop the quality of the reconstructions, indicating that both frameworks struggle to represent new shapes not available at training time. Surprisingly, DeFpSDF produces better scores on the test set w.r.f. the results on the train set, but still presenting quite poor performance in comparison with the other methods. Conversely, the problem of representing unseen shapes is inherently inexistent when adopting individual INRs, as shown by the numbers in the last row of Table 4bl which almost identical the ones presented in Table 4a
17

---

Published as a conference paper at ICLR 2023
GT
Individual INRS
OccNet
LMS
DeepSDF
Figure 10: Individual INRs vs. shared network frameworks (train shapes). Qualitative com- parison meshes from Manifold40 reconstructed from individual INRs or from shared network frameworks, when dealing with training shapes. OccNet stands for OccupaneyNetworrss LMS stands for LatentModulatedSiren.
GT
Individual INRS
OccNet
LMS
DeepSDF
Figure 11: Individual INRs vs. shared network frameworks (test shapes). Qualitative comparison of meshes from Manifold40 reconstructed from individual INRs or from shared network frameworks, when dealing with shapes unseed during training. OccNet stands for OccupancyNetworks, LMS stands for aatentMouulatedSSiee.
18

---

Published as a conference paper at ICLR 2023
We report in Fig. 10 and Fig. [1 the comparison described above from qualitative perspective. It is possible to observe that the visualizations confirm the results posted in Table 4 with shared network frameworks struggling to represent properly the ground-truth shapes, while individual INRs enable high fidelity reconstructions.
We believe that these results highlight that frameworks based on single shared network cannot be used as medium to represent shapes as INRs, because their limited representation power when dealing with large and varied datasets and because of their difficulty in representing new shapes not available at training time.
B OBTAINING INRS FROM 3D DISCRETE REPRESENTATIONS
In this section, we detail the procedure used when fitting INRs to create the datasets used in this work. Given dataset of 3D shapes we train a set of the same number of MLPs, fitting each one single 3D shape. Every MI.P is thus trained to approximate a continuous function that descrihes the represented shape, the nature of the function being chosen according the discrete representation in
which the shape is provided. We adopt MI.ss with multiple hidden layers of the same dimension as done in (Sitzmann et al., 2020b a, Dupont et al., 202 Strümpler al. [202]] Zhang al. (021]) I), interleaved by the sine activation function, as proposed in (Sitzmann et al. (2020b), to enhance the capability of the MLPs to the high frequency details of the input signal.
In its general formulation, an INR can be used to fit continuous function f RR + Rout, To do so, a training set composed of N points x; E Rd with
2,,,,.., .N, paired with values y; fxx;) E T is exploited to find the optimal parameters 0" for the MLP that implements the INR, by solving the optimization problem:
1 N
0- = arg min
Σ
(1)
where fe represents the function f approximated by the MLP with parameters and &' a loss function that computes the error between predicted and ground-truth values.
The output value fx:) is computed as a series of linear transformations, each one followed by a non- linear activation function (i.e., the sine function in our case), except the last one. Considering MLP m. the mapping between its layers T- 1 and I. consists in a linear transformation that maps the values h6-1 R RD-- from the layer L, 1 into the values h o O(WL h b/) E TR RR of the layer m being the matrix of weights E Dr D by being the biases vector E TR Pz, and o(-) the non-linearity (Sitzmann et 2020b If we now consider M MLPs used fit M different INRs and the mapping between the layers 1-1 and L, we can easily compute such mapping simultaneously for all the MLPs on modern GPUs thanks to tensor programming frameworks. The mapping consists indeed in straightforward tensor contraction operation, where the values h E R Mx D2 of the layer I, are mapped to the values h, W, h, + b, E RM of the layer L, with WL E Af x Dr- and b" E R Dr. Extending this formulation to all the layers of the chosen MLP architecture allows to fit multiple INRs in parallel.
L, with WL
In the following, we describe how train MLPs to obtain INRs starting from point clouds, triangle meshes and voxel grids.
Point clouds. The INR for a 3D shape represented by point cloud P. encodes the unsigned distance function (udf) of the point cloud P. Given a point p R2, the value udf(p) is defined minger ||p allg- i.e., the euclidean distance from p to the closest point q of the point cloud. After preparing training set of N points x, E R with = 1, 2, ..... N, coupled with their udf values ye E IR, the INR of the underlying 3D shape is obtained by training a MLP to regress correctly the udf values, with the learning objective:
N (y fo(x:))?
Lmse
that consists in the mean squared enror between ground-truth values y and the predictions by the MLP (x,). An alternative objective is converting the udf values y; into values y/ continuously spanned in the range [0. 1), with 0 and representing respectively the predefined maximum distance
19

---

Published as a conference paper at ICLR 2023
from the surface and the surface level set (i.e., distance equal zero). Then, the MLP optimizes the binary cross entropy between such labels and the predicted values, defined as:
N
Lice 1
bce to + (1 - /ee Mog -;),, y/
(3)
where is = (x)(x,)), with representing the sigmoid function. In our experiments, we found empirically that this second learning objective leads to faster convergence and more accurate INRs, and we decided to adopt this formulation when producing INRs from point clouds.
Triangle meshes. Triangle meshes are usually adopted to represent closed surfaces. This provides an additional information compared to the point clouds case, since the 3D space can be divided into the portion contained inside and outside the closed surface. Thus, the INR of a closed 3D surface represented by triangle mesh can be obtained by fitting the signed distance function (sdf) to the surface defined by the mesh. Given a point p E RR, the value sdf (p) is defined as the euclidean distance from p the closest point of the surface, with positive sign if p is outside the shape and negative sign otherwise. Similarly to the point clouds case, an INR for a 3D shape represented by a triangle mesh be obtained by pursuing the learning objective presented in Eq. 2 using training set composed of 3D points paired with their sdf values. However, it possible to adopt a learning objective based on the binary cross entropy loss reported in Eq. (3) also for triangle meshes, and we empirically observed the same benefits. Hence, we adopt it also when fitting INRs on meshes. In this case, the sdf values yi are converted into values y. E [0, 1], with 0 and 1 representing respectively the predefined maximum distance inside and outside the shape, i.e., 0.5 represents the surface level set.
Voxel grids. A voxel grid is a 3D grid of V cubes marked with label 1, if the cube is occupied, and label otherwise. In order to fit an INR on voxels, it possible learn to regress the occupancy function (occ) of the grid itself. The training set, in this case, contains V 3D points that corresponds to the centroids of the cubes that compose the voxel grid. Being each of such points x; associated to 0-1 label yoe it is straightforward to binary classification objective to train the MLP that implement the desired INR. More specifically, we adopt the learning objective defined as:
1 N N
ffocal
Σ
a(1 0.)" (.)) + +1 (1-a)17 a) y1)log(1 Ot),
(4)
where i o(fo(x))), while a and 7 are respectively the balancing parameter and the focusing parameter of the focal loss proposed in (Lin et al. 2017). We deploy a focal loss to alleviate the imbalance between the number of occupied and empty voxels.
C RECONSTRUCTING DISCRETE REPRESENTATIONS FROM INRS
In this section we discuss how it is possible to sample 3D discrete representations from INRs, which could be necessary to process the underlying shapes with algorithms that need an explicit surface
(e.g., Computational Fluid Dynamics (Baque et al. 2018 Toal & Keane 2011 Umetani & Bickel 2018) or simply for visualization purposes.
Point clouds from udf. To sample dense point cloud from INR fitted on its udf. we use slightly modified version of the algorithm proposed in (Chibane et al. 2020). The basic idea is to query the udf with points scattered all over the considered portion of the 3D space, projecting such points onto the isosurface according to the predicted udf values. In order to do that, let us define So the udf approximated by the INR with parameters 0. Given point p E IR3, it can be projected onto the isosurface by computing its updated position Ps as:
vpJo(p) [Vpfe(p)]]
Ps = p
(5)
This can be intuitively understood by considering that the negative gradient of the udf indicates the direction of maximum decrease of the distance from the surface, pointing towards the closest point on it. Eq. (5). thus, can be interpreted as moving p along the direction of maximum decrease of the udf of quantity defined by the value of the wdf itself in p, reaching the point ps the
20

---

Published as a conference paper at ICLR 2023
surface. One must consider, though, that fo is only an approximation of the real udf, which leads to two considerations. On a first note, the gradient of fo must be normalized (as done in Eq. (5)) while the gradient of the real udf has norm equal to 1 everywhere except the surface. Secondly, the predicted udf value can be imprecise, implying that p can still be distant from the surface after moving it according Eq. (5) To address the second issue, the 3D position of ps is refined repeating the update described in Eq. (5) several times. Indeed, after each update, the point gets closer and closer to the surface, where the values approximated by fe are more accurate, implying that the last updates should successfully place the point on the isosurface. Given an INR fitted on the udf of a point cloud, the overall algorithm to sample a dense point cloud from it composed of the following steps. Firstly, we prepare set of points scattered uniformly in the considered portion of the 3D space and we predict their udf value with the given INR. Then we filter out points whose predicted udf is greater than fixed threshold (0.05 in our experiments). For the remaining points, we update their coordinates iteratively with Eq. (5) (we found 5 updates to be enough). Finally, we repeat the whole procedure until the reconstructed point cloud counts the desired number of points.
Triangle meshes from sdf. An INR fitted on the sdf computed from a triangle mesh allows to reconstruct the mesh by means of the Marching Cubes algorithm (Lorensen & Cline 1987). We refer the reader to the original paper for a detailed description of the method, but we report here a short presentation of the main steps, for the sake of completeness. Marching Cubes explores the considered 3D space by querying the sdf with 8 locations at a time, that are the vertices of an arbitrarily small imaginary cube. The whole procedure involves marching from one cube the other, until the whole desired portion of the 3D space has been covered. For each cube, the algorithm determines the triangles needed to model the portion of the isosurface that passes through it. Then, the triangles defined for all the cubes are fused together to obtain the reconstructed surface. In order to determine how many triangles are needed for single cube and how to place them, for each pair of neighbouring vertices of the cube, their sdf values are computed and one triangle vertex is placed between them if such values have opposite sign. Considering that the number of possible combinations of the sdf' signs at the cube vertices is limited, it is possible to build a look-up table to retrieve the triangles configuration for the eube starting from the sdf signs at its eight vertices, combined in a 8-bit integer and used as key for the look-up table. After the triangles configuration for cube has been retrieved, the vertices of the triangles are placed on the edges connecting the cube vertices, computing their exact position by linearly interpolating the two sif values that are connected by each edge.
Voxel grids from occ. In order to reconstruct voxel grids from INRs, we adopt a straightforward procedure. Each INR has been trained to predict the probability of certain voxel to be occupied, when queried with the 3D coordinates of the voxel centroid. Thus, a first step to reconstruct the fitted voxels consists in creating a grid of the desired resolution V. Then, the INR is queried with the 1 centroids of the grid and predicts an occupancy probability for each of them. Finally, we consider as occupied only voxels whose predicted probability is greater than a fixed threshold, which we set to 0.4, as we found empirically that it allows for good trade-off between scattered and over- filled reconstructions.
D inr2vec ENCODER AND DECODER ARCHITECTURES
In this section, we describe the architecture of inr2vec encoder, along with the one of the implicit decoder used to train it (see Section 3
Encoder. 2nee encoder, detailed in Fig. 12 consists in a series of linear transformations, that maps the input INR weights into features with higher dimensionality, before applying max pooling to obtain a compact embedding. More specifically, the input weights are rearranged in a matrix with shape L(H + 1) x H. where И is the number of nodes in the hidden layers of the MLP that implements the input INR and I, is the number of linear transformations between such hidden layers (i.e., the MLP has I. + 1 hidden layers). The matrix is obtained by stacking I. matrices (one for each linear transformation), each one with shape (H + 1) X H, being composed of a matrix of weights with shape H x H and row of // biases. In our setting, each MLP has 4 hidden layers with 512 nodes: the final matrix in input to i inz2vec encoder has shape 3- (512+ + 1) x 512 = 1539 x 512. In the current implementation, the four linear mappings of the encoder transform each row of the input matrix into features with size 512, 512, 1024 and 1024, obtaining, at each step. features matrices with shape 1539 x 512, 1539 x 512, 1539 x 1024 and 1539 1024. Finally, the encoder applies
21

---

Published as a conference paper at ICLR 2023
LIVIAIM L. LINBANTOLL
LIVIOINAL L. LINEANTILLL
1539 512
1539 X 512
1539 x 512
1539 x 1024
1539 x 1024
Column wise
Max Max poo lin
INR embedding (1024 values)
LP//F//
UN/SAARA
UN/SA//ALL
Figure 12: inrevec encoder. With series of linear transformations and final column-wise max pooling, the encoder maps the input weights matrix into a compact embedding. LIN/BN/ReLU stands for linear transformation, followed by batch normalization and ReLU activation function.
LINEAR
ReLU
512
INR em bmbdddgnn
LINEAR
ReLU
512
LINEAR
RelU
512
LINEAR
RelU
512
LINEAR
512
IINFAR
positional encoding
Figure 13: inr2vec decoder. Our framework is trained with an implicit decoder, that maps an INR embedding concatenated with 3D query into the value of the implicit function at the query coordinates.
column-wise max pooling to compress the final matrix into single compact embedding composed of 1024 values. Between the linear mappings of the encoder, we adopt batch normalization and ReLU activation functions.
The implicit decoder that we adopt train inr2ved presented in Fig. 13 We designed it taking inspiration from (Park et al. 2019), since need decoder capable of reproducing the implicit function imput INR when conditioned on the embedding obtained by the encoder. Thus, the decoder takes in input the concatenation of the INR embedding with the coordinates of given 3D query. We adopt the positional encoding proposed in (Mildenhall et al. 2020) to embed the input 3D coordinates into higher dimensional space enhance the capability of the decoder capture the high frequency variations of the input data. The query 3D coordinates are mapped into 63 values that, concatenated with the 1024 values that compose the INR embedding, result in a vector with 1087 values input for inr2vec decoder. Internally, the implicit decoder is composed of 4 hidden layers with 512 nodes and of skip connection that projects the input 1087 values into vector of 512 elements, that are summed to the features of the second hidden layer before being fed to the transformation that bridges the second and the third hidden layers. Finally, the features of the last hidden layer mapped to single output, which is compared to the ground-truth associated with the input 3D query to compute the loss. Each linear transformation of the decoder, except the output one, is paired with the ReLU activation function.
Decoder.
MOTIVATION BEHIND inr2vec ENCODER DESIGN
We designed inrevec encoder with the goal of obtaining a good scalability in terms memory occupation. Indeed, naive solution to process the weights of an input INR would consist in an MLP encoder mapping the flattened vector of weights to the embedding of the desired dimension. However, such approach would require huge amount of memory resources, since input INR of layers 512 neurons would have approximately 800K parameters. Thus, an MLP encoder going from 800K parameters parameters. We report in Table detailed analysis of the parameters of our encoder w.r.t. the ones of encoder by varying the input INR dimension. As can notice the MI. encoder does
embedding space of size 1024 would already have totality of ~800M
22

---

Published as a conference paper at ICLR 2023
INR hidden dim.
INR #layers
INR #params
#params 2nr2vec encoder
#params MLP encoder
512
4
~800
~3M
~800
512
8
~2M
~3M
~2B
512
12
~3M
~aM
<3B
512
~4M
~33
<.B
1024
4
~3M
~3.5M
<3B
1024
8
~
~3355M
77.5
1024
12
~IM
<3.5M
121
1024
16
~15
~3.55
16
Table 5: Number of parameters of inr2vec encoder. Comparison between the number of parameters of inr2vec encoder and the number of parameters of a generie MLP encoder.
not scale well, making this kind of approach very expensive in practice, while inr2vec encoder scales gracefully to bigger input INRs.
F EXPERIMENTAL SETTINGS
We report here a detailed description of the settings adopted in our experiments.
INRs fitting, In every experiment reported in the paper, fit INRs on 3D discrete representations using MIPs having hidden layers with 512 nodes each. We implement MLPs using sine as a periodic activation function, as proposed in (Sitzmann et al. 2020b). The procedure adopted fit single MLP consists in querying it with 3D points sampled properly in the space surrounding the underlying shape. The MLP predicts a value for each query and it's trained by computing loss function between the predicted value and the ground-tran -rutt value of the fitted implicit function (i.e., udf for point clouds, sdf for meshes and occ for voxels). The set of training queries is prepared according to different strategies, depending on the nature of the discrete representation being fitted. For voxel grids, the set of possible queries consists of the 3D coordinates of all the centroids of the grid. For point clouds and meshes, instead, queries are sampled with different densities in the volume containing the fitted shape: indeed, for each shape, we prepare 500K queries by taking 250K points close to the surface, 200K points at medium-far distance for the surface, 25K far from the surface and other 25K scattered uniformly in the volume. The queries coordinates are computed by adding gaussian noise to the points of the fitted point cloud to points sampled uniformly from the fitted mesh surface. More precisely, close queries are computed with noise sampled from the normal distribution (O, 0.001). medium-far queries with noise from (0, 0.01). ). far queries with noise from (0, 0.1). The uniformly scattered queries are just computed by sampling each of their coordinates from the uniform distribution (1(-1, 1), being the considered shapes normalized in such volume. As for the ground-truth values, for voxels they consist simply in the occupied/empty label of the voxel associated to the query. For point clouds, for each query we compute its udf value by building KDTree on the fitted point cloud and looking for the closest point to the considered query (we used the Py Torch33D (Ravi et al., 2020) implementation of the KDTree algorithm). For meshes, finally, we compute the sdf of queries with the functions provided in the Open3D library (Zhou et al.. | For each of the considered modalities, at each step of the fitting procedure, we randomly sample 10K pairs of queries/grounddtuee trrth values from the precomputed ones, performing total of 500 steps for each shape. Thanks to the procedure detailed in Appendix B we are able to fit up to 16 multiple MLPs in parallel, using Adam optimizer (Kingma & Ba 2014) with learning rate set to 1e-4. On a final note, we fixed the weights initialization of the ML.S to be always the same, as we observed empirically this to be key convergence of inr2vec. This choice poses no limitation to the practical use of our framework and has also been adopted in recent works (Gropp et al. 2020 Sitzmann al. 2020a).
inr2vec training. According to what is described in Section 3 during training our framework takes in input the weights of a given INR and is asked to reproduce the implicit function fitted by the INR on set of predefined 3D queries. Such queries are prepared with the same strategies described in the previous paragraph and, similarly to what is done while fitting INRs, at each step the training loss for 1nee is computed on 10K queries randomly sampled from a set of precomputed ones. In every experiment, we train ineevec with AdamW optimizer (Loshchilov & Hutter 2017). learning rate le-4 and weight decay le-2 for 300 epochs, one epoch corresponding to processing all the INRs
html
23

---

Published as a conference paper at ICLR 2023
LINEAR
ReLU
512
INR embedding
PART SEGME EN NETENTTTIION LOG
LINEAR
Re
512
LINEAR
Re
5:12
LINEAR
512
LINEAR
512
LINEAR
class one hot emoodi ng
0. 1, m
7 R
pe
positional encoding
Figure 14: Part segmentation decoder: We train a decoder to predict the part segmentation label of a given 3D query when conditioned on the embedding of the input INR and on the one-hot encoding of the INR class.
that compose the considered dataset, processing each training step a mini-bated of 16 INRs. During training, we select the best model by evaluating its reconstruction capability validation set of INRs. When training on INRs obtained from point clouds, we compare the ground-truth set points with the ones reconstructed by inr2vec decoder. For voxels, compare the input and the output grid by comparing the point clouds composed by the centroids corresponding to occupied voxels. As for what concerns meshes, compare the clouds containing input and output vertices. In all cases, the reconstruction quality evaluated by computing the Chamfer Distance between ground-truth and output point clouds, as defined in (Fan et al., 2017). See Appendix of this document for details on how to sample discrete 3D representations from the implicit functions fitted by INRs and that inrevec is trained reproduce.
Shape classification. The classifier that we deploy on inr2vec embeddings is composed of three linear transformations, mapping sequentially the input embedding with 1024 features vectors of size 512. 256 and, finally. to vector with a number of values corresponding to the number of classes of the considered dataset. The final vector is then transformed to probability distribution with the softmax function. We use 1D batch normalization and the ReLU activation function between the classifier linear transformations. In all experiments, the classifier is trained for 150 epochs, with AdamW optimizer (Loshchilov & Hutter (017) and weight decay le-2. The learning rate is scheduled according to the OneCycle strategy (Smith Topin 2019), with maximum learning rate set to Ie-4. At each training step, the classifier processes mini we select the best model by computing the classification accuracy on a validation set of embeddings. The best model is used after training to compute the classification accuracy the test set, obtaining the numbers reported in the tables of the paper.
mini-batch batch counting 256 embeddings. During training,
Point cloud part segmentation. In order to tackle point cloud part segmentation starting from inr2vec embeddings, we adopt a decoder similar to the one that we use for reconstruction during inr2vec training. The part segmentation decoder, depicted in Fig. [4] is fed with the positional encoding of a 3D query together with the embedding of an input INR and predicts a K -ddimensional vector of segmentation logits for the given query. with representing the total number of parts of all the C' available categories. Moreover, as done in previous work (Qi et al. 2017a b) Wang et al. 2019b), we concatenate an additional C-dimensional vector to the input of the part segmentation decoder, conditioning the output of our decoder with the one-hot encoding of the input INR class. We conduct our experiments on the ShapeNet Part Segmentation dataset (Yi et al. 2016), that presents 16 categories labeled with two to five parts, for total of 50 parts (i.e., C=16 and K=50), According to a standard protocol (Qi et al.) 201 b. Wang et al. 2019b), during training we compute the cross-entropy loss function all the K logits predicted by decoder, while, test time, the final prediction is obtained considering only the subset of parts belonging to the specific class of the input INR. The part segmentation decoder is trained with the original point clouds available in the ShapeNet
24

---

Published as a conference paper at ICLR 2023
Part Segmentation dataset, where part labels are provided for each point of each cloud. At test time, though, we test both our decoder and the considered competitors on the point clouds reconstructed from the input INRs, since we want simulate the scenario of 3D shapes being available exclusively in the form of INRs. Thus, the protocol to obtaín a segmented point cloud starting from an input INR consists in reconstructing the cloud first (see Appendix [ and then in assigning a part label each point of the reconstructed shape with our part segmentation decoder. When ground-truth labels are required to compute quantitative results, we obtain them by comparing the reconstructed cloud with the original one and assigning to each point of the reconstructed shape the part label of the closest point in the original cloud. Our part segmentation decoder is trained for 250 epochs with Adam optimizer, OneCycle learning rate scheduling with maximum value set to le-4, weight decay equal to le-2 and mini-batches composed of 256 embeddings, each one paired with 3D queries from the original point clouds during training and from the ones reconstructed from the input INRs at test time. During training, we compute the class mIoU on the validation split and save the best model in order to compute the final metrics on the test set.
Shape generation. We perform unconditional shape generation by training Latent-GAN (Achlioptas et al., 2018) to generate embeddings indistinguishable from the ones produced by inr2vee on a given dataset. This approach allows us to train a shape generation framework with the very same architecture generate embeddings representing INRs with different underlying implicit functions, such as udf for the point clouds of ShapeNet10 and sdf for the models of cars provided by (Mescheder et al. 2019). We conducted our experiments using the official implementation setting all the hyperparameters to default. The generator network is implemented as a fully connected network with two layers and ReLU non linearity, that map an input noise vector with 128 values sampled from the normal distribution (N 0.2) to an intermediate hidden vector of the same dimension and then to the predicted embedding with 1024 values (we removed the final ReLU present in the original implementation). The discriminator is also fully connected network, with three layers and ReLU non linearity. The first layer maps the embedding produced by the generator to a hidden vector with 256 values, which are then transformed by the second layer into hidden vector with 512 values, that are finally used by the third layer, together with the sigmoid function, to predict the final score. According to the original implementation, we trained one separate Latent-GAN for each class of the considered datasets, using the Wasserstein objective with gradient penalty proposed in (Gulrajani et al. 2017) and training each model for 2000 epochs.
Learning a mapping between inr2vec embedding spaces. The transfer function between inr2vec embedding spaces is implemented as a simple fully connected network, with 8 linear layers interleaved by ID batch norm and ReL.U activation functions. All the hidden features produced by the linear transformations present the same dimension of the input embedding, i.e., 1024 values. The final linear layer predicts the output embedding, which is compared with the target one with a standard L.2 loss. We train the transfer network with Adam optimizer, constant learning rate and weight decay both set to le-4, stopping the training upon convergence, which we measure by comparing the shapes reconstructed by the predicted embeddings with the ground-truth ones on a predetermined validation split. Such validation metrics are used also to save the best model during training, which is finally evaluated on the test set.
G IMPLEMENTATION, HARDWARE AND TIMINGS
We implemented our framework with the Py Torch library, performing all the experiments on single NVIDIA 3090 RTX GPU. We created an augmented version of each considered dataset, in order to obtain roughly ~00K INRs, whose fitting requires around days in the current implementa- tion. Training in2vec requires another 48 hours, while all the networks adopted to perform the downstream tasks on inr2vec embeddings can be trained in few hours.
H TESTING ON ORIGINAL DISCRETE 3D REPRESENTATION
In the experiments "Shape classification" and "Point cloud part segmentation" presented in the paper, we evaluated the competitors on the 3D discrete representations reconstructed from the INRs fitted on the test sets of the considered datasets, since these would be the only data available at test
httpp:: https////ggihhbb.ccc/paaaaaaaaaa
25

---

Published as a conference paper at ICLR 2023
Point Cloud ShapeNetTO
Mesh Manifold40
Voxels
Method
ModelNet40
ScanNet
ShapeNet 10
PointNet (Qi ot al.
201
88.8
94.7
72.8
PointNet+++ et a.
2017b
91.0
95.2
76.3
DGCNN [Wang ct al.
20196
91.6
94.0
75.1
MeshWalker (Labav lal 2020
90.6
3DNt Maturana Scherer
2015
92.5
1nr2vec
87.0
93.3
72.1
86.3
93.0
Table 6: Shape classification results. We report here shape classification results when testing on the original discrete representations of the test sets instead of reconstructing them from the input INRs.
Rostance mla!
Maateboodd
sirplanc
comphooe
cop
chair
p0ine
kanf
kmmp
haptop
pinel
mokeel
Method
PoinsNet
PointNes
Quet al 7017a
2017S
83.0
84.4
82.8
78.8
80.5
81.7
77.9
36.5
85.2
74.4
78.6
89.0
90.2
08.3
77.9
90,
91.2
$4.5
43.2
98.6
220
98.3
94.6
62.7
58
64.2
72.7
75.6
81.0
80.9
DGINNN
Wing 20196
84,3
81.4
$1.6
82.2
80.9
75.7
90.7
80.9
90.2
86.9
82.6
94.8
64.8
92.8
81.0
60.6
74.7
51.8
80.5
71.1
79.5
72.9
72.3
70.7
87,4
64.1
89.4
$1.6
76.5
94.5
59.3
92.4
78.4
53.5
67.5
77.3
Table 7: Part segmentation results. In this table present part segmentation results when testing on the original discrete representations the test sets instead reconstructing them from the input INRs. We report the IoU for each class, the mean IoU over all the classes (class mloU) and the mean IoU over all the instances (instance mloU).
time in scenario where INRs are used to store and communicate 3D data. For completeness, we report here the scores achieved by the baselines when tested on the original discrete representations. without reconstructing them from the input INRs. Such results are presented in Table 6 for shape classification and in Table | for part segmentation. We report in the tables also the results obtained with our framework: they are the same reported in Table 2 for what concerns shape classification, since our classifier processes exclusively inreved embeddings, while they are different for part segmentation,
query points for segmentation decoder those from the discrete point clouds reconstructed from input INRs in 3 while we use those from the original point clouds in the experiment reported here, done for the competitors. The results reported in the tables show limited differences, either positive or negative, with the ones presented in Section 4 mostly within the range of variations due the inherent stochasticity of training. There few larger differences, like DGCNN on ModelNet40 (+1.7 when tested on the original discrete representations) or on ScanNet (-1.1 when tested on the original discrete representations), whose difference in sign however suggests neither of the two settings is clearly superior to the other.
I ALTERNATIVE ARCHITECTURE FOR inr2vec
As reported in Section 3 inr2vec encoder takes in input the weights of INR reshaped in suitable way, discarding the parameters of the first and of the last layers. this section we consider the possibility of processing all the weights of the input INR, including the input/output ones. To this end, one must properly arrange the input/output parameters since they feature different dimensionality from the ones of the hidden layers and cannot be seamlessly stacked together with them. More specifically, the first layer of an INR consists in a matrix of weights Win E BR D and in a vector of biases bi., #x with H being the dimension of the hidden features of the INR and D being the dimension of the inputs (i.e., 3 in our case of 3D coordinates). The output layer, instead, is responsible of transforming the final vector of hidden features to the predicted output, which is always a single value in the cases considered in our experiments (i.e., udf, sdff and occ). Thus, the last layer presents matrix of weights W out TR. 1xx and single bias bout- In order to include the input/output parameters in the matrix P presented in input to inr2vec encoder (see Section 3).
Architecture F-Score t
CD (mm) +
hidden layers
57.41
3.1
all layers
56.76
3.1
Table 8: Quantitative comparison between alternative inr2vec architectures. We compare the reconstruction capability of inr2vec when processing only the weights of the hidden layers ("hidden layers") or all the weights ("all layers") of the input INRs.
26

---

Published as a conference paper at ICLR 2023
inr2vec whidden layers* OUTPUT
inr2vec <all layers OUTPUT
INPUT INR
Figure 15: Qualitative comparison between alternative inr2vec architectures. We compare reconstruction capability of inr2vec when processing only the weights of the hidden layers ("hidden layers") or all the weights ("all layers") of the input INRs.
Win needs to be transposed, obtaining matrix with shape 3 H, bin is transposed done also for the biases of all the other layers, W out doesn't need any manipulation and we decided to repeat the single-valued bout H times. In this section, compare the formulation presented in Section (reported as "hidden layers") with the one proposed here (reported as "all layers"), looking at the reconstruction capabilities of the two variants of our framework when trained on ModelNet40, In Table 8, we report both the F-score (Tatarchenko et al.) 2019) and the Chamfer Distance (CD) (Fan al. 2017) between the clouds used to obtain the INRs presented in input to inr2vec and the ones reconstructed from inr2vec embeddings, while in Fig. 15 we show the same comparison from qualitative perspective. Results show that processing all the INR weights doesn't produce any significant difference w.r.t. ingesting only the weights of the hidden layers. However, the latter variant provides a slight advantage in terms of F-score, simplicity and processing time, motivating our choice to adopt it as formulation for inr2vec.
J ADDITIONAL QUALITITATIVE RESULTS
We report here additional qualitative results. In Fig. 16 Fig. and Fig. 18 we show some com- parisons between the discrete shapes reconstructed from input INRs and the ones reconstructed from inr2vec embeddings. Fig. 19 Fig. 20 and Fig. 21 present smooth interpolations between inr2vec embeddings. In Fig. 22 and Fig. 23 we propose additional qualitative results for the point cloud retrieval experiments. Fig. 24 shows qualitative results for point cloud part segmentation for all the classes of the ShapeNet Part Segmentation dataset. Fig. 25 Fig. [2] and Fig. [2 report shapes generated with Latent-GANs (Achlioptas et al. 2018) trained on inrevec embeddings. Finally, Fig. 28] Fig. 29 present additional qualitative results for the experiments of point cloud completion and surface reconstruction, tackled by learning a mapping between i inr2ver latent spaces.
27

---

Published as a conference paper at ICLR 2023
INPUT INR
inr2vec OUTPUT
INPUT INR
inr2vec OUTPUT
INPUT INR
inr2vec OUTPUT
Figure 16: inr2vec reconstructions when dealing with INRs fitted point clouds. Comparison between discrete shapes reconstructed from the INRs presented in input to inr2vec ("INPUT INR") and the ones reconstructed from 2 embeddings ("inr2vec OUTPUT").
INPUT
inr2vec OUTPUT
INPUT
inr2vec OUTPUT
INPUT INR
inr2vec OUTPUT
INR
INR
Figure 17: inr2vec reconstructions when dealing with INRs fitted on meshes. Comparison between discrete shapes reconstructed from the INRs presented in input inr2vec ("INPUT INR") and the ones reconstructed from inr2vec embeddings ("in22vec OUTPUT").
28

---

Published as a conference paper at ICLR 2023
INPUT INR
inr2vec OUTPUT
INPUT INR
inr2vec OUTPUT
INPUT INR
inr2vec OUTPUT
Figure 18: inr2vec reconstructions when dealing with INRs fitted on voxel grids. Comparison between discrete shapes reconstructed from the INRs presented in input to inr2vec ("INPUT INR") and the ones reconstructed from inr2vec embeddings ("inr2vec OUTPUT")
interpolation factor
1
Figure 19: inr2vec latent space interpolation. Given two inr2vec embeddings obtained from INRs fitted on point clouds, possible to linearly interpolate between them, producing embeddings that represent unseen INRs of plausible shapes.
29

---

Published as a conference paper at ICLR 2023
0
interpolation factor
1
Figure 20: inr2vec latent space interpolation. Given two 1nr2vec embeddings obtained from INRs fitted on meshes, it is possible to linearly interpolate between them, producing new embeddings that represent unseen INRs of plausible shapes.
interpolation factor
Figure 21: inr2vec latent space interpolation. Given two inr2vec embeddings obtained from INRs fitted on voxel grids, it is possible to linearly interpolate between them, producing new embeddings that represent unseen INRs of plausible shapes.
30

---

Published as a conference paper at ICLR 2023
QUERY
1-NN
2-NN
3-NN
4-NN
Figure 22: Point cloud retrieval (ModelNet40). Qualitative results of the point cloud retrieval experiment conducted on inr2vec latent space. We show the discrete shape reconstructed from the query INR on the left and the discrete clouds reconstructed from the closest inr2vec embeddings in the columns 2-5.
QUERY
1-NN
2-NN
3-NN
4-NN
Figure 23: Point cloud retrieval NO)))) Qualitative results of point cloud retrieval experiment conducted on inr2vec latent space. We show the discrete shape reconstructed from the query INR on the left and the discrete clouds reconstructed from the closest inr2vec embeddings in the columns 2-5.
31

---

Published as a conference paper at ICLR 2023
Figure 24: Point cloud part segmentation. Qualitative results of the part segmentation experiment conducted with our segmentation decoder conditioned on inr2vec embeddings.
BATHTUB
BOOKSHELF
CHAIR
BED
CABINET
LAMP
Figure 25; Shape generation (point clouds). We show point clouds reconstructed from embeddings
generated by Latent-GAN trained on inr2vec embeddings (one model for each class).
32

---

Published as a conference paper at ICLR 2023
TABLE
PLANN
MONITOR
SOFA
Figure 26: Shape generation (point clouds). We show point clouds reconstructed from embeddings
generated by Latent-GAll GAN trained on inr2vec embeddings (one model for each class).
Figure 27: Shape generation (meshes). We show meshes reconstructed from embeddings generated by a Latent-GAN trained on inr2vec embeddings.
33

---

Published as a conference paper at ICLR 2023
Figure 28: Point cloud completion. Qualitative results of the point cloud completion experiment conducted with transfer network that learns a mapping between 2nr2vec latent spaces.
Figure 29: Surface reconstruction. Qualitative results of the surface reconstruction experiment conducted with a transfer network that learns a mapping between inr2vec latent spaces.
K EFFECTIVENESS OF USING THE SAME INITIALIZATION FOR INRS
The need to align the multitude of INRs that can approximate a given shape is a challenging research problem that has be dealt with when using INRs as input data. We empirically found that fixing the weights initialization to shared random vector across INRs is a viable and simple solution to this problem.
We report here experiment to assess if order of data, other sources of randomness arising while fitting INRs, do affect the repeatability of the embeddings computed by inr2vec. We fitted 10 INRs on the same discrete shape for 4 different chairs, i.e., 40 INRs in total. Then, embed all of them with the pretrained inr2vec encoder and compute the L2 distance between all pairs of embeddings. The block structure of the resulting distance matrix (see Fig. 30 highlights how, under the assumption of shared initialization and hyperparameters, inr2vec is repeatable across multiple fittings.
Seeking for a proof with a stronger theoretical foundation, we turn our attention to the recent work git re-basin (Ainsworth al. 2022). where authors show that the loss landscape of neural networks contain (nearly) a single basin after accounting for all possible permutation symmetries of hidden
0.75
0.20
0.15
0.10
35
0.00
Figure 30: L2 distances between inr2vec embeddings. For each shape, fit 10 INRs starting from the same weights initialization (40 INRs in total). Then we plot the L2 distances between the embeddings obtained by inr2vec for such INRs.
34

---

Published as a conference paper at ICLR 2023
Los5
Loss
Loss
Loss
1.5
10
Interpolation Factor
Interpolation Factor
Interpolation Factor
Interpolation Factor
Figure 31: Linear mode connectivity study. Each plot shows variation of the loss function over the same batch of points when interpolating between INRs representing the same shape. The red line deseribes the interpolation between INRs initialized differently, while the blue line shows the same interpolation between INRs initialized from the same random vector. The yellow stars represent the loss value of the boundary INRs.
units. The intuition behind this finding is that, given two neural networks that were trained with equivalent architectures but different random initializations, data orders, and potentially different hyperparameters or datasets, it is possible to find a permutation of the networks weights such that when linearly interpolating between their weights, all intermediate models enjoy performance similar to them- a phenomenon denoted as linear mode connectivity.
Intrigued by this finding, we conducted study to assess whether initializing INRs with the same random vector, which we found to be key to inr convergence, also leads to linear mode connectivity. Thus, given one shape, we fitted it with two different INRs and then we interpolated linearly their weights, observing at each interpolation step the loss value obtained by the interpolated INR on the same batch of points. For each shape, we repeated the experiment twice, once initializing the INRs with different random vectors and once initializing them with the same random vector.
The results of this experiment are reported for four different shapes in Fig. 31 is possible to note that, as shown by the blue curves, when interpolating between INRs obtained from the same weights initialization, the loss value at each interpolation step is nearly identical those of the boundary INRs. On the contrary, the red curves highlight how there is no linear mode connectivity at all between INRs obtained from different weights initializations.
(Ainsworth al. (022)) propose also different algorithms to estimate the permutation needed to obtain linear mode connectivity between two networks. We applied the algorithm proposed their paper in Section 3.2 (Matching Weights) our INRs and observed the resulting permutations. Remarkably, when applied to INRs obtained from the same weights initialization, the retrieved permutations are identity matrices, both when the target INRs represent the same shape and when they represent different ones. The permutations obtained for INRs obtained from different initializations, instead, far from being identity matrices.
All these results favor the hypothesis that our technique of initializing INRs with the same random vector leads to linear mode connectivity between different INRs. We believe that the possibility of performing meaningful linear interpolation between the weights occupying the same positions across different INRs can be interpreted by considering corresponding weights as carrying out the same role in terms of feature detection units, explaining why the inr2vec encoder succeeds in processing the weights of our INRs.
This intuition can be also combined with the finding of another recent work (Yúce et al. 2022). that shows how the expressive power of SIRENs restricted to functions that can be expressed a linear combination of certain harmonics of the first layer, which thus serves as basis for the range of learnable functions.
As stated above, the evidence of linear mode connectivity between INRs obtained from the same initialization leads us to believe that the weights of the first layer extract the same features across different INRs. Thus, same basis of harmonics for all our INRs. We argue that this explains why it is possible to remove the first layer of the INRs presented in input to inr2vec (as empirically proved in Appendix ID: if the basis is always the same, it sufficient to process the layers from the second onwards, that represent the coefficients of the basis harmonics combination, described in (Yúce et al. 2022).
can think of using the same random initialization as a way to obtain the
35

---

Published as a conference paper at ICLR 2023
ModeINet4O (point clouds)
(b) Manifold40 (meshes)
ShapeNet10 (voxels)
Figure 32: t-SNE visualizations of 2nrevec latent spaces. We plot the t-SNE components of the embeddings produced by inr2vec on the test sets of three datasets, ModelNet40 (left), Manifold40 (center) and Shapenet10 (right). Colors represent the different classes of the datasets.
L T-SNE VISUALIZATION OF inr2vec LATENT SPACE
We provide in Fig. 32 the t-SNE visualization of the embeddings produced by inrvec when presented with the test set INRs of three different datasets. Fig. 32a shows this visualization for INRs representing the point clouds from ModelNet40, Fig. 32b for INRs representing meshes from Manifold 40, and Fig. 32c for INRs obtained from the voxelized shapes in ShapeNet10.
The supervision signal adopted during the training of framework does not entail any kind of constraints w.r.l. the organization of the learned latent space. Indeed, this was not necessary for our ultimate goal i.e., performing downstream tasks on the produced embeddings. However, it is interesting to observe from the t-SNE plots that inrevec favors spontancously a semantic arrangement of the embeddings in the learned latent space, with INRs representing objects of the same category being mapped into close positions as shown by the colors representing the different classes the considered datasets.
M ABLATION ON INRS SIZE
Fig. 33 reports study that conducted to determine the size of the INRs adopted throughout experiments. More specifically, we considered three alternatives of SIREN, all featuring hidden layers but different number of hidden features, namely 128, 256 and 512 respectively.
In the figure we show how the three SIREN variants perform in terms of being able of representing properly the underlying signal, which in this example is the orange plane the left. Since needed to create datasets comprising a huge number of INRs, we considered both the quality of the representation well as the number of steps of the fitting procedure, with the goal of finding the best trade-off between quality and fitting time.
The results presented in the figure highlight how SIREN with 512 hidden features can learn to represent properly the input shape in just 600 steps, while the other variants either take longer (as in the case of 256 hidden features) or obtain at all the same quality (when using 128 hidden features).
This experiment enabled to draw the conclusion that SIREN with hidden layers and 512 hidden features is the proper tool to obtain a good quality INR in short time.
N SHAPE RETRIEVAL AND CLASSIFICATION ON DEEPSDF LATENT CODES
In Appendix A we show that frameworks that adopt a shared network to produce INRs struggle to obtain a good representation quality, while individual INRs do not suffer of this problem by design. In this section we goes one step further and consider the possibility of peforming downstream tasks on the latent codes produced by DeepSDF (Park al. 2019).
36

---

Published as a conference paper at ICLR 2023
Fitting
2G0
400
600
800
1000
steps
RXXX
GT
128
XX*XX
256
XX*
512
Number
hidden features
Figure 33: Comparison between different hidden sizes for INRs. We report the fitting steps needed to obtain good representation for INRs featuring different number of hidden features.
ModelNet40 mAP@5
ModelNer40
Method
mAP@
mAP@ IQ
Method
Accuracy
PointNet QQi ct al. 2017a
80.1
91.7
94.4
PointNet (Qi et al
2017a
88.8
PointNet++ (Q el al 201
85,1
93.9
96.0
PointNet+ + (Qu el a
201 7b
89.7
DGCNN WWag et al 20196
83.2
92.7
95.1
DGCNN (Wang et al.
20196
89.9
1nr2vec
81.7
92.6
95.1
1nr2vec
87.0
DeepSDF
89.8
DeepSDF
64.9
Table 9: Comparison between inr2vec and DeepSDF. We report results in shape retrieval (left) and shape classification (right) when using standard baselines, inr2vec embeddings DeepSDF latent codes.
In particular, trained DeepSDF to fit the UDFs of augmented version of ModelNet40, composed of 100K point clouds. For a fair comparison, we set the dimension DeepSDF latent codes to 1024 i.e., the same used for inr2vec embeddings. Then performed the experiments of shape retrieval and classification using DeepSDF latent codes, with the same settings presented in Section 4 for our framework.
The results reported in Table highhight that the poor representation quality obtained with DeepSDF and shown to be an intrinsic problem with shared network frameworks in Appendix A has detrimental effect on the quantitative results, proving once again that INR frameworks based on a shared network cannot be deployed as tool obtain and process INRs effectively.
SHAPE GENERATION: ADDITIONAL COMPARISON
In Fig. 8b we show a qualitative comparison between shapes generated with our framework (Section Learning mapping between inr2vec embedding spaces.) and with competing methods, i.e.. SP-GAN et al. 2021) for point clouds and NetwpansyNetworks (Mescheder et al.) 2019) for meshes.
In Fig. we extend this comparison, by presenting samples obtained with our formulation applied to the voxelized chairs of ShapeNet10 and comparing them with samples produced by two additional methods that learn a manifold of individual INRs, namely GEM (Du et al. (202I) and GASP (Dupont 2021bb) for which used the original source code released by the authors. To generate the figure, despite the sampled shapes being voxel grids, we adopt the same procedure used by GEM and GASP and reconstruct meshes by applying Marching Cubes to extract the 0.5 isosurface.
Fig. 34 show that all the considered methods can generate samples with a good variety in terms of geometry. However, it is possible to observe how the qualitative comparison favors the shape
37

---

Published as a conference paper at ICLR 2023
GEM
GASP
in2vec
Figure 34: Shape generation: qualitative comparison. We show samples generated with GEM (uu
et al. 2021), GASP (Dupont et al. 2021b) and with our method.
Generated with inr2vec, that appear smoother than the ones generated by GASP and less noisy that the samples produced by GEM.
P INR CLASSIFICATION TIME: EXTENDED ANALYSIS
We report here the extended analysis of the inference times reported in Fig. where we present the classification inference time needed to process udf INRs by standard point cloud baselines - PointNet (Qi al., 2017a), PointNet++ (Qi al. 2017b) and DGCNN (Wang al.) 2019b) and by inr2vec encoder paired with the fully-connected network that we adopt to classify the embeddings (see Section
The scenario that we had in mind while designing inr2vec is the one where INRs are the only medium represent 3D shapes, with discrete point clouds not being available. Thus, in Fig. for PointNet, PointNet++ and DGCNN we report the inference time including the time spent to reconstruct the discrete cloud from the input INR. In Fig, and Table for the sake of completeness. we report also the baselines inference times assuming the availability of discrete point clouds, stressing however that this is unlikely INRs become standalone format to represent 3D shapes.
The numbers plotted in Fig. 35 and reported in Table TO show clearly that our framework presents big advantage w.r.f. the competitors. Indeed, by processing directly INRs where the resolution of the underlying signal is theoretically infinite 2nr2vec can classify INRs representing point clouds with different number of points with a constant inference time of 0.001 seconds.
The considered baselines, instead, are negatively affected by the increasing resolution of the input point clouds. While the inference time of PointNet and PointNet++ is still affordable even when processing 64K points, DGCNN gets drastically slow already at 16K points. Furthermore, if point clouds need to be reconstructed from the input INRs, the resulting inference time become prohibitive for all the three baselines.
38

---

Published as a conference paper at ICLR 2023
w/out pcd reconstruction
w/ pcd reconstruction
Inference time (sec)
-log scale- e-
10c
10a
PointNet
PaintNet**
DGCNN
10-2
10
inr2vec
2048
16K
32K
64K
2048
16K
32K
64K
Number of points
Figure 35: Time required to classity INRs encoding udf. We plot the inference time of standard baselines and of our method, both considering the case which discrete point clouds are available (left) and the one where point clouds must be reconstructed from the input INRs (right).
Inference Time (seconds)
Method
2048 pts
16K pts
32K pts
64K pts
PointNet
0.001
0.002
0.003
0.007
PointNet*
0.171
1.315
2.609
5,230
PointNet++
0.013
0.026
0.034
0.036
*
0.185
1.293
2.672
5.287
DGCNN
0.158
1.285
4.788
19.26
DGCNN*
0.325
2.612
7.426
24.436
inr2vec
0.001
0.001
0.001
0.001
Table 10: Time required classify INRs encoding udf. All the times are computed on a gpu NVidia RTX 2080 Ti. indicates that the time to reconstruct the discrete point cloud from the INR is included.
39


Reviews:
[{"review_id": "zS4i4phXvjf", "reviewer": "Reviewer_shEg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "As implicit neural representations (INRs) is now becoming a popular 3D representation, especially for reconstruction tasks. This paper presents an interesting question: is it possible to use INR to replace explicit representation (like point cloud and mesh) for downstream tasks, such as classifaction or segmentation? Thus, a trial is presented: a method is designed for encoding INR weights into a vector which can be used for several tasks. ", "summary_of_the_review": "In whole, I appreciate the efforts for this trial. It is indeed an interesting attempt. However, I have only one concern is the necessity to do downstream tasks using INRs. ", "strength_and_weaknesses": "Strenth:\n* The trial to encode INRs weights to a compact code and try to use it for downstream tasks is interesting. \n* The experiments well-verifies the possibility. \n\nWeaknesses: \n* If this is a right direction, then I have no concerns. My major concern is: it is not clear why we need such a trial.  Point cloud is the raw 3D data format which can be directly captured by 3D sensors. And, mesh is the raw data format created from some 3D modeling tools, such as Maya and 3DMax. Then, if we can do classification on point cloud and mesh, why we need to convert them into INRs and then conduct downstream tasks. If the conversion between different representations become easier, then it is also no need to propose a unified representation. \n* The results do not show better performance. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SJucaKCBBT", "reviewer": "Reviewer_rRwy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes inr2vec, a framework for embedding implicit neural representations (INR) into a latent representation. The latent representations can then be used as data for various downstream tasks including classification, generative modeling and shape completion, effectively leading to a framework for performing deep learning directly on INRs. The authors learn such embeddings by training an autoencoder on the weights of INRs fitted to various 3D data, with the encoder mapping the weights to a latent vector and the decoder mapping the latent to the weights of an INR that is decoded into the data it represents. The authors perform a series of experiments on various 3D shape representations (point clouds, voxels, meshes) and various tasks (point cloud retrieval, classification, part segmentation, generative modeling, point cloud completion and surface reconstruction), with interesting results.", "summary_of_the_review": "Overall I believe this is a strong paper which should be of interest to the ICLR community. The paper presents a novel method for performing deep learning on INRs with interesting experiments. While the results are not SOTA and there are a few other minor weaknesses, I still believe this paper is strong and as such I recommend acceptance.", "strength_and_weaknesses": "**Strengths**\n- With the growing interest in INRs, the paper tackles an important and timely problem (performing deep learning on INRs) which should be of interest to the ICLR community\n- The paper is well written and clear\n- The proposed method makes sense and is well suited for the tasks in the paper\n- The experiments are well done and thorough. The authors tackle an impressively large number of tasks\n- The appendix is very thorough and includes a lot of helpful extra clarifications as well as experimental results\n\n**Weaknesses**\n- Experimental results do not match SOTA methods on most tasks considered. However, this is explicitly mentioned as a weakness in the conclusion and I don't believe this is crucial. As this paper consists in one of the first attempts to perform deep learning on INRs, it is reasonable to not expect SOTA results\n- As mentioned in the paper, the work is closely related to [1], which also proposes to perform various deep learning tasks on INRs. While it is briefly discussed in the paper, I believe readers would benefit from a more detailed comparison. For example, the authors mention that [1] rely on a shared network whereas the proposed method processes individual INRs. However, the encoder/decoder architecture is effectively shared across INRs too, and the encoding/decoding steps limit the reconstruction quality too. It would be nice to clarify this point in a little more detail in the paper. Further, while there are some experiments comparing the proposed model to [1] in the appendix, there are no comparisons on the tasks of classification and shape generation, both of which are tackled by [1]. For example, it seems like [1] also does classification on ShapeNet10 - why not include a comparison to this in the paper?\n- The evaluation of the generative model is quite weak. There are no quantitative metrics and no qualitative comparisons to other baselines. While quantitative evaluation of generative models of 3D shapes is difficult, it would be nice to at least have qualitative comparisons. For example, [2, 3] follow a very similar setup to the paper (generating task/modality-agnostic INRs) and also have experiments on 3D shapes, which might be worth comparing with\n- In the timing experiments (Figure 6), I think it makes sense to include the same experiments *without* the time required to reconstruct the point cloud from the INR. It is unclear from this experiment if most of the time is spent converting the INR to point clouds or if it's the actual point cloud model that is slow. While I understand the motivation for including the INR to point cloud conversion time, I think this experiment would greatly benefit from showing both the experiment with and without the conversion time\n\n**Questions**\n- As mentioned in Section 3, the encoder takes as input all the *hidden* layers of the INR and does not include the input and output layers. How are these layers treated? For example, given a latent embedding, you would still need to the input and output layer to reconstruct the signal. Are these just stored separately? I could not find any information explaining this in the paper, which I feel is a fairly important point\n\n**Nitpicks**\n- It seems like most citations are missing brackets. Using something like `\\citep{}` should help\n- Some of the wording can be slightly misleading. For example, in Section 4, \"inr2vec is able to match, and in some cases slightly surpass, the considered baselines\" does not really align with the results. It seems like the method underperforms PointNet++ (often by a significant margin) on 8/9 experiments. The wording should be changed to more accurately reflect this\n\n\n[1] From data to functa: Your data point is a function and you can treat it like one\n\n[2] Learning signal agnostic manifolds of neural fields\n\n[3] Generative models as distributions of functions", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9VljWvNc0M3", "reviewer": "Reviewer_HTw1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a framework that computes a compact latent vector for an input implicit neural representation for 3D shape (i.e., encoded with MLP weights). It's trained on a dataset of INRs via reconstruction loss, and is able to obtain a compact embedding from an unseen INR (but still from the same data distribution) at inference time. The produced embeddings can be readily used for downstream tasks like classification or segmentation without the need of decoding an actual discrete representation. Through experiments on several different tasks, the authors demonstrate that the proposed framework achieve results on-par with simple baselines using discrete representation.", "summary_of_the_review": "This paper tries to address an interesting problem of directly applying deep learning models on INRs. The proposed framework is somewhat novel but has some clear limitations that may prevents its real-world applications. Experiments are comprehensive but the method gets inferior performance on nearly all tasks, casting doubts on whether we really need such framework in the first place. Overall, the paper's weaknesses outweighs its strengths. Therefore, I'm inclined to rejection.\n\nUpdate: given that the authors' rebuttal addressed my major concerns, I have raised my score from 5 to 6.", "strength_and_weaknesses": "Strengths:\n\n- This paper tries to address an interesting problem --- how to process INRs directly for downsream tasks without reconstructing the discrete counterparts. Given that nowadays INRs are popular to represent 3D shapes, directly processing them is potential very useful, if can be done properly.\n\n- The experiments are very comprehensive and lots of results are presented. The proposed framework is evaluated on many downstream tasks ranging from classification to generation.\n\nWeaknesses:\n- The proposed framework has some technical drawbacks that limits its application scope. For example, all input INRs need to be initialized with the same random vector (page 4 bottom). This requirement for training data is acceptable but it also applies to input INRs in testing time. This severely limits the generalization ability and such requirement is probably even not possible in real-world where INRs can come from anywhere. In addition, all input INRs need to have the same network structures since the encoder of inr2vec is a MLP of fixed structure.\n\n- Though the experiments are very comprehensive and cover many tasks, the quantitative results are inferior to compared baselines on nearly all tasks. Then the proposed framework is less favourable. Traditional discrete representions are easy to obtain (e.g., point cloud scan) and are ready to be processed by existing models. INRs, on the contrary, are much hard to obtain as it requires network training but better represent the signal. So I would expect the proposed framework to have superior performance, such that it would be worth applying deep learning models directly on INRs. \n\n- Missing one important baseline. Auto-decoder INRs, where a single network fits the whole dataset by condiitoning on different learned embeddings, is an important baseline. As the authors suggested in page 2 (\"Earlier work in .... downstream tasks\"), it is the natural solution to this research problem. Even though it may have clear drawbacks (i.e., insufficint capacity), it is a direct baseline that should be compared to, especially for those discriminative tasks.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4eqsEkVib0b", "reviewer": "Reviewer_VZbJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper targets a novel {inr2vec} problem setting: to squeeze the weights of a learned implicit neural representation (INR) into compact latent codes for various downstream tasks. By training an encoder-decoder to recover the learned implicit function represented by the INR, the encoded feature can be fed into learning pipelines to solve tasks such as classification, retrieval, part segmentation, unconditioned generation, surface reconstruction, and completion. Detailed experiments verify the efficacy of the proposed method.", "summary_of_the_review": "Overall, I think it is an interesting paper and is worth spreading in the community. The main concerns are summarized in the [Cons] section. I am currently positive, and I would like to see how the author responds and the opinions of other reviewers.", "strength_and_weaknesses": "Pros:\n+ The paper targets an interesting yet not well-explored problem to directly encode the learned INR into a compact latent vector. The solution is simple yet effective. Thorough experiments are conducted to show empirically that the encoded feature captures the high-level cues inherent in the geometric neural representation.\n+ The writing and organization are clear and straightforward. The figures well demonstrate the key components of the paper. Overall, I enjoy reading the paper.\n+ The design of the encoder-decoder is interesting. Unlike traditional autoencoding that encodes the weight values in the embedding, the supervision for recovering the learned function may be the key to success. Besides, the transfer between different embedding spaces is also interesting and impressive. \n\nCons:\nThough experimentally verified, the paper lacks explanations and theoretic proofs on several critical issues.\n1. As proved in [A], each feature unit in the first layer serves as a single frequency basis, whose spectral support is gradually expanded after the non-linear activations. That is to say, the input layer is the most crucial part for an INR to define the spectral basis. It is hard to understand why the input layer should be discarded and why their use does not change the inr2vec performance (page 4). With slightly different input layers, the INRs define drastically different functions. Though the same initialization leads to linear mode connectivity, I don't think initialization with the same random vector can mitigate this issue.\n2. The property of the embedding space is not clearly demonstrated or analyzed (e.g., through t-sne). The invariance against noise, transformation, or the over-parameterized network (a shape can be represented by multiple network parameters) is unknown. According to the experimental results, the embedding captures both local (part segmentation) and global (classification and retrieval) information. The distance within the embedding space reflects the geometric/categorical similarity (interpolation and retrieval). However, it is unclear how such an ideal embedding space is guaranteed by the proposed supervision manner.\n\n\nMinor issues:\n1. SIREN with 256 units in each layer is able to recover nice geometric details in scene scale. I don't understand why the paper adopts a (512 units with 4 hidden layers) network configuration (page 6) for the coarse ShapeNet geometry.\n2. INRs provide fine-grained and continuous geometric cues. However, the learned embeddings perform worse than the compared point cloud features. Though I think the novelty is sufficient, remarks concerning this issue and insights for future potentials should be added.\n\n[A] A structured dictionary perspective on implicit neural representations. Gizem Yüce*, Guillermo Ortiz-Jimenez*, Beril Besbinar and Pascal Frossard, CVPR 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster