Higher-Order Function Networks for Learning Composable 3D Object Representations
Authors: Eric Mitchell
Selim Engin
Volkan Isler
Daniel D Lee
Venue:  (2020)
Keywords: computer vision
3d reconstruction
deep learning
representation learning

Abstract: We present a new approach to 3D object representation where a neural network encodes the geometry of an object directly into the weights and biases of a second 'mapping' network. This mapping network can be used to reconstruct an object by applying its encoded transformation to points randomly sampled from a simple geometric space, such as the unit sphere. We study the effectiveness of our method through various experiments on subsets of the ShapeNet dataset. We find that the proposed approach can reconstruct encoded objects with accuracy equal to or exceeding state-of-the-art methods with orders of magnitude fewer parameters. Our smallest mapping network has only about 7000 parameters and shows reconstruction quality on par with state-of-the-art object decoder architectures with millions of parameters. Further experiments on feature mixing through the composition of learned functions show that the encoding captures a meaningful subspace of objects.

Published as a conference paper at ICLR 2020
HIGHER-ORDER FUNCTION NETWORKS FOR LEARN- ING COMPOSABLE 3D OBJECT REPRESENTATIONS
Eric Mitchell
Selim Engin"
Volkan Isler?
Daniel D Lee²
ABSTRACT
We present a new approach to 3D object representation where a neural network encodes the geometry of object directly into the weights and biases of second "mapping' network. This mapping network can be used to reconstruct an object by applying its encoded transformation to points randomly sampled from simple geometric space, such as the unit sphere. We study the effectiveness of our method through various experiments on subsets of the ShapeNet dataset. We find that the proposed approach can reconstruct encoded objects with accuracy equal to or exceeding state-of-the-an methods with orders of magnitude fewer parameters. Our smallest mapping network has only about 7000 parameters and shows reconstruc- tion quality on par with state-of-the-an object decoder architectures with millions of parameters. Further experiments on feature mixing through the composition learned functions show that the encoding captures a meaningful subspace objects.
INTRODUCTION
This paper is primarily concerned with the problem of learning compact 3D object representations and estimating them from images. If we consider an object to be continuous surface in IR$ it is not straightforward to directly represent this infinite set of points in memory. In working around this problem, many learning-base -ased approaches to 3D object representation suffer from problems related to memory usage, computational burden, or sampling efficiency. Nonetheless, neural networks with tens of millions of parameters have proven effective tools for learning expressive representations of geometric data. In this work, we show that object geometries can be encoded into neural networks with thousands, rather than millions, of parameters with little or no loss in reconstruction quality.
To this end, we propose an object representation that encodes an object as a function that maps points from canonical space, such as the unit sphere, the set of points defining the object. In this work, the function is approximated with a small multilayer perceptron. The parameters of this function are estimated by a 'higher order' encoder network, thus motivating the name for our method: Higher-Ores tigeer-Order Function networks (HOF). This procedure is shown in Figure 1. There are two key ideas that distinguish HOF from prior work in 3D object representation learning: fast weights object encoding and interpolation through function composition.
(1) Fast-weights object encoding: FFsgtweiighss' in this context generally refers to methods that use network weights and biases that are not fixed; at least some of these parameters are estimated on per-sample basis. Our fast-weights approach stands in contrast to existing methods which encode objects vector-valued inputs to a decoder network with fixed weights. Empirically, find that our approach enables a dramatic reduction (two orders of magnitude) the size of the mapping network compared to the decoder networks employed by other methods.
(2) Interpolation through function composition: Our functional formulation allows for interpolation between inputs by composing the roots of our reconstruction functions. We demonstrate that the
*ttanford University Samsung Al Center New York University of Minnesota Work performed
while an intern Samsung Al Center New York. See / /aai c-ny github. faccuu.../oooooo. and additional information. Correspondence to: Erie Mitchell hwnc.mmcchelleecsssttt....c
io/hof f for code

---

Published as a conference paper at ICLR 2020
9 8¹ Rebu6te3 + R101
f R R
0
X,EX
0
Overview of HOF. The encoder network Is encodes the geometry of the object pictured in each input image directly into the parameters of the mapping function fe. which produces a reconstruction as transformation of a canonical object (here, the unit sphere). Bottom: We visualize the transformation fo by showing various subsets of the inputs X and their corresponding mapped locations in red and green, respectively. In each frame, light gray shows the rest of X and dark gray shows the rest of the reconstructed object.
Figure 1: Top:
functional representation learned by HOF provides a rich latent space in which we can 'interpolate' between objects, producing new, coherent objects sharing properties of the 'parent' objects.
In order to position HOF among other methods for 3D reconstruction, we first define a taxonomy of existing work and show that HOF provides generalization of current best-performing methods. Afterwards, we demonstrate the effectiveness of HOF on the task of 3D reconstruction from an RGB image using subset of the ShapeNet dataset (Chang et al. 2015). The results, reported in Tables [ and 2 and Figure 2 show statc-of-the-an reconstruction quality using orders of magnitude fewer parameters than other methods.
2 RELATED WORK
The selection of object representation a crucial design choice for methods addressing 3D recon- struction. Voxel-based approaches (Choy et al. 2016 Häne et al. 2017) typically use uniform discretization of IR3 in order to extend highly successful convolutional neural network (CNN) based approaches to three dimensions. However, the inherent sparsity of surfaces in 3D space make vox- elization inefficient in terms of both memory and computation time. Partition-bases approaches such as octrees (Tatarchenko et al. 20177] Riegler al., 2017) address the space efficiency shortcomings of voxelization, but they are tedious to implement and more computationally demanding to query. Graph-based models such as meshes (Wang et al. 2018 Gkioxari et al. 2019; /Smith et al., 2019 Hanocka et al. 2019) provide a compact representation for capturing topology and surface level information, however their irregular structure makes them harder to learn. Point set representations, discrete (and typically finite) subsets of the continuous geometric object, have also gained popularity due to the fact that they retain the simplicity of voxel based methods while eliminating their storage and computational burden (Qi et al. 201 7a Fan et al. 2017 Qi et al. 2017b: Yang et al. 2018 Park al. 201 7a b) was an architectural milestone that made manipulating point sets with deep learning methods a competitive alternative to earlier approaches; however, PointNet is concerned with processing, rather than generating, point clouds. Further, while point clouds are more flexible than voxels in terms of information density, it is still not obvious how
let al. 2019). The PointNet architecture (Qi et al.
2

---

Published as a conference paper at ICLR 2020
Input Image
Ground Truth
FoldingNet
DeepSIDF
HOF
Figure From left to right: Input RGB image, ground truth point cloud, reconstruction from FoldingNet (Yang al. 2018). reconstruction from DeepSDD (Park et al. 2019), and method.
to adapt them to the task of producing arbitrary- varied-resolltion predictions. Independently regressing each point in the point set requires additional parameters for each additional point (Fan et al. 2017 Achlioptas al. 2018), which is undesirable property if the goal is high nigh-resolution point clouds.
Many current approaches to representation and reconstruction follow encoder-decoder paradigm, where the encoder and decoder both have learned weights that are fixed at the end of training. An image of 3D points is encoded
latent vector 'codeword' either with learned encoder as in Yang et al. 2018 Lin et al. (2018); Yan et al. (2016) or by direct optimization of the latent vector reconstruction based objective function in Park et al. (2019). Afterwards, the latent code is decoded by learned decoder into reconstruction of the desired object by one of two methods, which we call direct decoding and contextual mapping. Direct decoding methods directly map the latent code into a fixed set of points (Choy et al., [0166] Fan et al.) 2017 Lin et al.) 2018 Michalkiewicz al. 2019); contextual mapping methods map the latent code into function that can be sampled or otherwise manipulated to acquire reconstruction (Yang et al. 2018 Park ct al. 2019 Michalkiewicz et al. 2019 Mescheder et al., 2019). Direct decoding methods generally suffer from the limitation that their predictions are of fixed resolution; they cannot be sampled more or less precisely. With contextual mapping methods, is possible in principle to sample the object to arbitrarily high resolution with the correct decoder function. However, sampling can provide significant computational burden for some contextual mapping approaches as those proposed by Park al. (2019) and Michalkiewicz et al. (2019). Another hurdle the need for post-proe post-procsssing such as applying the Marching Cubes algorithm developed by Lorensen and Cline (1987). We call contextual mapping approaches that encode context by concatenating duplicate of latent context vector with each input latent vector concatenation (LVC) methods. In particular, we compare with LVC architectures used FoldingNet (Yang et al.) 2018) and DeepSDF (Park al. 2019).
itself with respect
HOF is contextual mapping method that distinguishes itself from other methods within this class through its approach to representing the mapping function: HOF uses one neural network to estimate the weights of another. Conceptually related methods have been previously studied under nomenclature such as the 'fast-weight" paradigm (Schmidhuber 1992 De Brabandere et al. 2016) Klein et al. 2015 Riegler et al. (2015) and more recently 'hypernetworks' (Ha et al. 2016). However, the work by Schmidhuber 1992) deals with encoding memories in sequence learning tasks. Ha et al. (2016) suggest that estimating weights of network with another might lead improvements in
However, this work does not leverage the key insight of using network parameters that are estimated per sample in vision tasks.
3

---

Published as a conference paper at ICLR 2020
3 HIGHER-ORDER FUNCTION NETWORKS
HOF is motivated by the independent observations by both Yang et al. (2018) and Park et al. (0199) that LVC methods do not perform competitively when the context vector is injected by simply concatenating it with each input. In both works, the LVC methods proposed required architectural
workarounds to produce sufficient performance on reconstruction tasks, including injecting the latent code multiple times at various layers in the network. HOF does not suffer from these shortcomings due its richer context encoding (the entire mapping network encodes context) in comparison Quantitative
with LVC. We compare the HOF and LVC regimes more precisely in Section 3.2 comparisons of HOF with existing methods can be found in Table 1
3.1 A WAST-WEIGHTS APPROACH TO 3D OBJECT REPRESENTATION AND RECONSTRUCTION
We consider the task of reconstructing object point cloud O from image. We start by training neural network Ge with parameters Φi (Figure top-left) to output the parameters 0 of a mapping function fe. which reconstructs the object when applied to a set of points X sampled uniformly from a canonical set such as the unit sphere (Figure top-right). We note that the number of samples in X can be increased or decreased to produce higher or lower resolution reconstructions without changing the network architecture or retraining, in contrast with direct decoding methods and some contextual mapping methods which use fixed, non-random samples from X (Yang et al., 2018). The input to Je an RGB image /; our implementation takes 64 x 64 3 RGB images inpur, but our method is general to any input representation
which corresponding differentiable encoder network can be constructed to estimate 9 (c.g. PointNet (Qi et al. 2017a) for point cloud completion). Given I, compute the parameters of the mapping network 8 as
8 9o
That is, the encoder dy 1R3x64464 + Rd directly regresses the d-dimensional parameters O, of the mapping network fo, R - R&, which maps c-dimensional points in the canonical space X to points in the reconstruction Ô (see Figure [.. We then transform our canonical space X with fe, in the same manner as other contextual mapping methods:
Ô = (fo, (xi) x; X
(2)
During training, we sample an image and the corresponding ground truth point cloud model O where O contains 10,000 points sampled from the surface of the true object. We then obtain the mapping Jo, = 9o ( and produce an estimated reconstruction of as in Equation 2 In our training, we only compute Jo, (x) for a sample of 1000 points in X. However, we find that sampling many more points (10-100x as many) at test time still yields high-quality reconstructions. This sample drawn from a uniform distribution over the set X. We then compute a loss for the prediction Ô using a differentiable set similarity metric such as Chamfer distance or Earth Mover's Distance. We focus on the Chamfer distance as both a training objective and metric for assessing reconstruction quality. The asymmetric Chamfer distance CD(X,Y) Y) is often used for quantifying the similarity of two point sets X and Y and is given as
CD(X,Y) = X
Σ
min |
1
(3)
x;EX
The Chamfer distance is defined even if sets X and Y have different cardinality. We train Jo to minimize the symmetric objective function E(Ō, 0) CD(O, + CD(O,)) as in Fan al. (217)).
3.2 COMPARISON WITH LVC METHODS
We compare our mapping approach with LVC architectures such as DeepSDF (Park et al.) 2019 and FoldingNet YYagg et al. 2018). These architectures control the output of the decoder through the concatenation of a latent "oodwoord vector z with each input x; E X. The codeword is estimated by an encoder Jouve for each image. We consider the case in which the latent vector is only concatenated
4

---

Published as a conference paper at ICLR 2020
with inputs in the first layer of the decoder network fo. which assume be MLP. We interested in analyzing the manner in which the network output with respect to x; may be modulated by varying
If the vector a; contains the pre-activations of the first layer of fo given an input point x, we have a; W* + 2 + b
where W W and b are fixed parameters of the decoder, and only z is function of I. If we absorb the parameters W and b into the encoder parameters ΦLVC (as W and b are fixed for all x;), we can define new, equivalent latent representation b* W + b W gonve (T) + b and new encoder function / with parameters Φive U { b) such that h(T) = b" This gives
aj W x1 + h(1)
Thus the LVC approach is equivalent estimating fixed subset of the parameters @ of the decoder fo on a per-sample basis (the first layer bias). From this perspective, HOF is an intuitive generalization: rather than estimating just the first layer bias, we allow our encoder to modulate all of the parameters in the decoder fe on a per-sample basis.
Having demonstrated HOF generalization of existing contextual mapping methods, in the next section, we present novel application of contextual mapping that leverages the compositionality of the estimated mapping functions aggregate features of multiple objects or multiple viewpoints of the same object.
3.3 EXTENDING CONTEXTUAL MAPPING METHODS: FEATURE AGGREGATION THROUGH FUNCTION COMPOSITION
An advantageous property of methods that use a latent codeword is that they have been empirically
shown to learn a meaningful space of object geometries, in which interpolating between object encodings gives new, coherent object encodings (Fan et al. 2017 Yang et al., 2018 Park et al. 2019
Groueix et al., 2018). HOF, on the other hand, does not obviously share this property: interpolating between the mapping function parameters estimated for two different objects need not yield new, coherent object as the prior work has shown that the solution space of 'good' neural networks is highly non-convex (Li al. 2018). We demonstrate empirically in Figure that naively interpolating between reconstruction function in the HOF regime does indeed produce meaningless blobs. However, with small modification to the HOF formulation in Equation 2 we can in fact learn rich space of functions in which we can interpolate between objects through function composition.
We extend the formulation in Equation 2 to where object is represented as the k-th power of the mapping fe;:
(4) where f is defined as the composition of f with itself (k- 1) times: f*(x)= ((F(k-1 where (x) A x. We call a mapping Sa, whose k-th power reconstructs the object O in image the k-mapping for O.
Ô ( E
This modification to Equation 2 adds an additional constraint to the mapping: the domain and codomain must be the same. However, evaluating powers of leverages the power of weight sharing in neural network architectures; for an MLP mapping architecture with I layers (excluding the input layer), evaluating its k-thh power is equivalent to an MLP with 1 layers with shared weights. This formulation also has connections to earlier work on continuous attractor networks as model for encoding memories in the brain as becomes large (Seung) 1998).
In Section 4.3 we conduct experiments in setting in which we have acquired RGB images / and J of two objects, O₂ and O,, respectively. Applying our encoder to these images, we obtain k-mappings fe, and fez. which have parameters 0, g (I) and 0, g)))). respectively. We hypothesize that we can combine the information contained in each mapping function fo, by evaluating any of the 2 possible functions of the form:
(5) where the parameters of each mapping fo, are either the parameters of fe, or fo,- Figures FaguresAand]] show that interpolation with function composition provides interesting, meaningful outputs in experiments with A = 2 and k- 4.
funcrp = (fe, o... o fex)
5

---

Published as a conference paper at ICLR 2020
Table I: Comparing various reconstruction architectures. Reported Chamfer distance values are multiplied by 100 for readability and include standard error in parentheses. HOF-1 and HOF-3 are HOF variants with I and hidden layers, respectively.
Method
CD(P.))
CD(T.P)
Average CD
Parameters
Layers
3D-R2N2
2.588
3.342
2.965
2,000,000+
12
PSG
1.982
2.146
2.064
EPCG
1.846
1,701
1.774
60,000,000+
FoldingNet
1.563 (0.0064)
1.534 (0.0053)
1.549 (0.0117)
1,056,775
6
DeepSDF
1.521 (0.0029)
0.962 (0.0018)
1.242 (0.0047)
1,578,241
HOF-1
1.517 (0.0040)
1.021 (0.0028)
1.269 (0.0068)
7,171
2
HOF-3
1.480 (0.0028)
1.014 (0.0020)
1.247 (0.0048)
34,052
4
EXPERIMENTAL EVALUATIONS
We conduct various empirical studies in order to justify two key claims, In Sections 4.1 and 4.2
compare with other contextual mapping architectures to demonstrate that HOF provides equal or better performance with significant reduction in parameters and compute time. In Section 4.3 demonstrate that extending contextual mapping approaches such as HOF with multiple compositions of the mapping function provides simple and effective approach to object interpolation. Further experimentation, including ablation studies and a simulated robot navigation scenario, can be found
in A.7
4.1 EVALUATING RECONSTRUCTION QUALITY ON SHAPENET
We test HOF's ability to reconstruct 3D point cloud of an object given single RGB image, comparing with other architectures for 3D reconstruction. We conduct two experiments:
1. We compare HOF with LVC architectures to show that HOF is a more parameter-efficient approach to contextual mapping than existing fixed-decoder architectures.
2. We compare HOF broader of state of the art methods larger subset of the ShapeNet dataset, demonstrating that it matches or surpasses existing state of the art methods in terms of reconstruction quality.
In the first experiment, we compare HOF with other LVC architectures on 13 of the largest classes of ShapeNet (Yan et al. 2016), using the asymmetric Chamfer distance metrics (Equation 3 as reported [Lin et al. (2018). 3D reconstruction on a broader selection of 55 classes the ShapeNet dataset, as in Tatarchenko et al.) (2019). In line with the recommendations Tatarchenko al. (2019), we report scores for this evaluation.
the second experiment, we compare HOF with other methods for
4.1.1 LVC ARCHITECTRRE COMPARISON
For this experiment, we compare HOF with LVC decoder architectures proposed in the literature. specifically those used in DeepSDF (Park et al. 2019) and FoldingNet (Yang et al.) 2018), as well several other baselines. Each architecture maps points from I to 3 in order to enable direct comparison. The dataset contains 31773 ground truth point cloud models for training/validation and 7926 for testing. For each point cloud, there 24 RGB renderings of the object from fixed set of 24 camera positions. For both training and testing, each point cloud is shifted so that its bounding box center is the origin in line with Fan al. (2017). At test time, there no post-processing performed on the predicted point cloud. The architectures we compare in this experiment are:
1. HOF-1: 1 hidden layer containing 1024 hidden neurons
2. HOF-3: 3 hidden layers containing 128 hidden neurons
3. DeepSDF as described in Park et al. (2019), with hidden layers containing 512 neurons each

---

Published as a conference paper at ICLR 2020
4. FoldingNet described in Yang al. (2018), with 2 successive 'folds', each with a 3-layer MLP with 512 hidden neurons
5. EPCG architecture as reported in Lin et al. 2018
6. Point Set Generation network (Fan et al.
2017 as reported in Lin et al. (0118
7. 3D-R2D2 (Choy et al.,
2016 as reported in Lin et al. (0018)
Results reported in Table 1 Chamfer Distance scores are scaled by 100 as in line with Lin al. (2018). We find that HOF performs significantly better than that direct decoding baseline of Lin et al. (2018) and performs on par with other contextual mapping approaches with 30x fewer parameters, In order to provide fair comparison with the baseline method, we ensure that ground truth objects are scaled identically to those in Lin et al. (2018). We report both 'forward' Chamfer distance CD(Pred, Target) and 'backward' Chamfer distance CD(Target, Pred), again in line with the contains class-wise breakdown. Qualitative comparisons of the outputs of HOF with state -ftte----thh-attttt architectures are shown in Figure 2
convention established by [in et al.] (2018). Table 7
4.1.2 SHAPENET BREADTH COMPARISON
Tatarchenko al. (019)) question the common practice in single-view 3D reconstruction of eval- uating only on the largest classes of ShapeNet. The authors demonstrate that reconstruction methods do not exhibit performance correlated with the size of object classes, and thus evaluating on smaller ShapeNet classes justified. We use the dataset provided by TTtarccenkk et al. (2019), which includes 55 classes from the ShapeNet dataset. The authors also suggest using the Fl score metric, defined the harmonic mean between precision and recall (Tatarchenko et al. 2019).
Table 2: F1/class weighted Fl score comparison of HOF with methods re- ported in Tatarchenko et al. (019). Higher is better.
We include comparisons with AtlasNet (Groueix et al. 2018), Octree Generating Networks (Tatarchenko et al.
Method
F1
cw-F1
Oracle NN
0.290
0.321
2017), Matryoshka Networks (Richter and Roth (2018). and retrieval baselines as reported by [atarccenkoo et al. (2019). We find that HOF performs competitively with all of these state-of-the-art methods, even surpassing them on many classes. We show summary statisties in Table The Fl column contains the average Fl score for each method, uniformly averaging over all classes regardless of how class imbalances in the testing set. The cw-Fl score column averages over class F1 scores weighted by the fraction of the dataset comprised by that class; that is, classes that are over-represented in the testing set are correspondingly over-represented in the cw-Fl score. On the mean Fl metric, HOF outperforms all other methods, including the 'Oracle Nearest- "eirest-Neighbor' approach described by TTtarccenko et al. (2019). The Oracle Nearest Neighbor the closest object in the training set as its prediction for each test sample. A complete class- wise performance breakdown is in the Appendix in Table g We find that HOF outperforms all comparison methods (including the Oracle) in 23 of the 55 classes. Excluding the oracle, HOF shows the best performance 28 of the 55 classes.
AtlasNet
0.252
0.287
OGN
0.217
0.230
Matryoshka
0.264
0.297
Retrieval
0.237
0.260
HOF (Ours)
0.291
0.310
4.2 RUNTIME PERFORMANCE COMPARISON
We compare HOF with the decoder architectures proposed in Park et al. (019) and Yang et al. 2018) in terms of inference speed. Figure 3 shows the results of this experiment, comparing how long takes for each network to map set of N samples from the canonical space X into the object reconstruction. We ignore the processing time for estimating the latent state z for DeepSDFFFollinggee and the function parameters 0 for HOF; we use the same convolutional neural network architecture with reconstructions (N > 1000),
modified output layer for both. We find that even for medium-resolutinn the GPU rumning times for the DeepSDF/FoldingNet architectures and HOF begin to diverge. This difference is even more noticeable in the CPU rumning time comparison (an almost 100x difference). This performance improvement may be significant for embedded systems that need to efficiently
7

---

Published as a conference paper at ICLR 2020
Runtime Comparison (GPU)
Comparison
HOF (opul
HDF 1 (cpu)
40F (gpu)
4GF (cpu)
DeepãOF (epu)
roldingiit (gps)
Foldeggint lepu)
Tima compute
Time compute
10-
1071
Number of points sampled
Number points sampled
Figure 3: Runtime analysis comparing HOF with DeepSDF and FoldingNet architectures. HOF-1 and HOF-3 are HOF with and 3 hidden layers, respectively. Computing environment details are given Section B.2
store and reconstruct 3D objects real time; our representation is small in size, straightforward to sample uniformly (unlike CAD model), and fast to evaluate.
4.3 OBJECT INTERPOLATION
To demonstrate that our functional representation yields an expressive latent object space, we show that the composition of these functions produces interesting, new objects. The top of Figure shows ssows in detail the composition procedure. If we have estimated 2-mappings for two objects O and OB, we demonstrate that fox (fog (X)) and fog (fo, (X)) both provide interesting mixtures of the two objects and mix the features of the objects in different ways; the functions are not commutative. This approach is conceptually distinct from other object interpolation methods, which decode the interpolation of two different latent vectors. In our formulation, visualize the outputs of encoder that has been trained to output 2-mappings in R In addition, the bottom of Figure demonstrates a smooth gradient of compositions of the reconstruction functions for two airplanes, when higher order of mappings (k: = 4) is used.
To further convey the expressiveness of the composition-baset object interpolation, we compare it against a method that performs interpolation the network parameter space. This latter approach resembles common way of performing object interpolation LVC methods: Generate latent codewords from each image, and synthesize new objects by feeding the interpolated latent vectors into the decoder. As proxy for the latent vector interpolation used in LVC methods, generate new objects as follows. After outputting the network parameters 9 and 0g for the objects O and OB, we use the interpolated parameters 8' (0, 0n2/2 to represent the mapping function. In Figure we show that our composition-base interpolation is more capable of generating coherent new objects whose geometric features inherited from the source objects are preserved better.
5 CONCLUSION AND FUTURE WORK
We presented Higher Order Function Networks (HOF), which generate a functional representation of an object from an RGB image. The function can be represented as a small MLP with Ffattweeihtt or weights that are output by an encoder network 9s with learned, fixed weights. HOF demonstrates state-of-thee-rt reconstruction quality, as measured by Chamfer distance and F1 score with ground truth models, with far fewer decoder parameters than existing methods. Additionally, we extended contextual mapping methods to allow for interpolation between objects by composing the roots of their corresponding mapping functions. Another advantage of HOF is that points the surface can be sampled directly, without expensive post- poose-prcessingg methods such as estimating level sets.
For future work, would like further improve the parameter-efficiencyy of HOF, for example with versions of HOF that output only a sparse but flexible subset of the parameters of the mapping function. In addition, connections with other works investigating the properties of 'high-quality'
8

---

Published as a conference paper at ICLR 2020
Ob
1,0)
f,00
1,00
00000000
FilS 000)
fA0000)
fd00000
Oa
On
[xc/A=fa=/x() fx
[a= Jo JAIX)
Js Ino
Figure 4: Top Left. An example of inter-class interpolation between two objects by function composition. We show the ground truth objects O and Og, single evaluation of their respective decoding functions (giving fA (X) and fa (x)), as well as the possible permutations of compositions, which makes up the leaf nodes in each tree. In (x)) we see the wings straighten but remain narrow. In (a(n(()), we observe the wings broaden, but they remain angled. Top Right. An example of inter-class interpolation, mixing table and rifle. We observe what might be interpreted as gun with legs in fa Bottom. An example of intra-class interpolation between objects with l
and a table with single coherent stock in ff))
neural network parameters and initializations such as HyperNetworks (Ha et al. 2016). the Lottery Ticket Hypothesis (Frankle and Carbin 2018). and model-agnostic meta learning (Finn et al. 2017).
There also many interesting applications of HOF in domains such robotics. A demonstrative application in motion planning can be found in Appendix [.2.2] and Engin et al. (2020) explore extensions of HOF for multi-view reconstruction and motion planning. Using functional representa- tions directly for example for manipulation or navigation tasks, rather than generating intermediate 3D point clouds, is also an interesting avenue of future work. We hope that the ideas presented in this paper provides a basis for future developments of efficient 3D object representations and neural network architectures.

---

Published as a conference paper at ICLR 2020
REFERENCES
Angel X. Chang, Thomas A. Funkhouser, Lconidas J. Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. Shapenet: An information-rich 3d model repository. CoRR, abs/1512.03012, 2015. URL
http://arxiv.. wttp:///ggabbaoooo.o http://arxXvv.or//////55222
03012
Christopher B Choy, Danfei Xu, Jun Young Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In European Conference on Computer Vision, pages 628-644. Springer, 2016.
Christian Häne, Shubham Tulsiani, and Jitendra Malik. Hierarchical surface prediction for 3d object reconstruction. In 2017 International Conference on 3D Vision (3DV), pages 412-420. IEEE, 2017.
Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox. Octree generating networks: Efficient convolutional architectures for high-resolution 3d outputs. In Proceedings of the IEEE International Conference on Computer Vision, pages 2088-2096, 2017.
Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. Octnet: Learning deep 3d representations at high resolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3577-3586, 2017.
Nanyang Wang. Yinda Zhang. Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings of the European Conference on Computer Vision (ECCV), pages 52-67, 2018.
Georgia Gkioxari, Jitendra Malik, and Justin Johnson. Mesh r-cnn. In Proceedings of the IEEE International Conference Computer Vision, 2019.
Edward Smith, Scott Fujimoto, Adriana Romero, and David Meger. GEOMetrics: Exploiting geometric structure for graph-encoded objects. In Proceedings of the 36th International Conference on Machine Learning, pages 5866-5878 2019.
Rana Hanocka, Amir Hertz, Noa Fish, Raja Giryes, Shachar Fleishman, and Daniel Cohen-Or. Meshenn: network with edge. ACM Transactions Graphics (TOG), 38(4):90, 2019.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 652-660, 2017a.
Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2017.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets metric space. In Advances in Neural Information Processing Systems, pages 5099-5108, 2017b.
Yaoqing Yang. Chen Feng. Yiru Shen, and Dong Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 206-215, 2018.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 165-174, 2019.
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds. In International Conference on Machine Learning. pages 40-49 2018
Chen-Hsuan Lin, Chen Kong, and Simon Lucey. Learning efficient point cloud generation for dense 3d object reconstruction. In Thirty-Second AAAI Conference Artificial Intelligence, 2018.
10

---

Published as a conference paper at ICLR 2020
Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee. Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision. In Advances in Neural Information Processing Systems, pages 1696-1704, 2016.
Mateusz Michalkiewicz, Jhony K. Pontes, Dominie Jack, Mahsa Baktashmotlagh, and Anders Eriksson. Deep level sets: Implicit surface representations for 3d shape inference. CoRR, abs/1901.06802. 2019.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4460-4470, 2019.
William E. Lorensen and Harvey F. Cline. Marching cubes: A high resolution 3d surface construction algorithm. In Maureen C. Stone, editor, SIGGRAPH, pages 163-169. ACM, 1987. ISBN 0-89791- 227-6. URL httt::/dblpp uni terrtrrerree de http://////////////eeeeeeee http://////ooooooaogg/////////// html #LorensenC87
Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks, Neural Computation, 4(1):131-139, 1992. doi: 10.1162/neco.11922...133 URL
https: (tt.a//.rrg/...112////...
Bert De Brabandere, Xu Jia, Tinne Tuytelaars, and Luc Van Gool. Dynamic filter networks. In NIPS, 2016.
Benjamin Klein, Lior Wolf, and Yehuda Afek. A dynamic convolutional layer for short rangeweather prediction. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, June 2015, doi: 10.1 10.110000v1::2211529991111 17. URL http://://ad..or/1101110000 evpr 22015.799110
Gernot Riegler, Samuel Schulter, Matthias Ruther, and Horst Bischof. Conditioned regression models for non-blind single image super-resolution. 2015 IEEE International Conference on Computer Vision (ICCV). IEEE, December 2015. doi: 10. 20.61 URL https
//doi. 1109/11c0ie.. 2015.67
David Ha, Andrew M. Dai, and Quoc Le. Hypernetworks. CoRR, absp//09.09190000 2016. URL
http::/
Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. A papier-máché approach learning 3d surface generation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 216-224, 2018.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets.
Bengio, Wallach, H. Larochelle, K. Grauman, Cesa- Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31. pages 6389-63838 6399. Curran Associates, Inc., 2018. URL http: htt http:///aapeee.e/ee .ips www.niiwss..paae.em wwppppe///pppppccpcppocpcee..eap 7875- 1175-vissalizaee
the loss l of www.-nnnuaalaeeseeee
H. Sebastian Seung. Continuous attractors and oculomotor control. Neural Nenworks, 11(7-8): October 1998. doi: 10.1016/s0893-6600 6080(98)00064-11. URL httttttt/pooo.oo/t org/
10.101660883--0009990006666
10. 00110166608899 6080 (98) 60064-1]
Maxim Tatarchenko, Stephan R Richter, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox. What do single-view 3d reconstruction networks learn? In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition, pages 3405-3414, 2019.
Stephan R. Richter and Stefan Roth. Matryoshka networks: Predicting 3d geometry via nested shape layers. In CVPR, pages 1336-1944. IEEE Computer Society, 2018
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Training pruned neural networks. CoRR, abs/1803.00 03665,, 2018. URL http:tt:://axxxi.
1833 03635
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic leta-le for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, pages 1126-11 135. JMLR. org, 2017.
11

---

Published as a conference paper at ICLR 2020
Selim Engin, Eric Mitchell, Daewon Lee, Volkan Isler, and Daniel D Lee. Higher order function networks for view planning and multi-view reconstruction. In International Conference on Robotics and Automation (ICRA). IEEE, 2020.
Jiajun Wu, Chengkai Zhang, Xiuming Zhang, Zhoutong Zhang, William T Freeman, and Joshua B Tenenbaum. Learning shape priors for single-view 3d completion and reconstruction. In Proceed- ings of the European Conference on Computer Vision (ECCV), pages
2018.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 4700-4708, 2017.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Geoffrey Gordon, David Dunson, and Miroslav Dudík, editors, Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pages 315-323, Fort Lauderdale, FL, USA, 11-13 Apr 2011. PMLR. URL http:///prooeeecing.....k www.:/pwssssee.eee org.eecccccccnnloooo.ooe////ooo.
12

---

Published as a conference paper at ICLR 2020
A
ADDITIONAL EXPERIMENTS
A.1
ARCHITECTURAL VARIATIONS
We compare HOF trained on the subset ShapeNet used in [Lin et al. (0188) with several architectural variations, including using Resnet18 rather than our own encoder architecture, using a fest-weights decoder architecture with 6, rather than 3, hidden layers, and using tanh rather than relu in the fast-weights decoder. Results are summarized in Table 3 We find that HOF is competitive in all of the formulations, although using the tanh activation function in the decoder function instead of the relu shows small degradation in performance. Future work might investigate more deeply what principles underlie the design of fast-weights architectures.
Table 3: Comparisons of class-weighted asymmetric Chamfer distances (CD(P.T) and CD(T.P) of architectural variants of HOF-3 (with k 1).
Base (HOF-3)
ResNet18 Encoder
Deep Decoder
Tanh (Decoder only)
1.480 / 1,014
1.529 0.989
1.499 / 0.998
1.652 1.066
A.2 THE ROLE OF THE SAMPLING DOMAIN
In the experimental results reported in the main paper. we sample imput points from the interior of the unit sphere uniformly at random. However, we might sample other topologies as input to the mapping function. this experiment, we compare the performance of HOF when we sample from the interior of the 3D sphere (standard configuration), the surface of the 3D sphere, the interior of the 3D cube, and the interior of the 4D sphere.
Table 4: Comparisons of HOF-3 using various different canonical objects for sampling.
3D Sphere
3D Sphere (Surface)
4D Sphere
3D Cube
1,480 / 1.014
1.494 1.263
1.478 0.912
1.481 0.999
A.3 COMPARING DIFFERENT VALUES OF K FOR COMPOSITION
Here, we compare the performance of various instances of HOF when we vary the value of k. or number of self-compositions. We use the HOF-1 decoder architecture (I hidden layer with 1024 neurons) and a Resnet 18 encoder architecture. This encoder architecture is different from the baseline encoder used for the results in Table I, which explains deviations in performance from those numbers. Although performance does not vary significantly, there are some differences in reconstruction quality across values of k. Future work might investigate how best to take use of self-compositional or recurrent decoder architectures to maximize both computational performance as well as accuracy.
Table 5: Comparisons of Resnet18 HOF-1 with different values of k
A:
k = 2
k:
h: = 4
1.635 / 1.008 1.574 / 1.000 1.601 1.081
1.655 0.953
A.4 PROJECTION REGULARIZATION
Intuitively, we might expect that fe( would approximate the Euclidean projection function; e.g. fx(X) s Projy- (X). However, qualitatively, we find that this is not the case. Our mapping fo learns a less interpretable mapping from the canonical set X to the object O. In order to encourage the mapping to produce more interpretable mapping from the canonical set X to the object O can regularize the transform fe to penalize the 'distance traveled' of points transformed by fo- A regularization term with small coefficient (A 0.01) is effective in encouraging this behavior.
13

---

Published as a conference paper at ICLR 2020
Figure 5: Slices of the sphere where the input points are sampled from and their projections in the predicted point set. Slices of the input sphere are colored red; the locations where this subset of the inputs are mapped are green. Above: minimizing the Chamfer distance only. Below: Minimizing Chamfer distance with regularization (Equation 6 In both cases, the mapping is smooth, but only with regularization mapping close to the intuitive projection mapping.
Making this change results in little deviation in performance, while providing more coherent mapping. Figure this distinction.
This penalty for the mapping computed by fo, for each point in the sample X given as
R(for 8)
Σ (x,, X
-
(6)
X
where X is a sample from the canonical set X. We might instead directly penalize the difference between fa, and the Euclidean projection over the sampled set as:
1 Σ X
R(fe,
llfe, (x:) collo: 12
X.EX
However, we find that this regularization can be overly constraining, for example in cases where points are sampled near the boundaries the Voronoi tesellation of the target point cloud. The formulation in Equation gives the mapping greater flexibility while still giving the desired semantics.
A.5 FOLE PATH GENERATION
From Chamfer distance Fl scores alone, it is difficult to determine method's reconstruction quality is meaningfully different from another's. In this section, we introduce a new benchmark to evaluate the practical implications of using 3D reconstructions for collision-frre path generation. We compare the reconstructions of HOF with Lin et al. (018)), which is the most competitive direct decoding method according to Table This experiment intended give an additional perspective on what difference in average Chamfer distance to the ground truth object means. We show that given an RGB image, we can efficiently find near-optimal path P between two points around the bounding sphere of the object without colliding with it, and without taking a path much longer than the optimal path P* where the optimal path defined the shortest collision-fre frree path between two given points. complete definition of the experiment and its implementation are given in Section B.2.2
14

---

Published as a conference paper at ICLR 2020
view
view
view
Baselines
Optimal path
EPCG
HOF
All paths
Figure 6: Collision-fres path generation. The paths are color-coded as Blue: baseline showing L distance, Green: baseline going around a bounding box around the object, Yellow: with GT voxels as obstacles, Magenta: with EPCG voxels, Cyan: with HOF voxels. The rightmost figure shows all paths together viewed from three different view points. Best viewed in color.
We quantify the quality of our predictions by measuring both i) the proportion of predicted paths P that are collision-free and ii) the average ratio of the length of collision-free estimated path P and the corresponding optimal path P*
These metrics conceptually mirror the backward and forward Chamfer distance metrics, re- spectively; low collision rate corresponds to few missing structures in the reconstructed object (backward Chamfer, surface coverage), while successful paths close to the optimal path length correspond to few extraneous structures in the reconstruction (forward Chamfer, or shape similarity).
We find that HOF provides meaningful gains over the reconstruction method recently proposed in Lin et al. (2018) in the context of path planning around the reconstructed model. HOF performs significantly better both terms of path length as well as collision rate. However, although [in the reconstruction task with objects in canonical frame, in context of robotics, learning in a viewer-centric frame necessary. has been noted Wu et al. (2018) that generalization might be casier when learning reconstruction in viewer-oriented frame. We test this theory by training on both objects in their canonical frame as well as in the 'camera' frame. We rotate each point cloud Y into its camera frame orientation using the azimuth and elevation values for each image. We rotate the point cloud about the origin, keeping the bounding box centered at (0,0,0). Trained and tested in the viewer-centric camera frame, HOF performs better than in the canonical frame, giving Chamfer distance scores of 1.486 0.979 (compared with 1.534 1.046 for the canonical frame). The most notably improved classes in the viewer-centric evaluation are cabinets and loudspeakers; it is intuitive that these particularly 'boxy' objects might be better reconstructed in viewer-centric frame, their symmetric nature might make difficult identify their canonical frame from single image.
et al. (2018) results were reported
Results of this comparison, as well as other ablation studies, are reported Supplementary Tables [nd The path quality performances of the baseline metrics, EPCG Lin et al. (2018) and HOF are presented in Table
Table 6: Mean values for the collision- free path generation success rate and the optimality of the output paths. Higher values indicate better performance. Details about baseline metries Shortest L and Shortest Around Bounding Box (SABB) are listed in Supplementary Section B.2.2
Method
Shortest L1
SABB
EPCG
LLin et al. 2018)
Ours
Success rate 0.341 +: 0.35
1.0 + 0.0
0.775 + 0.19
0.989 + 0.06
Optimality
1.0 + 0.0
0.960 + 0.05
0.994 + 0.02
0.998 + 0.01
A.6 PARAMETER INTERPOLATION vs FUNCTION COMPOSITION
In order to illustrate the non-trivial mappings learned by HOF for > 1, we compare the reconstruc- tions acquired from interpolating naively between decoder parameters of two different objects and the reconstructions acquired by composing the two reconstruction functions. Results are shown in Figure 7
15

---

Published as a conference paper at ICLR 2020
Compedtin
Cerpostie
oéper
Oxject
Orper
Pranear
Tarrer
Compodton
Componbte
Olynn
roper R
Figure 7: Inter-class interpolation between two objects using function composition and parameter interpolation. We see that the composition-based interpolation preserves some geometric features such as chair legs, car roof, and airplane wings. Direct interpolation of the network parameters fails to meaningfully capture features from the parent objects. We use h 4, and (fB fo fA fx)(X) for the interpolations with composition.
A.7 FULL CLASS BREAKDOWN FOR LVC EXPERIMENT CHAMFER DISTANCE SCORES
Table shows class-wise Chamfer Distances for HOF and the baseline methods in the LVC experiment.
Table 7: Class-weighted asymmetric Chamfer distance results for our method compared to other recent methods for 3D reconstruction from images as reported in et al. (2018). We use the HOF-3 architecture with k = 1.
Category 3D-R2N2
PSG
EPCG
HOF (Ours)
Airplane
2.399 / 2.391
1.301 / 1.488
1.294 / 1.541
0.936 / 0.723
Bench
2.323 2.603
1.814/1.988
1.757
1.487
1.288 0.914
Cabinet
1.400
2.463 / 2.444
1.814 1.072
1.764 / 1.383
Car
1.664 3.146
1.800/2.053
1.446 1.061
1.367 / 0.810
Chair
1.080
1.887 /2.355
1.886/
1.670 / 1.147
Display
/2.953
2.142 1.440
1.765 1.130
Lamp
5.698/7.33)
2.347 /2.212
2.635 / 44.459
2.054 1.325
Loudspeaker
2.487 /4.203
3.215 /215/22785
2.371 1.771 1.706
2.126 / 1.398
Rifle
1.316/1.358
1.289 / 1.510
1.066 / 0.817
Sofa
2.306 1 3.196
2.592/2.788
1.917 1.423
1.666 / 1.064
Table
2.128 3.1134
1.874/2.225
1,689 / 1.620
1.377 0.979
Telephone
1.874/2.734
1.989
1.939 / 1.198
1.387 / 0.944
Watereraft
3.210/3.614
1.715 1.877
1.813 / 1.550
1.474 0.967
Mean
2.588/ 3.3442
1.982 / 2.146
1.846 / 1.701
1.534 1.046
A.8 FULL CLASS BREAKDOWN FOR BROAD EXPERIMENT F1 SCORES
Table shows clsss-wise performance of HOF as well as each method compared in the broad ShapeNet comparison reported by Tatarchenko al. (2019).
16

---

Published as a conference paper at ICLR 2020
B TRAINING/TESTING DATASET AND IMPLEMENTATION DETAILS
In the reconstruction experiment, Chamfer Distance scores are scaled up by 100 as in Lin et al. (2018) for easier comparison. For the numbers reported in Table 7
the best performance of 3d-r2n2 (5 views as reported in Lin et al. (2018)). ). In comparing with methods like FoldingNet Yang et al. 20188) and DeepSDF Park al. (2019) we focus on efficiency of representation rather than reconstruction quality. The performance comparison in Figure 3 and the ablation experiment in Tables B attempt to compare these architectures this way (FoldingNet slightly shallower version of the DeepSDF architecture; 6 rather than 8 fully-connected layers).
B. DATASET
We use two different datasets for evaluation. First, in Table I we use the ShapeNet train/validationn/ees splits of subset of the ShapeNet dataset (Chang al. (013) described in Yan al. (2016). The dataset can be downloaded from https: /tops://giii ht////ttubbbb thub. com/ xxyxnnnnnn /nyan/nii / ps1 6_P.N Point clouds have 100k points. Upon closer inspection, have found that this subset includes some inconsisten//noisy labels, including:
1. Inconsistency object interior filling (e.g. some objects are only surfaces, while some have densely sampled interions)
2. Objects with floating text annotations that are represented in the point cloud model
3. Objects that are inconsistently small (scaled down by a factor of 5 or more compared to other similar objects)
Although these types of inconsistencies are rare, they are noteworthy. We used them as-is, but future contributions might include both 'cleaned' and 'noisy' variants of this dataset. Learning from noisy labels is an important problem but is orthogonal to 3D reconstruction.
In our second experiment. we use broader dataset based on ShapeNet, with train and test splits taken from Tatarchenko et al. (2019). The dataset can be downloaded from what.d.
B.2 IMPLEMENTATION DETAILS
B.2.1 NETWORK ARCHITECTURE AND TRAINING
For the problem of 3D reconstruction from an RGB image, which address here, we represent go as a convolutional neural network based on the DenseNet architecture proposed in Huang et al. (2017). We call this our baseline encoder network. The baseline encoder network has 3 dense blocks (each containing 4 convolutional layers) followed by 3 fully connected layers. The schedule of feature maps is [16, 32, 64] for dense blocks. Each fully connected layer contains 1024 neurons.
We use two variants of the mapping architecture fa. One, which we call HOF-1, is MLP with hidden layer containing 1024 neurons. A second version, HOF-3, is an MLP with hidden layers, each containing 128 hidden units. Both formulations use the ReLU activation function Glorot et al. (0011). Because 9사 and fe are all differentiable almost everywhere, we can train the entire system end-to-end with backpropagation. We use the Adam Optimizer with learning rate 1e-5 and batch size 1, training for epochs for experiments (1 epoch 2e 725k parameter updates). Training HOF from scratch took roughly 36 hours.
B.2.2 PATH PLANNING EXPERIMENT
We use the class 'chair' from the dataset described in Section in our experiments. The objects from this class have considerable variation and complexity in shape, thus they are useful for evaluating the quality of the generated paths.
Path planning is performed in a three dimensional grid environment. All the objects in our dataset fit inside the unit cube. Given the predicted point cloud of an object, first voxelize the points by constructing an occupancy map n x n x n centered at the origin of the object with voxel size 2/n. Next, generate start and end points follows. We choose unit vector sampled uniformly
17

---

Published as a conference paper at ICLR 2020
random and compute d v We the end points of and -d the start and goal locations. For each method, we generate the paths with the A algorithm using the voxelization of the predicted point clouds obstacles, and the sampled start and goal positions. The movement is rectilinear the voxel space and the distances are measured with the 1 metric. In the experiments occupancy grid of size 32 32 32, and sample 100 start and goal location pairs per model.
In addition the paths generated using the predictions from the EPCG et (2018) and HOF methods, we present two other baselines (Figure The first baseline Shortest L1 outputs the shortest path with the L1 metric ignoring the obstacles the scene, and the second baseline Shortest Around Bounding Box (SABB) takes the bounding box of the ground truth voxels as the environment to generate the path.
Figure 8: 2D illustration of the baseline path generation methods. The dotted path in blue is produced by Shortest Li and the dashed path in green by SABB. In our experiments, the rectilincar shortest path as the output of Shortest L1.
We present the path generation results in Table 6 The baseline Shortest L1 gives the optimal solution when the path is collision-free. However, since most of the shortest paths go through the object, this baseline has a poor success rate performance. In contrast, SABB output paths are always collision-free the shortest path is computed using the bounding box of the true voxelization as the obstacles in the environment. The length of the paths generated by SABB are longer compared to the rest of the methods since the produced paths are 'cautious' not collide with the object. These two baselines are the best performers for the metric they are designed for, yet they suffer from the complementary metric. Our method on the other hand achieves almost optimal results both metrics due the good quality reconstructions.
B.2.3 COMPUTING ENVIRONMENT
All GPU experiments were performed on NVIDIA GTX 1080 Ti GPUs. The CPU running times computed one of 12 cores of Intel 7920X processor.
18

---

Published as a conference paper at ICLR 2020
Table 8: F-score evaluation (@1%) on the Tatarchenko et al. 2019
dataset
Category
AtlasNet
OGN
Matryoshka
Retrieval
Oracle NN
HOF (Ours)
airplane
0.39
0.26
0.33
0.37
0.45
0.313
ashcan
0.18
0.23
0.26
0.21
0.24
0.239
bag
0.16
0.14
0.18
0.13
0.15
0.192
basket
0.19
0.16
0.21
0.15
0.15
0.217
bathtub
0.25
0.13
0.26
0.22
0.26
0.256
bed
0.19
0.12
0.18
0.15
0.17
0.195
bench
0.34
0.09
0.32
0.3
0.34
0.359
birdhouse
0.17
0.13
0.18
0.15
0.15
0.175
bookshelf
0.24
0.18
0.25
0.2
0.2
0.274
bottle
0.34
0.54
0.45
0.46
0.55
0.497
bowl
0.22
0.18
0.24
0.2
0.25
0.267
bus
0.35
0.38
0.41
0.36
0.44
0.365
cabinet
0.25
0.29
0.33
0.23
0.27
0.311
camera
0.13
0.08
0.12
0.11
0.12
0.153
can
0.23
0.46
0.44
0.36
0.44
0.416
cap
0.18
0.02
0.15
0.19
0.25
0.174
car
0.3
0.37
0.38
0.33
0.39
0.323
cellular
0.34
0.45
0.47
0.41
0.5
0,415
chair
0.25
0.15
0.27
0.2
0.23
0,259
clock
0.24
0.21
0.25
0.22
0.27
0,264
dishwasher
0,2
0.29
0.31
0.22
0.26
0,393
display
0.22
0.15
0.23
0.19
0.24
0.26
earphone
0.14
0.07
0.11
0.11
0.13
0.23
faucet
0.19
0.06
0.13
0.14
0.2
0.167
file
0.22
0.33
0.36
0.24
0.25
0,21
guitar
0.45
0.35
0.36
0.41
0.58
0.329
helmet
0.1
0.06
0.09
0,08
0.12
0.515
jar
0.21
0.22
0.25
0.19
0.22
0.114
keyboard
0.36
0.25
0.37
0.35
0.49
0.263
knife
0.46
0.26
0.21
0.37
0.54
0,504
lamp
0.26
0.13
0.2
0.21
0.27
0.309
laptop
0.29
0.21
0.33
0.26
0.33
0.284
loudspeaker
0.2
0.26
0.27
0.19
0.23
0.252
mailbox
0.21
0,2
0.23
0.2
0.19
0.292
microphone
0.23
0.22
0.19
0.18
0.21
0.315
microwave
0.23
0.36
0.35
0.22
0.25
0.279
motorcycle
0.27
0.12
0.22
0.24
0.28
0.27
mug
0.13
0.11
0.15
0.11
0.17
0.128
piano
0.17
0.11
0.16
0.14
0.17
0.19
pillow
0.19
0.14
0.17
0.18
0.3
0.227
pistol
0.29
0.22
0.23
0.25
0.3
0.307
pot
0.19
0.15
0.19
0.14
0.16
0.2
printer
0.13
0.11
0.13
0.11
0.14
0.17
remote
0.3
0.33
0.31
0.31
0.37
0.395
rifle
0.43
0.28
0.3
0.36
0.48
0.47
rocket
0.34
0.2
0.23
0.26
0.32
0.369
skateboard
0.39
0.11
0.39
0.35
0.47
0,384
sofa
0.24
0.23
0.27
0.21
0.27
0.262
stove
0.2
0.19
0.24
0.18
0.19
0,247
table
0.31
0.24
0.34
0.26
0.34
0.325
telephone
0.33
0.42
0,45
0.4
0.5
0,408
tower
0.24
0.2
0.25
0.25
0.25
0.336
train
0.34
0.29
0.3
0.32
0.38
0.375
vessel
0.28
0.19
0.22
0.23
0.29
0.299
washer
0.2
0.31
0.31
0.21
0.25
0.268
19


Reviews:
[{"review_id": "Syee2fmjYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method for single image 3D reconstruction. It is inspired by implicit shape models, like presented in Park et al. and Mescheder et al., that given a latent code project 3D positions to signed distance, or occupancy values, respectively. However, instead of a latent vector, the proposed method directly outputs the network parameters of a second (mapping) network that displaces 3D points from a given canonical object, i.e., a unit sphere. As the second network maps 3D points to 3D points it is composable, which can be used to interpolate between different shapes. Evaluations are conducted on the standard ShapeNet dataset and the yields results close to the state-of-the-art, but using significantly less parameters.\n\nOverall, I am in favour of accepting this paper given some clarifications and improving the evaluations.\n\nThe core contribution of the paper is to estimate the network parameters conditioned on the input (i.e., the RGB image). As noted in the related work section this is not a completely new idea (cf. Schmidhuber, Ha et al.). There are a few more references that had similar ideas and might be worth adding: Brabandere et al. \"Dynamic Filter Networks\", Klein et al. \"A dynamic convolutional layer for short range weather prediction\", Riegler et al. \"Conditioned regression models for non-blind single image super-resolution\", and maybe newer works along the line of Su et al. \"Pixel-Adaptive Convolutional Neural Networks\".\n\nThe input 3D points are sampled from a unit sphere. Does this imply any topological constraints? Is this the most suitable shape to sample from? How do you draw samples from the sphere (Similarly, how are the points sampled for the training objects)? What happens if you instead densely sample from a 3D box (similar to the implicit shape models)?\n\nOn page 4 the mapping network is described as a function that maps c-dimensional points to 3D points. What is c? Isn't it always 3, or how else is it possible to composite the mapping network?\n\nRegarding the main evaluation: The paper follows the \"standard\" protocol on ShapeNet. Recently, Tatarchenko et al. showed in \"What Do Single-view 3D Reconstruction Networks Learn?\" shortcomings of this evaluation scheme and proposed alternatives. It would be great if this paper could follow those recommendations to get better insights in the results.\nFurther, I could not find what k was set to in the evaluation of Tab. 1. It did also not match any numbers in Tab. 4 of the appendix. Tab. 4 shows to some extend the influence of k, but I would like to see a more extensive evaluation. How does performance change for larger k, and what happens if k is larger at testing then on at training, etc.?\n\nThings to improve the paper that did not impact the score:\n- The tables will look a lot nicer if booktab is used in LaTeX\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SklBkB-f9r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work is focused on learning 3D object representations (decoders) that can be computed more efficiently than existing methods.  The computational inefficiency of these methods is that you learn a (big) fixed decoder for all objects (all z latents), and then need to apply it individually on either each point cloud point you want to produce, or each voxel in the output (this problem exists for both the class of methods that deform a uniform distribution R^3 -> R^3 a la FoldingNet, or directly predict the 3D function R^3 -> R e.g. DeepSDF). The authors propose that the encoder directly predict the weights and biases of a decoder network that, since it is specific to the particular object being reconstructed, can be much smaller and thus much cheaper to compute.\n\nThe authors then note the fact that their method lacks a continuous latent space that allows for interpolation, as provided by existing (VAE-like) methods. They propose to solve this by learning an MLP that produces the output by recurrent application, and then composing subapplications of different networks as a type of interpolation.\n\n-------------------\n\nI like this work, it addresses a real problem in a number of models for 3D representation learning (similar models are also used for e.g. cryo-EM reconstruction). While the fast weights approach is not totally original, its application to this problem is novel and very well-suited to it. I was a bit surprised by just how much the decoder network could be shrunk by using fast weights. \n\nThe paper is also quite well written. I especially like how Section 2 synthesizes existing work into model categories which make it easier to think about their relationships. I also think the explanation in Sec. 3.2, while kind of obvious, is a nice way think about decoder vs. fast weights.\n\nI like that the authors are straightforward about the deficiency of the method (i.e. that you can't interpolate in latent space). Their proposed solution of functional composition is exceedingly clever but in my opinion too impractical to really be useful. It adds extra complexity, requires you to do function composition which may be less expressive and takes more coomputation, etc. And to what end? The purpose of generative models is not to interpolate per se; the interpolation is really a sanity check that the model is capturing the underlying distribution rather than just memorizing training examples. The function composition doesn't capture that. I think the authors should just acknowledge that you can't soundly *sample* from their generative model the way e.g. VAE or GAN allows (their function composition is not a sampling method). But I think there are lots of useful things you can do without that capability, e.g. do 3D point cloud completion, go image -> structure, etc. I think this function composition angle should be deemphasized in the title/abstract, but I think the paper stands  reasonably on its own without that.\n\nNits:\n- In Figure 2 it's pretty hard to see the differences between the methods. What exactly is being visualized here? DeepSDF shold be visualizing surface normals vs. HOF which is point clouds, right?\n- For predicting a deformation R^3 -> R^3 function composition sort of makes sense, but how generalizable is this approach e.g. to directly predicting a function R^3 -> R (a la DeepSDF)? I think there are ways this function composition approach could generalize, e.g. using skip connections and layer dropout (which encourages layers to be composable).\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HyxflLtAYB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper describes a contextual encoding scheme for reconstruction of 3D pointclouds from 2D images. An encoder outputs the parameters of a hierarchy of reconstruction networks that can be applied in succession to map random samples on a unit sphere to the surface of the reconstructed shape. \n\nStrengths:\nThe author's model was quite novel in my opinion. Deep 2D->3D is becoming a crowded space and there are many other models that encode image inputs, and many others that perform recursive or composition-based decoding. However, the particular link here was interesting, and I appreciate the small number of parameters resulting in solid reconstruction performance. While most related work was covered well, I believe the authors could have a more up-to-date list of recent work that reconstructs triangle-mesh representations from images [A-C] (especially since several of these methods has an architecture that involves encoding and subsequent compositional refinement). \n\nSome of the reconstructions shown in this paper are quite impressive, and the quantitative results show outperforming 2 recent methods. I did appreciate also the novel path-based evaluation of shape accuracy in the Appendix, although it would have been helpful to see more discussion of this in the main paper. \n\nAreas for improvement:\nI found that the core technical description was quite brief and would have benefited from simply more detail and space. You have argued that your method is sensible to try (cog. sci motivations), and shown that one instance works, but what can we expect in a more mathematical or general sense? Can any sizes of encoder and mapping network fit together? How does the number of mapping layers effect performance? Won't we eventually expect vanishing/exploding gradients with particular activation and can one address this in some way? \n\nI note that recent papers in this field tend to perform significantly more extensive experimental evaluation, typically selecting a wider range of competitors and using a number of more standardized metrics including IOU, F1 score and CD and typically repeating these at a variety of resolutions or on additional datasets or category splits etc. \n\nDecision: \nWeak reject because the idea is quite interesting, but I believe a more thorough explanation and expanded experimental comparison would be of great help to ensure the community can appreciate this work. \n\nAdditional citations suggested: \n\n[A] Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images. Wang, Zhang, Li, Fu, Liu and Jiang. ECCV 2018. \n[B] MeshCNN: A Network with an Edge. Hanocka, Hertz, Fish, Giryes, Fleishman and Cohen-Or. SIGGRAPH 2019.  \n[C] GEOMetrics: Exploiting Structure for Graph-Encoded Objects. Smith, Fujimoto, Romero and Meger. ICML 2019. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission presents an approach to single-view 3D reconstruction. The approach is quite creative and involves predicting the weights of a network that is then applied to a point set. The presentation is good. The experimental protocol is well-informed and the results are convincing. The reviewers' concerns have largely been addressed by the authors' responses and the revision. In particular, R2, who gave a "3", posted "I would now advise to raise my score (3 previously) to a be in line with the 6: Weak Accept given by the other reviewers." This means that all three reviewers recommend accepting the paper. The AC agrees.
Decision: Accept (Poster)