Higher-Order Function Networks for Learning Composable 3D Object Representations

We present a new approach to 3D object representation where a neural network encodes the geometry of an object directly into the weights and biases of a second ‘ mapping ’ network . This mapping network can be used to reconstruct an object by applying its encoded transformation to points randomly sampled from a simple geometric space , such as the unit sphere . We study the effectiveness of our method through various experiments on subsets of the ShapeNet dataset . We find that the proposed approach can reconstruct encoded objects with accuracy equal to or exceeding state-of-the-art methods with orders of magnitude fewer parameters . Our smallest mapping network has only about 7000 parameters and shows reconstruction quality on par with state-of-the-art object decoder architectures with millions of parameters . Further experiments on feature mixing through the composition of learned functions show that the encoding captures a meaningful subspace of objects . ‡ 1 INTRODUCTION . This paper is primarily concerned with the problem of learning compact 3D object representations and estimating them from images . If we consider an object to be a continuous surface in R3 , it is not straightforward to directly represent this infinite set of points in memory . In working around this problem , many learning-based approaches to 3D object representation suffer from problems related to memory usage , computational burden , or sampling efficiency . Nonetheless , neural networks with tens of millions of parameters have proven effective tools for learning expressive representations of geometric data . In this work , we show that object geometries can be encoded into neural networks with thousands , rather than millions , of parameters with little or no loss in reconstruction quality . To this end , we propose an object representation that encodes an object as a function that maps points from a canonical space , such as the unit sphere , to the set of points defining the object . In this work , the function is approximated with a small multilayer perceptron . The parameters of this function are estimated by a ‘ higher order ’ encoder network , thus motivating the name for our method : Higher-Order Function networks ( HOF ) . This procedure is shown in Figure 1 . There are two key ideas that distinguish HOF from prior work in 3D object representation learning : fast-weights object encoding and interpolation through function composition . ( 1 ) Fast-weights object encoding : ‘ Fast-weights ’ in this context generally refers to methods that use network weights and biases that are not fixed ; at least some of these parameters are estimated on a per-sample basis . Our fast-weights approach stands in contrast to existing methods which encode objects as vector-valued inputs to a decoder network with fixed weights . Empirically , we find that our approach enables a dramatic reduction ( two orders of magnitude ) in the size of the mapping network compared to the decoder networks employed by other methods . ( 2 ) Interpolation through function composition : Our functional formulation allows for interpolation between inputs by composing the roots of our reconstruction functions . We demonstrate that the 1Stanford University 2Samsung AI Center - New York 3University of Minnesota †Work performed while an intern at Samsung AI Center - New York . ‡ See https : //saic-ny.github.io/hof for code and additional information . Correspondence to : Eric Mitchell < eric.mitchell @ cs.stanford.edu > . functional representation learned by HOF provides a rich latent space in which we can ‘ interpolate ’ between objects , producing new , coherent objects sharing properties of the ‘ parent ’ objects . In order to position HOF among other methods for 3D reconstruction , we first define a taxonomy of existing work and show that HOF provides a generalization of current best-performing methods . Afterwards , we demonstrate the effectiveness of HOF on the task of 3D reconstruction from an RGB image using a subset of the ShapeNet dataset ( Chang et al. , 2015 ) . The results , reported in Tables 1 and 2 and Figure 2 , show state-of-the-art reconstruction quality using orders of magnitude fewer parameters than other methods . 2 RELATED WORK . The selection of object representation is a crucial design choice for methods addressing 3D reconstruction . Voxel-based approaches ( Choy et al. , 2016 ; Häne et al. , 2017 ) typically use a uniform discretization of R3 in order to extend highly successful convolutional neural network ( CNN ) based approaches to three dimensions . However , the inherent sparsity of surfaces in 3D space make voxelization inefficient in terms of both memory and computation time . Partition-based approaches such as octrees ( Tatarchenko et al. , 2017 ; Riegler et al. , 2017 ) address the space efficiency shortcomings of voxelization , but they are tedious to implement and more computationally demanding to query . Graph-based models such as meshes ( Wang et al. , 2018 ; Gkioxari et al. , 2019 ; Smith et al. , 2019 ; Hanocka et al. , 2019 ) provide a compact representation for capturing topology and surface level information , however their irregular structure makes them harder to learn . Point set representations , discrete ( and typically finite ) subsets of the continuous geometric object , have also gained popularity due to the fact that they retain the simplicity of voxel based methods while eliminating their storage and computational burden ( Qi et al. , 2017a ; Fan et al. , 2017 ; Qi et al. , 2017b ; Yang et al. , 2018 ; Park et al. , 2019 ) . The PointNet architecture ( Qi et al. , 2017a ; b ) was an architectural milestone that made manipulating point sets with deep learning methods a competitive alternative to earlier approaches ; however , PointNet is concerned with processing , rather than generating , point clouds . Further , while point clouds are more flexible than voxels in terms of information density , it is still not obvious how to adapt them to the task of producing arbitrary- or varied-resolution predictions . Independently regressing each point in the point set requires additional parameters for each additional point ( Fan et al. , 2017 ; Achlioptas et al. , 2018 ) , which is an undesirable property if the goal is high-resolution point clouds . Many current approaches to representation and reconstruction follow an encoder-decoder paradigm , where the encoder and decoder both have learned weights that are fixed at the end of training . An image or set of 3D points is encoded as a latent vector ‘ codeword ’ either with a learned encoder as in Yang et al . ( 2018 ) ; Lin et al . ( 2018 ) ; Yan et al . ( 2016 ) or by direct optimization of the latent vector itself with respect to a reconstruction-based objective function as in Park et al . ( 2019 ) . Afterwards , the latent code is decoded by a learned decoder into a reconstruction of the desired object by one of two methods , which we call direct decoding and contextual mapping . Direct decoding methods directly map the latent code into a fixed set of points ( Choy et al. , 2016 ; Fan et al. , 2017 ; Lin et al. , 2018 ; Michalkiewicz et al. , 2019 ) ; contextual mapping methods map the latent code into a function that can be sampled or otherwise manipulated to acquire a reconstruction ( Yang et al. , 2018 ; Park et al. , 2019 ; Michalkiewicz et al. , 2019 ; Mescheder et al. , 2019 ) . Direct decoding methods generally suffer from the limitation that their predictions are of fixed resolution ; they can not be sampled more or less precisely . With contextual mapping methods , it is possible in principle to sample the object to arbitrarily high resolution with the correct decoder function . However , sampling can provide a significant computational burden for some contextual mapping approaches as those proposed by Park et al . ( 2019 ) and Michalkiewicz et al . ( 2019 ) . Another hurdle is the need for post-processing such as applying the Marching Cubes algorithm developed by Lorensen and Cline ( 1987 ) . We call contextual mapping approaches that encode context by concatenating a duplicate of a latent context vector with each input latent vector concatenation ( LVC ) methods . In particular , we compare with LVC architectures used in FoldingNet ( Yang et al. , 2018 ) and DeepSDF ( Park et al. , 2019 ) . HOF is a contextual mapping method that distinguishes itself from other methods within this class through its approach to representing the mapping function : HOF uses one neural network to estimate the weights of another . Conceptually related methods have been previously studied under nomenclature such as the ‘ fast-weight ’ paradigm ( Schmidhuber , 1992 ; De Brabandere et al. , 2016 ; Klein et al. , 2015 ; Riegler et al. , 2015 ) and more recently ‘ hypernetworks ’ ( Ha et al. , 2016 ) . However , the work by Schmidhuber ( 1992 ) deals with encoding memories in sequence learning tasks . Ha et al . ( 2016 ) suggest that estimating weights of one network with another might lead to improvements in parameter-efficiency . However , this work does not leverage the key insight of using network parameters that are estimated per sample in vision tasks . 3 HIGHER-ORDER FUNCTION NETWORKS . HOF is motivated by the independent observations by both Yang et al . ( 2018 ) and Park et al . ( 2019 ) that LVC methods do not perform competitively when the context vector is injected by simply concatenating it with each input . In both works , the LVC methods proposed required architectural workarounds to produce sufficient performance on reconstruction tasks , including injecting the latent code multiple times at various layers in the network . HOF does not suffer from these shortcomings due to its richer context encoding ( the entire mapping network encodes context ) in comparison with LVC . We compare the HOF and LVC regimes more precisely in Section 3.2 . Quantitative comparisons of HOF with existing methods can be found in Table 1 . 3.1 A FAST-WEIGHTS APPROACH TO 3D OBJECT REPRESENTATION AND RECONSTRUCTION . We consider the task of reconstructing an object point cloud O from an image . We start by training a neural network gφ with parameters φ ( Figure 1 , top-left ) to output the parameters θ of a mapping function fθ , which reconstructs the object when applied to a set of points X sampled uniformly from a canonical set such as the unit sphere ( Figure 1 , top-right ) . We note that the number of samples in X can be increased or decreased to produce higher or lower resolution reconstructions without changing the network architecture or retraining , in contrast with direct decoding methods and some contextual mapping methods which use fixed , non-random samples from X ( Yang et al. , 2018 ) . The input to gφ is an RGB image I ; our implementation takes 64× 64× 3 RGB images as input , but our method is general to any input representation for which a corresponding differentiable encoder network can be constructed to estimate θ ( e.g . PointNet ( Qi et al. , 2017a ) for point cloud completion ) . Given I , we compute the parameters of the mapping network θI as θI = gφ ( I ) ( 1 ) That is , the encoder gφ : R3×64×64 → Rd directly regresses the d-dimensional parameters θI of the mapping network fθI : Rc → R3 , which maps c-dimensional points in the canonical space X to points in the reconstruction Ô ( see Figure 1 ) . We then transform our canonical space X with fθI in the same manner as other contextual mapping methods : Ô = { fθI ( xi ) : xi ∈ X } ( 2 ) During training , we sample an image I and the corresponding ground truth point cloud model O , where O contains 10,000 points sampled from the surface of the true object . We then obtain the mapping fθI = gφ ( I ) and produce an estimated reconstruction of O as in Equation 2 . In our training , we only compute fθI ( x ) for a sample of 1000 points in X . However , we find that sampling many more points ( 10-100× as many ) at test time still yields high-quality reconstructions . This sample is drawn from a uniform distribution over the set X . We then compute a loss for the prediction Ô using a differentiable set similarity metric such as Chamfer distance or Earth Mover ’ s Distance . We focus on the Chamfer distance as both a training objective and metric for assessing reconstruction quality . The asymmetric Chamfer distance CD ( X , Y ) is often used for quantifying the similarity of two point sets X and Y and is given as CD ( X , Y ) = 1 |X| ∑ xi∈X min yi∈Y ||xi − yi||22 ( 3 ) The Chamfer distance is defined even if sets X and Y have different cardinality . We train gφ to minimize the symmetric objective function ` ( Ô , O ) = CD ( Ô , O ) + CD ( O , Ô ) as in Fan et al . ( 2017 ) .

Summary: This work is focused on learning 3D object representations (decoders) that can be computed more efficiently than existing methods.  The computational inefficiency of these methods is that you learn a (big) fixed decoder for all objects (all z latents), and then need to apply it individually on either each point cloud point you want to produce, or each voxel in the output (this problem exists for both the class of methods that deform a uniform distribution R^3 -> R^3 a la FoldingNet, or directly predict the 3D function R^3 -> R e.g. DeepSDF). The authors propose that the encoder directly predict the weights and biases of a decoder network that, since it is specific to the particular object being reconstructed, can be much smaller and thus much cheaper to compute.