Bayesian Meta-Learning for Few-Shot 3D Shape Completion 

1 INTRODUCTION . The task of estimating 3D geometry from sparse observations , commonly referred to as shape completion , is a key perceptual challenge and an integral part of many mission-critical problems , including robotics ( Varley et al. , 2017 ) and autonomous driving ( Giancola et al. , 2019 ; Stutz & Geiger , 2018 ) . Recently , a series of methods ( Mescheder et al. , 2019 ; Park et al. , 2019 ) have achieved great success by using the observations to infer the parameters of an implicit 3D geometric representation of the targets object . However , with some notable exceptions ( Yuan et al. , 2018 ) , such methods require relatively dense observations to achieve high accuracy , which is usually impractical in real situations . In this paper we introduce a novel methodology that enables state-of-the-art shape completion of previously unseen objects from highly sparse observations . Our insight comes from the following simple intuition : `` Can we leverage the geometric information available in one object to improve shape completion results on another target object ? '' Meta-learning is an emerging field of study in machine learning that serves this very purpose . By training a model on multiple inter-related tasks , it learns how to learn new tasks efficiently from a small amount of observations . Recently proposed meta-learning methods often achieve this by parameterizing the input-output relationship with task-specific latent variables and training a separate , task-agnostic model/mechanism that can infer these task-specific variables from sparse observations of the target task ( Chang et al. , 2015 ; Finn et al. , 2017 ; Garnelo et al. , 2018 ) . We can cast the shape completion problem as a Bayesian meta-learning problem by treating each object as a task and its sparse observations as the corresponding contextual dataset . In popular Bayesian variants of meta-learning ( Edwards & Storkey , 2017 ; Eslami et al. , 2018 ; Garnelo et al. , 2018 ) , the task-specific latent variables are treated as random variables , and the aforementioned task-agnostic model ( i.e . the encoder ) is represented as a posterior distribution of the latent variables conditioned on sparse observations . In this study , we combine probabilistic meta-learning with recent shape completion methods that represent the geometry of a given object with implicit parameters , such as the parameters of a signed distance function ( SDF ) . By training an encoder that computes the posterior distribution of these implicit parameters conditioned against sparse observations , we develop a framework that enables the few-shot learning of implicit geometric functions . Under appropriate regularity conditions , the computation of correct posterior distribution leads to optimal prediction in the sense of Bayes Risk ( Maeda et al. , 2020 ) . Our proposed approach is a natural extension of many implicit approaches , in the sense that it introduces an additional encoder function that represents the posterior distribution of the geometry-describing implicit parameters . More specifically , we build upon the Bayesian approach of ( Maeda et al. , 2020 ) whose posterior estimate behaves asymptotically well with respect to the size of contextual dataset , and combine their method with Implicit Geometrical Regularization ( IGR ) ( Gropp et al. , 2020 ) . We use IGR as the baseline , and demonstrate its efficacy on two benchmark datasets ( ShapeNet and ICL-NUIM ) , especially when the observations are very sparse . 2 RELATED WORK . 2.1 3D DECODING REPRESENTATIONS . Unlike images , that contain a clear pixel-based structural pattern , there is no unified representation for 3D object reconstruction that is both computationally and memory efficient . In terms of used 3D representations , existing methods can be broadly divided into the following categories : Voxel-based methods are a generalization of 2D pixels into 3D space , and thus constitute a natural extension for classical image-based methods . Early works focused on 3D convolutions operating on dense grids ( Choy et al. , 2016 ) to generate an occupancy function that determines whether each cell is inside an object or not , however these were limited to relatively small resolutions . To address the high memory requirements of dense voxel grids , various works have proposed 3D reconstruction in a multi-resolution fashion ( Häne et al. , 2017 ) , with the added complexity of requiring multiple passes to generate the final output . More recently , OccNet ( Mescheder et al. , 2019 ) proposes encoding a 3D description of the output at infinite resolution , and shows that this representation can be learned from different sensor modalities . Signed Distance methods are an alternative to occupancy functions , where instead of the occupancy state we learn a function describing the signed distance to the object surface ( Dai et al. , 2017 ; Stutz & Geiger , 2018 ) . This approach builds upon earlier fusion methods that utilize a truncated signed distance function ( SDF ) introduced in ( Curless & Levoy , 1996 ) . DeepSDF ( Park et al. , 2019 ) represents 3D space as a continuous volumetric field , and requires at training time the ground-truth SDF calculated from dense input data using numerical methods . Implicit Geometric Reconstruction ( IGR ) ( Gropp et al. , 2020 ) is a SDF variant that uses Eikonal regularization , thus enforcing that the output of the decoder will be the SDF of `` some '' surface . This is an effective way of mitigating the impact of outliers in the final generated surface , and is used as the starting point for our proposed meta-learning approach to shape completion . Point-based methods directly output points located on the object surface , thus eliminating the need for a dense representation of the 3D space . Earlier works such as PointNet ( Charles et al. , 2017 ; Qi et al. , 2017 ) combined fully connected networks with a symmetric aggregation function , thus achieving permutation invariance and robustness to perturbations . ( Fan et al. , 2017b ) introduced point clouds as a viable output representation for 3D reconstruction , and ( Yang et al. , 2017 ) proposed a decoder design that approximates a 3D surface as the deformation of a 2D plane . Point Completion Network ( PCN ) ( Yuan et al. , 2018 ) is a recent architecture that enables the generation of coarse-tofine shapes while maintaining a small number of parameters . However , a common limitation of all these methods is that they do not describe topology , and thus are not suitable for the generation of watertight surfaces . Also , to change the number of output points , methods like PCN have to re-train their networks entirely , while SDF-based methods learn the geometry in an implicit form and thus can generate any amount of points . Mesh-based methods choose to represent classes of similarly shaped objects in terms of a predetermined set of template meshes . First attempts focused on graph convolutions alongside the mesh ’ s vertices and edges ( Guo et al. , 2015 ) , and more recently as a direct output representation for 3D reconstruction ( Kanazawa et al. , 2018 ) . These methods , however , are only able to generate meshes with simple topologies ( Wang et al. , 2018 ) , require a reference template from the same object class ( Kanazawa et al. , 2018 ) and can not guarantee water-tight closed surfaces ( Groueix et al. , 2018 ) . A learnable extension to the Marching Cubes algorithm ( Lorensen & Cline , 1987 ) has been proposed in ( Liao et al. , 2018 ) , however this approach is limited by the memory requirements of the underlying 3D voxel grid . 2.2 ENCODING AND DECODING MECHANISM . In shape completion methods , one must construct a geometry of object from point clouds of varying size that are scattered over various regions . Thus , all methods use some mechanism to encode the sparse point clouds to a tensor of fixed size , and decode this tensor to produce the final output . Existing methods also differ by the design of these decoding and encoding mechanisms . Methods like DeepSDF and IGR train an auto-decoder , and do not train separate encoder functions at all . To find the geometry-describing implicit parameters for each object , these methods apply likelihood-based gradient descent on randomly initialized latent variables , using just sparse observations from the target object . Thus , these methods by design do not use observations from multiple objects to train an object-agnostic mechanism that can efficiently learn the latent variables . Meanwhile , PCN and OccNet both train a complex encoder function that maps sparse observations to latent variables . Although their decoders differ entirely ( PCN directly outputs 3D points , while OccNet outputs binary values ) , they both use an encoder that aims to capture the hierarchical structure of the object ’ s geometry . More specifically , PCN ’ s encoder is equipped with a mechanism to extract information from sparse observations in two steps , one aimed at extracting global information and another at extracting local information . On the other hand , OccNet ’ s encoder is a version of PointNet ( Charles et al. , 2017 ) that uses max pooling with respect to the sparse observation set . Our method also uses an encoder function , however it differs from those mentioned above in the sense that it represents a posterior distribution , rather than a deterministic function . Under an appropriate set of regularity conditions , inference made from the predictions of posterior distribution is optimal in terms of the Bayes risk ( Maeda et al. , 2020 ) . Furthermore , because our encoder is probabilistic , it is capable of outputting multiple candidate shapes for a given sparse observation . 3 META-SHAPE COMPLETION . 3.1 PROBLEM SETTING . Let Dk = { x ( k ) n ∈ R3|n = 1 , · · · , Nk } be an arbitrary set of points located on the surface of a 3D object k. The goal of shape completion for object k is to use Dk to infer its surface . There are various approaches to this problem : if the surface is closed , one can use an occupancy function Focc ( x ) that tells whether a point x is inside or outside the surface . One may also use a signed distance function Fsdf ( x ) that evaluates how far point x is to the surface , with positive values indicating that it ’ s outside and negative values indicating that it ’ s side . In both approaches , the transcribed goal is to use Dk to find the object-k-specific function Fk that best describes its geometry . Our proposed method focuses on the latter approach , and in the following sections we show how Bayesian meta-learning can be used to learn a probabilistic SDF for shape completion . 3.2 META-LEARNING FOR SHAPE COMPLETION . Meta-learning exploits underlying similarities among tasks to enable the transfer of knowledge between tasks , so that information gained from solving one problem can improve performance on another . Traditional meta-learning methods work by parameterizing their models with two types of parameters : task-specific and task-agnostic . Following this basic philosophy , we represent the signed distance function ( SDF ) of all observed objects using object-specific parameters hk and object-agnostic parameters θ . More specifically , we write our SDF for object k as SDFθ ( x ; hk ) , and determine hk using the θ-parameterized function that encodes Dk . The function that maps Dk to hk is often referred to as encoder in the meta-learning literature . The SDF , on the other hand , plays the role of decoder . In a probabilistic setting , we consider a probabilistic SDF of form N ( s|SDFθ ( x ; hk ) , σ2θ ) . Now , if we interpret this estimation problem as a case of minimizing the Bayes risk ( Maeda et al. , 2020 ) , we can show that the optimal solution is given by the predictive distribution pθ ( s|x ; Dk ) computed using the posterior distribution p ( h|Dk ) : p ( s|x , Dk ) = ∫ p ( s|x , hk ) p ( hk|Dk ) dhk . ( 1 ) The estimation of the posterior p ( hk|Dk ) is a challenging task : for the approximation to be valid , it must be able to accept unordered sets Dk of various lengths , while satisfying all other properties of a posterior distribution . For example , its variance must approach zero as we take the number of observed points to infinity . According to the theory of ( Maeda et al. , 2020 ) , we can construct an approximation that satisfies all these requirements using a Gaussian distribution pθ ( hk|Dk ) =∏d i=1N ( hk , i|µi , θ ( Dk ) , σ2i , θ ( Dk ) ) with mean and variance as follows ( we use hk , i to denote the i-th element of hk ) : µi , θ ( Dk ) = σ 2 i , θ ( Dk ) Nk∑ n=1 fi , n g2i , n , σ2i , θ ( Dk ) = ( Nk∑ n=1 ( 1 g2i , n − 1 g2i,0 ) + 1 g2i,0 ) −1 . ( 2 ) In the above , fi , n = fi , θ ( x ( k ) n , y ( k ) n ) and gi , n = gi , θ ( x ( k ) n , y ( k ) n ) are neural networks parameterized by θ . When making a probabilistic inference , we sample from this Gaussian posterior approximation and feed it to the decoder SDF . A schematic diagram of our proposed meta-learning shape completion method can be found in Figure 1 . The training proceeds by making predictions about target points using contextual information from various datasets . To prepare the mock-target points at training time , we decompose Dk into Dctxtk and Dtargk for each task k and use the following classic ELBO ( Ranganath et al. , 2014 ; Kingma & Welling , 2014 ) to optimize the predictive distribution given Dk : Lk ( θ ) : =− ∫ pθ ( hk|Dk ) ( Nk∑ n=1 log pθ ( s ( k ) n |x ( k ) n , hk ) + log pθ ( hk|Dctxtk ) ) dhk −H ( pθ ( hk|Dk ) ) . ( 3 ) Here s ( k ) n is the true signed distance from point x ( k ) n to the surface of object k. In our problem setting , s ( k ) n is 0 for all ( n , k ) because all the observed points are on the surface of the objects . Figure 1 contains a diaram of our proposed algorithm . To test the sheer efficacy of our meta-approach , we only use MLP for encoder in this study , and do not explore the specific architecture that is suited for the shape-completion problem .

Summary: This paper introduces a meta-learning approach for the neural implicit representation of 3D shapes. The main idea, in my understanding, is to consider the points in the input point cloud as few-shot examples of the object so that each of them can be encoded in a way to best approximate the entire object information. The experiments show that the network can reconstruct the entire shape well even with a very small number of the input points, such as 50 and 100. For better reconstruction, the authors also proposed to use some implicit function regularizations, which are introduced in a previous work (Gropp et al., 2020).