Bayesian Meta-Learning for Few-Shot 3D Shape Completion 
Authors: Masanori Koyama
Toshiki Nakanishi
Shin-ichi Maeda
Vitor Campagnolo Guizilini
Adrien Gaidon
Venue:  (2021)
Keywords: shape completion
Meta-learning
Few-shot
3D reconstruction

Abstract: Estimating the 3D shape of real-world objects is a key perceptual challenge. It requires going from partial observations, which are often too sparse and incomprehensible for the human eye, to detailed shape representations that vary significantly across categories and instances. We propose to cast shape completion as a Bayesian meta-learning problem to facilitate the transfer of knowledge learned from observing one object into estimating the shape of another object. To combine the Bayesian framework with an approach that uses implicit 3D object representation, we introduce an encoder that describes the posterior distribution of a latent representation conditioned on sparse point clouds. With its ability to isolate object-specific properties from object-agnostic properties, our meta-learning algorithm enables accurate shape completion of newly-encountered objects from sparse observations. We demonstrate the efficacy of our proposed method with experimental results on the standard ShapeNet and ICL-NUIM benchmarks. 

Under review as a conference paper at ICLR 2021
BAYESIAN META-LEARNING FOR FEW-SHOT 3D SHAPE COMPLETION
Anonymous authors Paper under double-blind review
ABSTRACT
Estimating the 3D shape of real-world objects is a key perceptual challenge. It requires going from partial observations, which are often too sparse and incompre- hensible for the human eye, to detailed shape representations that vary significantly across categories and instances. We propose to cast shape completion as a Bayesian meta-learning problem to facilitate the transfer of knowledge learned from ob- serving one object into estimating the shape of another object. To facilitate the learning of object shapes from sparse point clouds, we introduce an encoder that de- scribes the posterior distribution of a latent representation conditioned on the sparse cloud. With its ability isolate object-specific properties from abject-agnostic properties, our meta-learning algorithm enables accurate shape completion of newly-encountered objects from sparse observations. We demonstrate the efficacy of our proposed method with experimental results on the standard ShapeNer and ICL-NUIM benchmarks.
INTRODUCTION
The task of estimating 3D geometry from sparse observations, commonly referred to shape completion, is a key perceptual challenge and an integral part of many missionncciticall problems. including robotics (Varley et al., 2017) and autonomous driving (Giancola et al. 2019 Stutz & Geiger 2018). Recently, a series of methods (Mescheder et al. 2019 Park et al.) [019 have achieved great success by using the observations to infer the parameters of an implicit 3D geometric representation of the targets object. However, with some notable exceptions Yuan et al., 2018), such methods require relatively dense observations to achieve high accuracy, which is usually impractical in real situations. In this paper we introduce a novel methodology that enables state-of-the-an shape completion of previously unseen objects from highly sparse observations.
Our insight comes from the following simple intuition: "Can we leverage the geometric information available in one object to improve shape completion results on another target object?" Meta-learning an emerging field of study in machine learning that serves this very purpose. By training model on multiple inter-related tasks, it learns how to learn new tasks efficiently from a small amount of observations. Recently proposed meta-learning methods often achieve this by parameterizing the input-output relationship with task-specific latent variables and training a separate, task-agnostic model/mechanism that can infer these task-specific variables from sparse observations of the target task (Chang et al. 20155 Finn et al. [2017] Garnelo et al. 2018). We can cast the shape completion problem as Bayesian meta-learning problem by treating each object as task and its sparse observations as the corresponding contextual dataset.
In popular Bayesian variants of meta-learning (Edwards& /torkey| 2017 Eslami et al., 2018 Garelo et al.) 2018). the task-specific latent variables are treated as random variables, and the aforementioned task task-agnostic model (i.e. the encoder) represented as a posterior distribution of the latent variables conditioned on sparse observations. In this study, we combine probabilistic meta-learning with recent shape completion methods that represent the geometry of a given object with implicit parameters, such as the parameters of a signed distance function (SDF). By training an encoder that computes the posterior distribution of these implicit parameters conditioned against sparse observations, we develop framework that enables the few-shot learning of implicit geometric functions. Under appropriate regularity conditions, the computation of correct posterior distribution leads to optimal prediction in the sense of Bayes Risk (Maeda et al. 2020 Our proposed approach is natural extension of many

---

Under review as a conference paper at ICLR 2021
implicit approaches, in the sense that it introduces additional encoder function that represents the posterior distribution of the
implicit parameters.
More specifically, we build upon the Bayesian approach of (Maeda et al. 2020) whose posterior estimate behaves asymptotically well with respect to the size of contextual dataset, and combine their method with Implicit Geometrical Regularization (IGR) (Gropp et al. 2020). We use IGR as the baseline, and demonstrate its efficacy on two benchmark datasets (ShapeNet and ICL-NUIM), especially when the observations are very sparse.
2 RELATED WORK
2.1 3D DECODING REPRESENTATIONS
Unlike images, that contain a clear pixel-based structural pattern, there is no unified representation for 3D object reconstruction that is both computationally and memory efficient. In terms of used 3D representations, existing methods can be broadly divided into the following categories:
Voxel-based methods are a generalization of 2D pixels into 3D space, and thus constitute a natural extension for classical image-based methods. Early works focused on 3D convolutions operating on dense grids (Choy et al., 2016) to generate an occupancy function that determines whether each cell is inside an object or not, however these were limited to relatively small resolutions. To address the high memory requirements of dense voxel grids, various works have proposed 3D reconstruction in a multi-resolution fashion (Häne et al. 2017), with the added complexity of requiring multiple passes to generate the final output. More recently. OceNet (Mescheder et al. 2019) proposes encoding a 3D description of the output infinite resolution, and shows that this representation can be learned from different sensor modalities.
alternative occupancy functions, where instead of the occupancy state we learn a function describing the signed distance to the object surface (Dai et al. 2017
Signed Distance methods
Stutz & Geiger, 2018). This approach builds upon earlier fusion methods that utilize truncated signed distance function (SDF) introduced in (Curless & Levoy 1996). DeepSDF (Park et al., 2019) represents 3D space continuous volumetric field, and requires at training time the ground-truth
SDF calculated from dense input data using numerical methods. Implicit Geometric Reconstruction (IGR) (Gropp et al., 2020) is a SDF variant that uses Eikonal regularization, thus enforcing that the output of the decoder will be the SDF of "some" surface. This is an effective way of mitigating the impact of outliers in the final generated surface, and is used as the starting point for our proposed meta-learning approach to shape completion.
Point-based methods directly output points located on the object surface, thus eliminating the need for a dense representation of the 3D space. Earlier works such as PointNet (Charles et al. 2017] Qi et al. 2017) combined fully connected networks with a symmetric aggregation function, thus achieving permutation invariance and robustness to perturbations. (Fan et al., (017b) introduced point clouds as a viable output representation for 3D reconstruction, and (Yang et 20117 proposed decoder design that approximates 3D surface as the deformation of 2D plane. Point Completion Network (PCN) (Yuan et al.) 2018) is recent architecture that enables the generation of coarse-to- fine shapes while maintaining small number of parameters. However, common limitation of all these methods is that they do not describe topology, and thus are not suitable for the generation of watertight surfaces. Also, change the number of output points, methods like PCN have to re-train their networks entirely, while SDF- methods learn the geometry in an implicit form and thus can generate any amount of points.
Mesh-based methods choose to represent classes of similarly shaped objects in terms of a predeter- mined set of template meshes. First attempts focused on graph convolutions alongside the mesh's 's vertices and edges (Guo et al.) 2015). and more recently direct output representation for 3D) reconstruction (Kanazawa et al. 20188). These methods, however, are only able to generate meshes with simple topologies (Wang et al. 2018), require reference template from the same object class (Kanazawa et al., 2018) and cannot guarantee water-tight closed surfaces (Groueix et al., 2018). A learnable extension to the Marching Cubes algorithm (Lorensen & Cline! [987) has been proposed in (Lio et al. 2018), however this approach is limited by the memory requirements the underlying 3ID voxel grid.
2

---

Under review as a conference paper at ICLR 2021
2.2 ENCODING AND DECODING MECHANISM
In shape completion methods, one must construct geometry of object from point clouds of varying size that are scattered over various regions. Thus, all methods use some mechanism to encode the sparse point clouds tensor of fixed size, and decode this tensor produce the final output. Existing methods also differ by the design of these decoding and encoding mechanisms.
Methods like DeepSDF and IGR train an auto-decoder, and do not train separate encoder func- tions at all. To find the geometry-describing implicit parameters for each object, these methods apply likelihood-based gradient descent on randomly initialized latent variables, using just sparse observations from the target object. Thus, these methods by design do not use observations from multiple objects to train an object-agnostic mechanism that can efficiently learn the latent variables. Meanwhile, PCN and OccNet both train a complex encoder function that maps sparse observations to latent variables. Although their decoders differ entirely (PCN directly outputs 3D points, while OccNet outputs binary values), they both use an encoder that aims to capture the hicrarchical structure of the object's s geometry. More specifically, PCN's encoder is equipped with a mechanism to extract information from sparse observations in two steps, one aimed at extracting global information and another at extracting local information. On the other hand, OccNet's encoder is a version of PointNet (Charles et al. 2017) that uses max pooling with respect to the sparse observation set.
Our method also uses an encoder function, however it differs from those mentioned above in the sense that it represents posterior distribution, rather than deterministic function. Under appropriate set of regularity conditions, inference made from the predictions of posterior distribution is optimal in terms of the Bayes risk (Maeda et al., (2020). Furthermore, because our encoder is probabilistic, is capable of outputting multiple candidate shapes for a given sparse observation.
3 META-SHAPE COMPLETION
3.1 PROBLEM SETTING
Let D3 = (x 1, NN) be an arbitrary set of points located on the surface of a 3D object k. The goal of shape completion for object l is to use Dj to infer its surface. There are various approaches to this problem: if the surface closed, one can use an occupancy function (o)e that tells whether point is inside or outside the surface. One may also use signed distance function Faar (x) that evaluates how far point æ is to the surface, with positive values indicating that it's outside and negative values indicating that it's side. In both approaches, the transcribed goal is to use D2 to find the object-A-specific function F& that best describes its geometry. Our proposed method focuses on the latter approach, and in the following sections we show how Bayesian meta-learning can be used to learn probabilistic SDF for shape completion.
3.2 META-LEARNING FOR SHAPE COMPLETION
Meta-learning exploits underlying similarities among tasks to enable the transfer of knowledge between tasks, so that information gained from solving one problem can improve performance on another. Traditional meta-learning methods work by parameterizing their models with two types of parameters: task-specific and task-agnostic. Following this basic philosophy, we represent the signed distance function (SDF) of all observed objects using object-specific parameters hg and object-agnostic parameters A. More specifically, we write our SDF for object l as Fo((;; ha), and determine h& using the 6-parameterized function that encodes DR- The function that maps Da to h4 is often referred to as encoder in the meta-learning literature. The SDF, on the other hand, plays the role of decoder. In a probabilistic setting, we consider a probabilistic SDF of form (s ha), oz))
Now, if we interpret this estimation problem case of minimizing the Bayes risk Maeda et al. 2020), we can show that the optimal solution is given by the predictive distribution pe r; Dz) computed using the posterior distribution p(h] D+):
p(s|2, / p(spz, (w)p(he) Da)dhs.
(I)
3

---

Under review as a conference paper at ICLR 2021
Enceder for Shapellet0
Rell
Inpul xt"
PALL
PALL
PeU
Query Input
Estimated query output
x7
Eacodar
12 2
Decoder
hx
Encoder IOUMUIM
Aggrogator
Mo (8)
Enoodor
x
Figure 1: Schematic of our proposed SDF-bbsed meta-shape completion method.
The estimation of the posterior Dx) is a challenging task: for the approximation to be valid, it must be able accept unordered sets Dx of various lengths, while satisfying all other properties of a posterior distribution. For example, its variance must approach zero as we take the number of observed points to infinity. According to the theory of (Maeda al.) 2020), we can construct an approximation that satisfies all these requirements using a Gaussian distribution (h ( = IT (h,, (u,,o(DD) o with mean and variance as follows (we use hy.i denote the i-th element of hg):
NL Jim > nmm
N2
(D())) =
(2)
9h
g
91,0
In the above, Jin = Sú,0 (an 3/m and gi.n = Fi.o( (1 ya are neural networks parameterized by 0. When making probabilistic inference, we sample from this Gaussian posterior approximation and feed it to the decoder SDE. A schematic diagram of our proposed meta-learning shape completion method can be found in Figure
(k) (K)
(k)
The training proceeds by making predictions about target points using contextual information from various datasets. To prepare the mock-target points at training time, we decompose D₁ into Detat and Daan for each task l and the following classic ELBO (Ranganath al. 2014 Kingma & Welling 2014) to optimize the predictive distribution given D3:
No.
(x) = =- --/-rolhaDDa
Σ
log po
(5 (m!"), (bx)) + log (hu (helDgtttt) dhg - ((po(he(DH))
(3)
(k) Here 8n the true signed distance from point 2i to the surface of object k. In our problem setting, sn is 0 for all (n, k) because all the observed points are on the surface of the objects. Figure contains a diaram of our proposed algorithm. To test the sheer efficacy of our meta-approachl we
only use MLP for encoder in this study, and do not explore the specific architecture that is suited for the shape-completion problem.
3.3 EIKONAL REGULARIZATION
The datasets Dr considered in shape completion tasks usually only contains points the object surface. However, it is difficult to enforce the learned function to be a signed distance function (SDF) just by enforcing that is value should be close Regularization (IGR) (Gropp et al. 2020) is a regularization based on the theory of Eikonal partial differential equations, which states that any function F' satisfying
observed points. Implicit Geometric
F(x) = Vr E B, [|VF())] = Vr E TR
(4)
a signed distance function for the surface B. To better encourage our decoder to describe a valid SDF, therefore augment loss function by an extra term Leik- ElllV h&)l 1]]. We estimate this expectation by sampling x in the soft neighordhood of the object surface. For more details, refer the reader to Section 4.
4

---

Under review as a conference paper at ICLR 2021
3.4 NORMAL VECTOR REGULARIZATION
If the training dataset contains ground-truth surface normal vectors, we can also add a regularization term encourage the gradient of the estimated SDF at observed points agree with the true normal vectors. This is regularization used in the original IGR (Gropp et al. 2020) implementation as well.
3.5 Loss FUNCTION AND
LATENT OPTIMIZATION
The loss function we minimize at training time for object given by
(5) where À is an empirically chosen regularization parameter. At inference time, we can also further fine-tune the encoder output by optimizing the likelihood of the sparse observation. That is, if n is the the mean of the encoder conditioned against D2 (i.e. E[h[D&], we additionally apply the following iterative updates on the encoder's 's mean:
L (0) ACeik.
p
P1-1
E
Vn logo 1
(6)
At inference time, we feed n to the decoder instead of p When we apply this post-encoder latent optimization, method becomes version of IGR that is additionally equipped with encoder.
4 EXPERIMENTS
We conducted series of experiments to evaluate the efficacy of our proposed Meta-Shape Completion (MSC) method, relative to other well-known published methods. Particularly, we focus on very sparse scenarios, and show that the introduction of Bayesian meta-learning enables the generation of state-of-tt the-art shape predictions from a very small number of observed samples. Similarly, we show that our proposed method generalizes better to novel and unseen objects.
4.1 DATASETS
ShapeNet (Chang et al. 2015). We used the Synthetic ShapeNet CAD models as the primary source of evaluation for this paper. Following the procedure described (Park et al. 2019). we first applied an affine transformation to each object, so the center of mass is located at the origin, and rescaled all points so that the maximum distance from vertex origin is To exclusively sample points on the object-surfaces, we first generated 100 view-points uniformly around each object, and used OpenGL obtain images from these view-por peintsontss We then identified the set of points that are observable from these 100 view points as surface points, and took samples from these points. This modified dataset is henceforth referred to as Disemboweled ShapeNet. For IGR and MSC, we used the above procedure to generate 200k pairs of coordinate and normal vectors each object surface. For DeepSDF, followed the original procedure and computed the ground tround-truth SDF at randomly generated 500k: points in addition the 200k surface points. For OccNet, randomly sampled 100k points from those used in DeepSDF, and annotated whether they are inside or outside the object. For PCN, we sampled 16384 surface points as ground-truth.
ICL-NUIM (Handa et al. 2014). This is a dataset consisting of RGB-D images from two different scenes: living rooms and office rooms. The task on this dataset reconstruct 3D shapes from depth images. As pre-processing, we normalized location and scale of each scene using the same procedure described above for ShapeNet, and used Open3D ZZhou et al. 2018) to obtain normal vectors. Because Open3D sometimes produces normal vectors with inconsistent orientations, we conducted an extra step of inconsistency correction in which we made their inner products with the camera vector to be all positive.
4.2 EVALUATION
As our evaluation metric for each method, we calculated the Chamfer distance (Fan et al., 2017a) between ground-truth and generated point clouds. If P, and Pa point clouds in 3D space, the
5

---

Under review as a conference paper at ICLR 2021
Table I: Results on ShapeNet trained categories (Mean Chamfer Distance per point).
Row Chamfer Distance (X10
Noemalized Chamfer Dustance (wuzt. MSC opt)
Method
300
1000
Gana: Densificatoon
11.01
10.32
PCN Yuas 2018
=1:00
#0181
0011
3.90 +5.19
3.92
1.70 =1.47
=0.37
OceNor Ipaeer or
2019
6.58 99557
1,76 1:597
0.61 1:402
0.45 1.22
1674 1:61 39
9.42 1:2.91
4.55 1:5444444
3.55 -:5
DeepSD or 2019
5,38 -3.1
5.06
13.44
4.72 2:333
4,58 -3.32
13.16
1:25
24.62 1:4.41
31.89 254.77
200.37
IGR (Gmppe pe 0040
3.61 -4.54
3333
+4.36
3.14 +..24
3.09
-4.11
8.64
+23.38
15.65 +37.04
24.28
+51.47
28.52 +39.50
MSC (wo opt)
25.0
0.68
1:337
0.40 10,99
0.39
-2.42
MSC (w)
0.67 -9,00
0.38 +.0.2
0.29 =10.94
0.29
m1.29
0.22 +1.92
0.90 +10.93
1,08 =7.07
1.24 =13,47
Table 2: Results on ShapeNet novel categories (Mean Chamfer Distance per point).
Mothod
50
Raw Chamter Distance (X11300)
100
300
1000
50
Normalized Channter Distance (WEE. MSC wpo
100
300
1080
Games Densilication
2,97
-0.80
0.45 1:022
0.13
4.33 $4.9%
4,02 2:.73
2,09 22.55
0.75 10.92
PCN Yean al 2018
2.06
0.34
0.20 +00.7
2.94
2.74
1.18
OceNer
508 112.85
1.37
0.68
12.76
117.52
1.53
Deep Park 20p
786
=4.11
7.34
7:21
15.05 #:2.991
26.07 #37,53
28.73
#6001
KGR
Gimplren 2007
3,90 +3.47
=3.27
3.88 +3.27
3.4 43222
7.24 122883
11:20
15.34 2:24.5
16.72 +26.53
MSC (wr opo
1.60 1:.13
0.90 22.86
0.62 #1.74
0.53 11120
MSC (w/
0.68 +0.71
0.47 -0.57
0.41 +-0.50 50
0:40 +0050
0.75 0..8
0.79 -0.80
0,98 +1122
1.12 +1.09
Chamfer distance between them is given by:
d(P1, = Σ
min b - 1/13/1111
+ Σ
min Jr, -2,12/122
iEP;
iEP
JEP;
Let p& be the ground-truth point cloud of the object k. and PM be the point cloud of object k generated by method M. To evaluate the performance of method M. we computed di d(Ppe PPM) for each & and used it to calculate the following values:
K
K
JM
d
J.M dooom
(ik /d
K
1=1
where Mo is our proposed method, so that JM is a measure of method M's performance relative our own performance. We refer the former value average chamfer distance, and the latter as normalized chamfer distance. For the ICL-NUIM experiments we used an assymetric variation of the Chamfer Distance, where only the first sum term in Equation 7 is used. This is because the trouh point clouds for this dataset are very sparse and lack many portions of the scene to be reconstructed.
To generate ground-truth point clouds, we randomly sampled 30k points from the raw meshes in our Disemboweled Shapener dataset. Each point on the object surface was sampled in two steps: first, we selected triangle on the polygonal mesh with probability proportional to the area of the triangle. Then, we sampled a point from the uniform distribution over the selected triangle. DeepSDF, IGR and OccNet use implicit functions to represent the geometry of each object, so their decoders do not directly output a point cloud. For these methods, we used Marching Cubes (Lorensen & Cline 1987) to generate a point cloud of size 30k. PCN, on the other hand, directly outputs 3D points from each set of sparse observations. Following the original implementation, we used those to generate a point cloud of size 16384.
4.3 SHAPE COMPLETION ON SHAPENET
To verify the ability of our proposed method to complete the shape of unseen categories, we split the object-categories of Disembowled ShapeNet into two groups: one to be used to train the model (training), and one to be used as newly-encountered object type (novel). For the exact splits, please refer to the Appendix. In the first set of experiments, we trained the model on the training dataset of training categories, and evaluated the model on the test set of the training categories. Results for this first set of experiments can be found on Table T In the second set of experiments, we trained the model the test set
each model the training set of training categories and evaluated of novel categories. Results for this second set of experiments can be found on Table [ In both cases, our proposed method (MSC) was evaluated with and without post-encoder latent optimization.

---

Under review as a conference paper at ICLR 2021
Input Points (NK)
PCN
OccNet
DeepSDF
IGR
MSC (w/o opt)
MSC (vv opt)
Ground Truth
Figure 2: Shape completion results on ShapeNet for chairs (training category).
Input Points
PCN
OrcNet
DeepSDFF
MSC (wo opt)
MSC opti
Ground Truth
Figure 3: Shape completion results on ShapeNet for caps (novel category).
As we can see in both tables, our proposed method consistently outperforms all the other methods for both novel and training categories when the number of observation points 100 or less, while achieving competitive results when higher densities are available. In particular, when the number of observations 50, our method greatly outperforms PCN, the current SOTA these tasks.
The strength of our method can be verified qualitatively as well. Figures 2 and 3 are examples of completed shapes obtained by different methods for various ShapeNet categories. As shown, our method tends to output smooth surface predictions regardless of the number of observations. This most likely because our encoder is learning a task-agnostic property of smoothness, that can be transferred across different objects. Meanwhile, the performance of IGR differs drastically across trained and novel categories, both in terms of Chamfer distance and output appearance. DeepSDF performs poorly on all categories, most likely because it not allowed to use ground-truth SDF test-time. When the number of observations is small, DeepSDF tends to output a complex nonsensical surface that looks like a combination of all the training objects. also tends to produce disconnected artifacts that are located far away from the real surface. OccNet also tends to produce many artifacts when the number of observations small. We hypothesize that these methods are failing to isolate category-agnose agnossic properties from category sapegor properties. As a general remark, all methods struggle with reproducing detailed topology even with large number of observations, particularly in the case of holes and large gaps, possibly due to model limitations. For a detailed analysis of the category-wi wise performance of various methods, please refer to the Appendix.
4.4 SCENE COMPLETION OF ICL-NUIM DATASET
Table summarizes the performance of various methods on ICL-NUIM, for the task of scene completion. Similarly to ShapeNet, MSC significantly outperforms all others when the number of observations is 300 or less, while still achieving competitive results when a higher density of observations is available. [igures[andd Jonnaa examples of scene completion results produced by different methods. DeepSDF and OccNet both tend to produce very complex shapes with many disconnected artifacts, and Dee in particular seems to not converge as we increase the number of observations. PCN performs poorly when compared to the shape completion task on ShapeNet, mostly densifying observed areas rather than extrapolating this information to other portions of the scene. Our method, on the other hand, succeeds in correctly recognizing the boundary of objects even when large areas, such walls and the floor, are missing in the ground-truth.
7

---

Under review as a conference paper at ICLR 2021
Table 3: Results on the ICL-NUIM dataset (Mean Asymmetric Chamfer Distance per point).
Metlod
Row Asymmetric Chamler Distance
T000
Normalized Asymmecic Chamder Distance
100
X00
MSC
Gauss Densification
0.22
1.4.6
11.74
110.00
=2339
=0:00
0.28
7.94
=0.53
OneNir Metcerore
2019
0:49
0.30:0:10
0.35 10.15
0.24 00.15
1:45
2055
20.44
1.55 =042
DeepSD pa ai 2019
3.33 23110
2:12-5:48
100 22:71
0.57 11.36
10.09 120.04
9:51 1.3435
5.91 1:7.61
3.45 =8.17
IGR (hopp 14 00201
3.11 +5.76
2.28
=5.01
1 94.26
1,30 -4.07
8.49 +:7.27
9.02 +19.90
8.77 +22.49
7.98 #24.74
MSC (wlo opt)
0.62 1.57
0.32
.0.74
0.21 10.56
0.19 10.53
MSC (w7 op)
0.20 =0.33
0.15
=0.15
0.14 +0,03
0.14 -0.03
0.59 +1.36
0,66 =8.57
0,80 =0.16
0,87 =0.14
Input (NG)
PCN
OccNet
DeepSDF
IGR
MSC (w/o opt)
MSC opt
Ground Truth
Figure 4: Scene completion results on ICL-NUIM for office rooms.
Input Points (N0)
PCN
OccNNt
DeepSEDF
opt)
MSC opt)
Ground Truth
Figure Scene completion results on ICL-NUIM for living rooms.
CONCLUSION
In this paper we introduce the concept of meta-learning to the task of shape completion using implicit representations of 3D surfaces. Our proposed encoder mechanism allows the learning of object-agnostic properties separately from object-specific properties, thus succeeding in training a model that can consistently produce smooth predictions from highly sparse observations, achieving state-of-the-arr under these conditions. Although we have used IGR, an SDF-based method, as the basis for our implementation, our proposed meta-shape completion algorithm (MSC) can be equally applied to different implicit surface representations by simply changing the decoder. Furthermore, in this paper we have used a simple MLP-based encoder to learn task-specific parameters, while other methods like PCN use hierarchical models to capture both global and local geometric properties of objects. We believe that using these more complex models will lead to substantial improvements to proposed method, however this is left for future work. In conclusion, there seems to be lot of room left for the application of meta-learning to scene completion tasks, and further studies in this direction may allow to develop models that can be used in wide range of applications.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, al. Shapenet: An information-rich 3d model repository. arXiv preprint 1rXiv:1512.0300 2015.
R. Q. Charles, H. Su, M. Kaichun, and L. J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 77-85, 2017.
Chris Choy, Danfei Xu, Jun Young Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. volume 9912, pp. 628-644- 644, 10 2016. ISBN
doi: 10.
Brian Curless and Marc Levoy. volumetric method for building complex models from range images. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, SIGGR APH '96, pp. 303-312, New York, NY, USA, 1996. Association for Computing Machinery. ISBN 0897917464. doi: 10.1 00.114//37770.2237666 URL httpt:ttt///ttt /ttp///o..o..//00///////////////
1145/237172 0t00/2237770.32266
Angela Dai, Charles Ruizhongtai Qi, and Matthias Nießner. Shape completion using 3d-encoder- predictor cnns and shape synthesis. In Proc. Computer Vision and Pattern Recognition (CVPR). IEEE, 2017.
Harrison Edwards and Amos Storkey. Towards neural statistician. In ICLR, 2017.
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, and Karol Gregor. Neural scene representation and rendering. Science, (2006339)):20442222 2018. ISSN 0036-8075.
H. Fan, H. Su, and L. Guibas. A point set generation network for 3d object reconstruction from single image. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.
2017a.
H. Fan, H. Su, and L. Guibas. A point set generation network for 3d object reconstruction from a single image. In 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 2463 2271-247,2222 2017b.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning- 70, pp. 1126-1135. JMLR. org, 2017.
Moanning-Volumee Volume
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint 10X.vv1807.01660 2018.
Silvio Giancola, Jesus Zarzar, and Bernard Ghanem. Leveraging shape completion for 3d siamese tracking. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regular- ization for learning shapes. arXiv preprint arXiv:2002. 2020.
Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan Russell, and Mathieu Aubry. AtlasNet: A Papier-Mäché Approach to Learning 3D Surface Generation. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018.
Kan Guo, Dongqing Zou, and Xiaowu Chen. 3d mesh labeling via deep convolutional neural networks. ACM Transactions on Graphics, 35:1 1-1,, 12 2015. doi: 10.1145/2835487.
A. Handa, T. Whelan, J.B. McDonald, and A.J. Davison. A benchmark for RGB-D visual odometry, 3D reconstruction and SLAM. In IEEE Intl. Conf. on Robotics and Automation, ICRA, Hong Kong, China, May 2014.

---

Under review as a conference paper at ICLR 2021
C. Häne, S. Tulsiani, and J. Malik. Hierarchical surface prediction for 3d object reconstruction. In 2017 International Conference on 3D Vision (3DV), pp. 412-420, 2017.
Angjoo Kanazawa, Michael J. Black, David W. Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In Computer Vision and Pattern Regognition (CVPR), 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of the 2nd International Conference on Learning Representations, 2014.
Yiyi Liao, Simon Donné, and Andreas Geiger. Deep marching cubes: Learning explicit surface representations. Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
William E Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics, 21(4):163-169, 1987.
Shin-ichi Maeda, Toshiki Nakanishi, and Masanori Koyama. Meta learning Bayes risk minimiza- tion. arXiv:2006.01488. 2020. URL
01488
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sehastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4460-4470, 2019.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove, Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 165-174, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W. W, 2017.
Charles R. Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Pointnet++:: Deep hierarchical feature learning point sets in metric space. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pp. 5105-5114 Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Rajesh Ranganath, Sean Gerrish, and David Blei. Black box variational inference. In Artificial Intelligence and Statistics, pp. 814-822, 2014.
David Stutz and Andreas Geiger. Learning 3d shape completion from laser sean data with weak supervision. In IEEE Conference Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, 2018
Jacob Varley, Chad DeChant, Adam Richardson, Joaquín Ruales, and Peter Allen. Shape completion enabled robotic grasping. 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2442-244222 222-2244,444 2017.
Nanyang Wang. Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In ECCV, 2018.
Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian. Foldingnet: Interpretable unsupervised learning 3d point clouds. ArXiv,
2017.
Wentao Yuan, Tejas Khot, David Held, Christoph Mertz, and Martial Hebert. Pen: Point completion network. In 2018 International Conference 3D Vision (3DV), pp. 728-737. IEEE, 2018.
Qian-Yi Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3d: A modern library for data processing. arXiv preprint arXiv: 801 008447 2018.
10

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 IMPLEMENTATION DETAILS
DeeDSDF PPrr et al. 2019 and IGR (Gropp et al. 2020). We implemented both methods using Pytorch (Paszke et al. 2017). matching their model architectures, initialization procedures and published results. As done in the original study of (Park et al.] 2019), however, we used ReLu in place of softplus for the Deep SDF decoder. To create the mesh with which the Chamfer distance is calculated, we used the Marching Cubes algorithm (ooeenen & Cline 1987) with 256 resolution to convert SDF to mesh. To find the optimal & with respect to the likelihood, we applied gradient descent erroro.
from random initial value 2u sampled from (0, 0.012). For the stepsize, we chose where erroro is the error computed with 2g. We used the Adam optimizer, with batch size b = 32, alpha a = 3.2 x 10-4, and learning rates Ir 1.0 10- for DeepSDF and Ir = 1.0 x 10 for IGR. We trained for 5000 epochs, halving the learning rate at every 500 epochs. We used 16384 surface points to compute the loss for each object. To generate points for the evaluation of the Eikonal term, we randomly selected 5k points from the 16384 the surface, and sampled one point each from a Gaussian distribution centered around it, with variance of 4.0 x 10 and X = 1.0 as the Eikonal regularization parameter. We trained our IGR model with the version of the algorithm in (Gropp et al. 2020) that uses the surface normal vectors.
OceNet (Mescheder et al. 2019). We used the authors' official Pytorch implementation! and trained under the same conditions as described in the original paper. For the encoder architecture (Figure 6 we used the ResNedPoiniNet class, and for the decoder architecture class. We trained the decoder over 5k epochs with batch size b = 64 and learning rate Ir = 1.0 10 For each object, we used 300 surface points the encoder, and produced 2048 binary points from the decoder. To stabilize training, we followed the same procedure described in the original paper and added a Gaussian noise with standard deviation
used the DecoderBatchhorm
5.0 x 10-3 3 to the encoder input.
PCN Yuuan et al. 2018). We implemented PCN using Py Torch based on the authors' official Ten- sorflow implementation matching their model architectures, initialization procedures and published results. To train, we followed the same procedure as the original paper, feeding k surface points to the encoder. We used batch size b- = 32. Starting from the initial value of 0.0001, we reduced the learning rate by a factor of 0.7 at every 50k iterations and trained the model for total of 5k epochs.
MSC (Ours). Figure of the main paper illustrates our proposed encoder network. The dimension of (i.e., the output of the encoder) is set to 1024 for ShapeNet40 and 512 for the ICL-NUIM dataset. We used the same decoder as IGR. At training time, we set the number of contextual surface points to be 16384 and evaluated the loss using target sample points of size ranging from 1 ~ 5000, chosen at random at each iteration. We trained our model for 5k epochs with batch size = 32 and learning rate Ir 10-1, using the same Eikonal regularization term IGR. For the ShapeNet experiment, training the model required 16 days with 16 GPUs (NVIDIA V100). The encoding of 1000 contextual points took 0.003 seconds on average, and 22.660 seconds to create the mesh surface using the Marching
https: httter//gtauu..co///ttttoooouo/r
networs
Input PC
FC-1024
ResNet Block
Max Pool
Expand and Concal
RexNet Block
Max Pool
PC-112
1024
512
512
I.....4
Figure 6: Encoder of OccNet (Mescheder al. 2019). OccNet uses version of PointNet that carries out max pooling procedures in multiple places. Compared to our encoder (Figure 1, main paper) this architecture is much more complex. Future work will involve extending MSC to different encoder architectures.
11

---

Under review as a conference paper at ICLR 2021
Cubes algorithm (which requires evaluation at multiple points). For the ICL-NUIM experiment, it took 30 hours with 32 GPUs to train the entire model.
A.2 RESULTS ON SHAPENET
Tables 4 are contain more detailed shape completion results for each ShapeNet training category, and Table O contains similar results for each ShapeNet novel category. Qualitative results for selected objects can be found in Figures 2-12. Generally speaking, performance of all methods vary greatly across categories, and as expected, all methods perform better on trained than novel categories. Some objects were particularly difficult for all methods, such as headphones, for which it difficult to determine whether the target object is ring-shaped not until there is a large amount of contextual points. Following the same logic, all methods also performed poorly on objects with intricate details, such motorcycles and keyboards.
As described in the main paper, PCN tends to outperform MSC when there is a large number of observations. However, in many of theses cases simple Gaussian densification also yields good results. In fact, Gauss densification outperforms all methods on most categories when the number of observations is large as 1000. This suggests that the representation power of these methods is still not sufficient. Meanwhile, MSC outperforms all methods on most of both novel and training categories when the number of observations is small. Particularly, when the number of observations is 50, our method with post-encoder optimization outperforms all methods on all novel categories.
When the number of observations is 100, MSC with post-encoder optimization outperforms other methods in 10 out of 15 categories. However, we note that the categories where MSC is outperformed (pistol, motorcycle, guitar, earphone, microphone) correspond to objects with fine details, and in 2 of them the best performing method is Gauss densification.
IGR, which represents a version of our method without the encoder, sometimes produced nonsensical outputs even with large number of observations, apparently switching between different objects. Similarly, DeepSDF has a tendency of producing outputs that look like combination of many objects. It possible that, because IGR and DeepSDF are not equipped with an encoder, they are indeliberately mixing object-specific with object-agnostic features by encoding some object-specific properties into their decoder model. DeepSDF also produced disconnected artifacts in many cases, mostly because we only allow the use of contextual observations at test time (e.g. information like ground truth sdf values at points not on the surface and ground truth normal vectors are inaccessible at test time with sparse observations). OccNet also produced such artifacts when the number of observations is very small. Our proposed MSC method, on the other hand, makes relatively conservative smooth prediction when the observations are too sparse, and incrementally increases the complexity of the output shapes as more points become available.
A.3 RESULTS ON ICL-NUIM
Figures 13-15 are selected scene completion results obtained on the ICL-NUIM dataset. Although both the ground-truth and contextual point-cloud have missing large portions because of data col- lection procedure (collected from single view), our MSC is able to generate smooth surfaces that cover the unobserved areas of the environment. Interestingly, in this case IGR also does a good job producing smooth surfaces, most likely because all rooms are somewhat familiar to each other (i.e. oppecc-specific properties can be shared between classes without significant detrimental effects). DeepSDF again produces disonnected artifacts on this dataset as well, Failures cases for DeepSDF indicates that the Eikonal term plays important role in producing smooth surfaces, especially when the model is not allowed to use normal vectors. Finally, PCN fails to reconstruct unobserved areas of the environment, producing instead what looks like densified version of observed points.
12

---

Under review as a conference paper at ICLR 2021
Table 4: Results for each ShapeNet trained category (1/2)
Caregory
Context
Gauss densification
PCN
OceNer
Method DeepSDF
IGR
MSC (wlo opo)
MSC (w/ opt
50
1.22 + 0.46
0.84
00.44
1.45 # 2,84
3.33 1.56
3307 = 1.62
0.69 = 1.60
0.31 = 0.40
Plane
100
0.54 # 0.19
0.36
0.23
0.37 # 1.07
3.34 + 1.54
2,84 1,43
0.40 = 1.49
0.19 :: 0.33
300
0.06
0.14
3.32
0.17 1
1000
0.05 1 0.02
0,07
= 0.09
0.10 2 0.47
3,38
2:70 1.26
0:20 0.42
0.18 2 0.34
6.30 1: 1.24
4,41
= 1.47
3.84 1 3.24
1.93 1 1.00
0.34 0.93
1.96 1.31
1:01 1: 0.67
Gurbage Can
3.0 0.56
D33
1.04
0.56 0.45
300
0.97 0:188
0,24::0.17
0:53 1.09
1:30 = 0.92
0.74
0.85
0.42 0:334
0.37 :: 0.31
1000
0.29 1 0.06
0.18
0.12
0.13 1: 0.93
1,22 1 0.82
0.70 0.75
0.35
= 0.258
0.32 1 0.29
50
6.28 1: 1:75
4.14
1.49
5.61 1: 3.68
3:26 = 1.36
2.43 1.26
1.69
1.11
0.88 1: 0.94
100
3.05 4: 0.94
2.09
00.72
2.25 4: 2.12
3,15 4 1.34
2:26
1.20
1.01 1.11
0.58 2: 0.89
Basket
300
0.93 0.29
0.46
0044
1:45 2.05
3302
2:12 1.23
0:47 0.73
1000
0.28 0.11
0.27
0.37
1.11 x: 1.54
2.76
2:15
0.91
0:47 0.89
4.96 1: 1.64
3.07
1.12
8.63 1: 6.15
1.20
1.33
1:57 1,06
1:39 1.06
0.64 1: 0.51
Bathtub
300
100
0.78 4: 0.28
2.44 0.89
0.36
2016
0.78 1: 1.47
3:41
4,15
4.16
1.67
1.64
1.11 1:58
0.38
0.49
0.68
0.27 1: 0.35
1000
0.23 + 0:08
0.16
1::6=:00 0.12
1.54 =: 0.90
3386 1 1:37
1:05
0.34 = 0.41
0:30 1: 0:40
3,95 + 1:29
1.55
7:05 9.35
3.17 1.79
0.53 1.47
1:23 1.00
0.57 0..33
Bottle
300
100
0.56 1. 0.20
1,56 0.67
0.29
125
0.66
0.16
0.16 0.36
0.52 2 0.44
2.64 = 1.63
1.89 1.18
0 1:41
0.43 1.39
0.42 = 0.44
0.22 0.31
0,16 1: 0.33
0.23 2 0.39
1000
0.16 + 0.06
0.09
0.08
0.12
1.61
1.009
0.41
1.31
0.18 00.38
0.13 0.27
50
3,78 a 1.02
2:95
= 1.33
5.68 a: 3.19
4.22 0 1.08
2:41 1.09
0.58 0.03
0.21 2: 0.24
Bowl
100
0.39
1:25
3.82
1.10
0008 0.05
300
0.59 0.19
0.26
0.15
235 1: 4.00
3:93
1.35
1.89 1.02
0.12 = 0005
0.07 1: 0.05
1000
0.19 a 0.06
0.11
2 0.05
1.68 a: 3.30
3.48 1.42
1:85 1.08
0.11 =: 0.05
0.07 1 0.05
6.31
1,46
1.46
10.50
3.30
1.59
1.58
1.00
0.95 0.56
Cabinct
100
3100 0.67
1,66
0.63
3.27 2 5.27
2,47 = 1:53
0.68 0.99
0.71 0.47
0.51 2 0.31
0398
+ 0.20
01:3-
0.14
13.11 0.32
2,08 + 1:30
11:00 0.90
- 0.26 11:88
0131 0.30
0.28 0.06
0.18
1,80
0.61 0.88
0.30 0.28
50
4,18 1: 1.39
4,01
= 1.91
2.61 2: 2.38
4.07
1 2.95
327 2. 2,58
2:61 a 2.62
1:20 1 0.88
100
118 0.22
2:14
122
0395 0.85
2.25 2.98
3:42 2.41
1.50
0:91 0.79
Cumera
300
0.66 + 0.22
0.75
0.54
0.43 0.53
3.91 2.95
3:45 2.50
1.06
0.75 0.76
11X0
0.19 + 0.06
11:88
0.49
1337 + 0.35
3391 + 2.03
5009 2:36
1.19
1.67
0081 0.72
6,70 1: 1.38
4,27
3:01 ::
1.85
0.16 0.18
0.93 0.63
Can
100
3.17 0.54
2.32
0.91
0.25 a: 0.27
1.49 a 0.48
0.11 0.10
0.56 a 0.58
0.64 1: 0.78
300
0.98 + 0.15
0.37
= 0.10
0.17 = 0.30
1.34 = 0:41
1111= 0.12
0.13 = 0.11
0.10 = 0.15
1000
0.28
0.044
0,05
1,26 0:40
0.08
0.10 0.15
50
3.97 0.77
3.65
1135
2.16 1: 2.62
3.87 + 1.30
0.79 2: 0.72
0.74 0.52
0.50 1: 0.36
Car
300
100
0.62
1,95 0.356
1.83 0.33
0.37 1: 0.73
0.16
3.84 1: 1.15
3.71 1.11
0.61 1: 0.64
0.36
= 0.27
0.25
0.27 1: 0.24
0.19 0.21
[000
0.19 +: 0:03
0.11 0.07
0.15 +: 0.33
3,56
1.11
0.42
1 0.56
0.19 0.22
0.18 = 0.20
3.44 0.86
1,73=00 -081
+ 10.76
3.19
0.600 2.56
0.44
0.20 0.28
Cellular Phone
100
1.74 0.40
0.91
= 0.37
1.08 1: 7.49
4,53 = 3:18
0.38:220 2.14
0.26 =: 1:21
0.09 1 0.13
300
0.58 0.12
0.18
0.07
0.34 1: 3.11
3.18 a: 2.67
0.58
1: 2.31
0.09 0.18
0.07 1: 0.11
1000
0.17 1: 0.03
0.05
0.03
0.27 2.80
2.67 2.49
0.56 2.30
0.08 0.13
0.05 1 0.08
50
4.31 + 1.81
2.88
1.40
8.52 = 7.35
4.60 1.62
2.09 1.26
2.36
= 9.64
0.33 :: 0.72
Chair
T1X
218 + 0.82
1.28=00
0.71
1.38 + 5.66
4.33
2:38 1.53
1177 0.9
1133 + 0.35
300
0.28
0.35
3:54
1.52
0.23 0.28
1000
0.18 1. 0.08
0.16
0.12
0:26 a. 1.04
3.74 1.52
2.16 1.41
0.35 0.49
0.22 1: 0.30
3.77
1.71
2.35
1.44
9,44 13.75
4.008 3.23
1.29 1.86
1.08 1.17
0.58 0.64
Clock
1.8
1.19=030
2.47
3:55
1.29
0.65
800
0000 0:25
0.32
022
1108 + 2.65
2,89 m 3.22
1-18 1.07
0422
0.30 0.34
1000
0,13 2 0.07
0.16=
= 0.17
0.56 2 2.53
2:44 2.72
1.17 1.27
0.37
= 0.86
0.28 2 0.36
30
8.39 1 1.09
3:95
2.35
6.77 1. 2.26
2.52 1 1:25
0.31 0.37
2:48 1.13
1.49 1 0.80
Disbwasher
300
100
3.99 + 0.58
1.33 0.16
3.01
-0--0000 0.83
0.08 0.16
0.42 + 0.62
1.98 0.78
1.96 + 0.44
0.20 0.32
0..30
1.26 0.83
0.15
1.00 + 0.63
0:37
0.04
0.10
0.10
1.89
1.12 0.30
3.74 1 1.27
2.10
1,31
642 : 7.18
5.90 2.81
2.34 2.60
1.39 3,25
0.49 2: 0.50
Display
100
1.88 0.60
L.72
5.11 2.44
2:766
1.28
0.29
800
0.62 + 0.20
0.32
0.18
0.53 + 2.64
4.38 +: 2:25
2.14 2.65
0.1 0.67
0.26 + 0.38
1000
0.19 2 0.05
0.14
0:4:00 0.12
0:31 + 1.96
4.01 25 2.19
2:15 13 2.39
0.37 = 0.37
0.26 : 0.40
2.02 +: 1.24
1.68
1.35
5.79 1. 13.81
10:42 =: 4.56
14.30
8.31
2.10 3.23
0.61 1: 0.93
Fancet
300
100
0.28 1: 0.18
0.93 2 0.60
0.80
0.37
= 0.42
2,56 1: 10.08
1.24 à: 7.29
10.15
9.48 4.21
2 4.49
14.14 7.89
13:21 7.44
0.92 2.35
1.16 =: 2.22
0.33 2 0.52
0.31 4. 0.62
0.09
0.05
0.28
0.40
0:57 2.85
9,24
13.19 7.70
2:26
1028 0.57
7.18 + 1.60
4,42
1.56
8:09 + 6.47
2,87 +: 1.75
0.92 2.21
2.13 = 1.01
1.14 + 0.67
Hile Cabiner
300
3.59 0.9
1.07 1 0.24
0:40
0.56
0.26
0.15 2: 0.32
1,16
2:11 1.43
0.78 2 2.19
0.23
0.31
0.71 0.55
0.35 0.45
1000
0.31 4: 0.07
0.14
0.23
0.18 4: 0.39
1.99 4 1.45
0.78 a: 2.24
0.17 = 0.28
0.21 4: 0.36
4.69 0.83
3,66
2.65
1.94 0.68
T.08
1.62 = 0.87
Helmet
100
2:35 0.37
1.72
0.49
1.22 + 0.85
18332
2.00 1.20
1:29 1.07
1.45 7 1.21
300
0.74 2 0.15
0:53=
2 0.21
1:01 1 0.89
1,71 2 0.75
1.87 1.07
1,48 0.98
1.09 1: 1.02
0.04
0.19
0.98
1,56
1.04 0.96
50
4.47 1
1.51
3,02
1.11
4:15 1: 6.73
2,52
2 1.95
1.31 1.33
2:05
2 2.51
1.02 1 0.84
100
1.18 2: 0.72
1.77=0.8
0.81
1:61 a: 2.50
2.22 1.52
1.16
1:08 1.00
0.69 :: 0.77
300
0.67
0.24
0.29
0.33
1:32 + 5.35
1.78
1.28
0.85= T.05
1356 0.77
1000
0.20 1 6.07
0.27
0.31
1.77 1172:.61 4.61
1.00
1.07
1.08
2 1.19
0.78 = 0.99
0.55 1 0.80
0.65 1: 0.32
0.28
51.60 1 27.64
12.54
3.51
2.82
0.60 1: 2.26
Knife
100
0.29 + 0.15
0.18
0.12
33.18
+ 31.98
12.40
+ 3.55
8.04
3.49
0.49 2.06
1.4% + 8.07
200
0.09 1 0.04
0,07
0.05
17.02
4 29.70
11.06
2 3.50
7.79
3.61
0.16 = 0.14
1:20 7.11
0.04 0.01
11.11 23.88
3.64
1338
13

---

Under review as a conference paper at ICLR 2021
Table 5: Results for each ShapeNet trained category (2/2)
Carepory
Context
Mothod DeepSDF
Gauss densification
PCN
OceNet
IGR
MSC (wil opú
MSC (w/ opt)
50
2,45 # 1,65
1,99 + 1,55
9.29 = 15.61
8.10 = 5.98
6.13 + 7.96
3.38 10.31
1,00 # 1.33
Lamp
300
100
1.08 = 0.74
0.34
102 0.84
3.34 = 9:39
1.21
7.69 2: 6.16
5.70 + 7.63
5.55
1,78 3:22
0.66 :: 0.74
1000
0.09 =: 0.06
0.31 0.35
1.13 2 6.67
6.89
25 5.88
5,40 7.66
1.32 7.39
0.60 2: 1.48
50
3.75 1 0.79
1.86 1.8.2
9.86 1 4.99
10:40
2.83
14.41 1.6 6.24
2.09 1 8.32
0.20 0.25
Laptop
300
100
0.37 0.15
1.73 0:34
D.18
4:07
0.60 = 2.76
1.77
10:22
10:35
25 2,47
2.83
12,60 + 6.05
13.52
1.14
0..44:00.44
0.09 :: 0.19
0.20
1000
0.19 1. 0.01
1,08 1 0.04
0.12 12 2:35
10:07
2.60
12:77 1.:11:10 3.41
0.10 110011000 0.16
0.09 1 0.21
50
6.31 1: 2.22
4,25
2.05
8.28
= 7.44
3.30
2: 2.20
1.06
1.41
1.81 1.10
1.10 =: 0.72
Speaker
100
3,12 4: 1.07
2.03
4: 0.88
1.79 + 3.57
2.78 a: 1.94
0.89 4: 1.24
0.83
1 0.65
0.68 + 0.56
300
0.98 0.34
1:44
1/44::022
0.31 0050
2:45 1.95
0.88
0:41
0.50 0.50
1000
0.29 0.10
0.29
0.67
2:23
0.87
0.47 0.55
50
5:1 2.35
2.61
1: 2:01
5,06 =: 12.01
6.39
4.55
1.52
4 3.61
1.30 0.94
0.00 =: 0.46
Mailbox
300
100
0.52 1: 0.36
1.68
0.31 1: 0.16
0.81
0.15 22 0.18
5.14 as 3.81
4.28
4.04
4.00
3.58
0.44 0.50
0.39 1: 0.41
0.46
1000
0.16 0,10
0.17
0.14 0.17
5:05
3: 3.87
4.10 1:00::4:1 4.15
01446 0060
0.29 = 0.33
8,02 + 1.14
5.70
4.1
3:21 0.54
0.45
2.14 0.84
1.56 1.12
Microwive
300
100
3.80 0.37
1,23 0.13
0.39
2.63
0.166
0.778
0.63
0.08
=: 0.14
2 0.66
2.81 0.777
2.56
1: 0.69
0.33 0.77
0.33 1 0.64
0.37 1:371100 0.57
0.95 0.71
0.31 1: 0.37
0.62 2: 0.50
0.35 0.03
0.04
0.06 0.06
2.40 0.66
0.34
0.11 0.08
0.32 0.35
50
5,46 4 1.02
4,58 1 1.82
2.09 = 1.46
2:16
a: 0.52
2,71 a 1.06
1.53-4 0.96
0.66 =: 0.69
Mug
TOW
138 0.48
2:35 1089
01.32 0.39
300
0.86 1 0.13
0.42 0.22
0.23 2 0.47
1.30 0.41
2,13 0.95
0.39 0.39
0.13 =: 0.14
1000
0.26 a 0.04
0.18 0.14
0.17 a 0.30
1.72 a 0.44
1.95 4 0.93
0.29 1 0.29
0.12 1 0.16
4.78 1.80
1.76
4.96 4.91
5:13 2.85
3,55
0.59
Piaso
T0U
900
0.71 + 1.28
2,33::000 0.91
1:47
1,66
0.75
1124
0.68
0.20 0.16
2 0.66
4.79 2.01
5:08
25 2.71
3.34 2.33
0.633 1 1:40
0.86 4 0.56
0.42 + 0.
0.57 : 0.37
0.08
0.22
3.30
50
4,46 a 0.89
3.79
3.07 2
3:57 2 1.97
0.86
1.56.100 0.90
1,27 1 0.96
0.65 a 0.23
Pillow
300
TO0
0.67 0.14
2,20 0.30
1.42
1186
0.72
0.07
3:10
3009
25 2.05
1.56
0.72
0.68
0.222.4 0.26
0.31
0.16 0.30
0.15 0.24
1000
0.19 + D.0S
1:11 + 1306
1.006
00088
2,34 1.36
0.63
0.18 + 1.22
0.11 + 0.22
5:04 1,86
3,30
1.65
1.97
2.00
0.95 0.71
Pot
100
2.43 1: 0.88
1,8 1: 0.7.2
1.82 2: 2.66
2:37 1.60
1.72
1.32
1,26 1.12
0.69 1: 0.61
3001
0.78 9 0.27
0.36 0.33
1,38 = 1.09
2:07 1.36
1.03
0.87 4 0.34
0.39 = 0.56
0.24 0.09
1,30
1:99
1.32
0.98
0.62
50
6.01 1 1.68
4.72 2.00
3.29 1: 2.77
3.46 2: 1.11
1,86 1 1.27
1,89 1 0.92
1.35 = 0.90
Printer
100
2.97 1 0.87
1.39 1.00
0.09 = 0.47
3:31 1.54
1.61
1.29 1999111 1:05
0.79 1 0.33
0.26
0.24
1.38
0.39
1000
0.27 + 0.07
0.22 + 0.10
0.24 0:23
3308 =: 1.28
1:30
0.44 0.36
0.58 = D:45
0.91 0.49
1:00
091 3.62
2.66
4.2%
0.65
0.33
Rifle
300
100
0.12 4 0.07
0.10 1 0.22
0,10
0.29 0.19
1: 0.07
0.11 0.15
0.07 1 0.05
7.74
7.74 2: 2.56
a: 2:37
3:13
2.70
1: 2.97
0.33 0.37
0.24 0.28
0.17 =: 0.31
0.20 =: 0.24
1000
0.05 0.02
0.07 0.06
0.08
a 0.05
7.73 a 2.51
2,43 2.90
0.22 0.28
0.18 0.30
50
US32 4 0.61
1.99 0.76
0.41 2 0.19
6.77 2,50
4,22 2.322-00 4.61
0.34 0.36
0.29 = 0:23
Rocket
300
TOU
0.16 0:07
0.39--333 1.32
0:09
0.10
0.13 - 0.10
6:97 = 1.85
1.72
4.75 1775--00 3.76
1134 1:00
0.25 + 0:51
1000
0.03 0.02
0.09 1 0.07
0.08
2 0.06
8.11 22 2.59
3.30
0.25 1 0.19
0.22 1: 0.27
50
4,87 1.29
2,77 1.29
3.53 3.29
5.45 1.91
2.74 3.08
1,22 0.84
0.59 0.45
Sofa
100
2,45
0.64
0.44
5.40
2004
2.57
0.26
500
0.79 4 0:20
1:41 1.15
0.17 = 038
5:24
2.00
2,31 5.64
0.24 1.245
0.17 0:21
1000
0.23 1 0.05
0.13 0.07
0.13 1 0.26
3.14
2: 2,14
2,52
4 3.55
0.20 1 0.25
0.17 2
30
5.81 1. 2.31
4.10 2.08
10.01
2 8.9
3.87
2 2.55
1.27 2.56
2.41 2:70
1.08 1 0.64
100
2,88 + 1.19
LX3 + 0.97
2,53 + 6.8
2.96 1.98
1.66 2.55
0.988 0.78
0.59 + 0.53
Scove
300
0189 0:35
0.116
1.00
2.25
0.49
0.37 0.39
0.26 0.10
0.19
1.83
0.40
50
4,43 1: 1.65
2.47 1 1.49
5.35 = 5.29
7.11 4.14
6.78 3: 5.05
1.66 1: 5.01
0.78 1.05
Table
300
100
0.60 + 0.23
1.99 0.72
0.29
0.65
0.19
0.61 =: 2.18
1.60
6:41 + 3.73
3.98
6.25
4.92
0.44 + 1.60
1.40
0.32 + 1.15
0.70
1000
0,18 0.07
0.15 3 0.13
0,40 1.44
6.21
d: 3.69
6.19 1.19:324 4.88
0.42 3.71
0.31 3: 1.00
50
3-13 4 0.85
1.79 0.77
19.31
10.40
5.92 1: 3.27
0.73 1: 1.67
0,57.1 1.02
0.23 =: 0.30
Telephone
100
1.75 0.41
6.97
1 0.38
3,86 2 7.16
4.30
2: 2.75
0.72 1 1.20
0.19:1 0.29
0.11 2: 0.18
300
0.57 4 0.12
0.19 0.10
0.40 3.23
3.03
2.26
0.62 1.42
0.12 0.24
0.09 0,16
(.17 DIDS
0,30
2006
0063
0.10
50
3,51 + 2.05
3A02 2.133
11.13 = 13.40
5:70 d: 4.04
2.36 +: 2.15
1.61 1:25
0.92 # 0.82
Tower
300
100
0.50 0.3)
1.71 0.97
0.30
2.07
0.19 0.25
3.64 2.87
3.32
1,60
0.30 0.S2
0.45 0:46
0.78
1000
0.15 1 0.08
0.20 1: 0.18
0.21 4 0.29
3.88 4: 3.57
1.47 4 1.64
0.46 0.47
0.44 4 0.53
2.31 1.09
2.29
2.15
3.08
0.37 0.50
Train
100
1.7 0.53
TIKU--USS
1121 - 0.20
3:85 4: 1.31
3,38 03360-00 2.55
0.39 2 0:45
0.23 = 0.18
300
0.36 0.17
0.26 0.14
0.09 1 0.08
3.63
2: 2.04
3.24 1 2415
0.23 0.21
0.22 1 0.20
0.05
0.09
0.09
2.83
50
2.07 1 0.93
L61 0.93
5.27 2 8.20
4.99
2 2.18
3,23
1.03 2 1.88
0.54 1 0.51
Vessel
100
1:00 0.45
1385 0.48
1.38 4.73
5:05
2,14
2388
0.688 0.39
01.3 0.37
X00
0.31 4.15
11.29
0.62
247
4:00
2002
2.75
4
0.33
1000
0.10 1 0.04
0.18 0.21
0.16 1.42
5:21 2
2,55
1.93
0.30.1 0.75
0.33 0.37
1.32
7.44
1.10
0.64
Wisher
100
300
4.02 + 0.42
1.27 4 0.16
0.49 1 0.13
2.66 0.26
0.11 = 0.10
1:38
2.15
2.15 0.71
1.72 2 0.40
0.32
0.36
1 0:41
0.44
0.25 1.25:100 0.26
1,26 0387
0.49 = 0.56
0.97 + 0.66
U.36 0.05
0.09
0.25
0.35
0.18....1
0.46 0.49
14

---

Under review as a conference paper at ICLR 2021
Table 6: Results for each ShapeNet novel category
Category
Context
Gauss densiliation
PCN
UkeNeer
DEspSTE
TGK
MIC (W7 opu
1.29
1 1.49
5.11
1.38
2448
Bag
100
300
2.21 1: 0.59
0.67 + 0.21
0.64 + 0.32
1.79 0.62
0.85 4: 2.88
0.24 0.21
4.02 4: 2.14
3.52 + 1,80
1.54 1.39
1.37 + 1.30
0.86 + 1,46
1.37 2 1.37
0.57 0.40
0.74 0.51
0,06
0.23
1.38 1.27
0.65 0.49
50
5.35 1: 1.52
3,79 +: 1.56
1.64 6.41
1.26:22..5
3.72 4: 2.99
2.28 1.94
1.33 0.94
Red
T00
2,69 3 0.74
L.3 3 0.69
1.51 4.69
4.13 00 2.08
3.32 2 2.89
1.51 1.40
0.99 0.86
0.26
1.06
4.06
3,00
1.07
0.91
1000
0.26 0008
0.51 0.34
0.05 0.34
3.97 2.22
3.05 + 2.93
1,02 0.80
0322 0.84
1.18
1 1.02
2.72
Hench
300
100
0.40 0.18
1.38 1 0.57
0.27 0.16
0.78 0.16
0.39 1.36
1,97 4.33
9.19 3.93
0.142-4.00
3.76 2: 2.39
3.57 2: 2,63
0.71 2: 1.50
0.43 = 0.48
0.36 0.41
0.42 0.49
1000
0.12 0.05
0.17 4: 0.13
0.26 0.65
9.09 4.10
3.78 a 2.54
0.42 0.86
0:35 0.40
30
5.00 1.82
3,09 +: 1.88
3,83 3.330
3.99
5.32
2.30 =: 1.75
2.13 1.17
1.27 0.68
Birdhouse
T00
2,52 + 0330
190 0.82
0393 2. 2.7
3.76 3.49
2,08 + 1.55
1:24 0.74
0.69 0.48
1.35
0.68
1000
0.24 0.08
0.29 0.18
0.21 0.26
3.25
1.92 1.49
0.60 0.41
0.33 0.39
30
4,91
1.64
191 1 1.44
11.14
8.91
1.36
2.67
1.95 1 2.89
1,92 1.37
1.06 0.56
Bookshelf
1.50 0.67
3:58
1.52 2.65
1:16
300
0.78 + 1:20
0000
0.32
0.55 1.25
2.80 2.44
1.78 2.71
0.88 + 0.3%
1000 0.46
1000
0,24 1 0,08
0.42 1
0:47 0.33
2.65 2.25
1.76 2 2.70
0.81 =: 0.52
0.00 1:0.0
30
3.69 1. 1.04
3.30 1.12
3.81 1: 4.88
6.06 2: 1.76
2.30 1.62
0.86 2: 0.69
0.53 0.40
Bus
100
300
0.55
1.79 + 0.50
0.15
0.51 2.133
1.88 + 0.55
0.10 0.18
0:41 + 1.18
6.01 + 1.04
5,86
1.93 1.50
1.28
0.38 + 0.32
0.18 0.22
0.30 0.28
0.24
0.16
0.08
0.13
5.66
1:33 1:21
0.13
50
3.61 2 0.78
3.91
6.01 4:00
535
3 1.89
2.31 1.06
1.24 0.62
Cap
100
300
0.49 4 0.099
0.47 + 0.13
0.64
3.32 3.16
3.77
3.69 0.98
3.08
2.94 1.76
1.71
1.58 0.27
0.67 0.54
1000
0.14 : 0000
0.28 0.10
1.39 1:29::22 2.35
3.65 =: 1.07
2.90 1.91
1.52 0.77
0.52 1: 0.38
0.00
1.0% 0.72
21,36
6.04
1:27 0.87
2.53+ 16.75
Keyhoard
100
1.11 0.30
0.47 0.31
6,00
10.78
6.8222882
1.08 0.90
0.56 1.64
0.17 2 0.25
300
0.38 1 0.30
0.24 1: 0.24
1.47 1 6.24
7.55 2.84
1.02 a: 0.79
0.43 1.27
0.19 0.36
0.12 0003
0.26
9.05
1.02 0.85
0.56
50
2,75 4: 1.05
2.50 + 1.15
9,06 13.27
6.08 3.39
7:005 :: 4.08
8,05 9.19
2.30 2,40
Earphone
300
T0D
0.40 0.19
0.33
133 0.85
2.71 9:27
12.48
5:28 22 3.36
6.61 4.00
4.19
3.54 11.01
4,7
1.49 1.50
1000
0.12 1: 0.06
1.42 4. 0.83
2.12 7.08
5.30 a: 3.47
6.54 a: 4.50
3.99 :: 7.08
1.34 1.45
0.42
19:57
11.76
5.88 =: 2.25
Guitar
100
0.42 0.18
0.25 :: 0.13
25.87
22.62
10:92 01922:220 3.90
3.78 2.16
1.05 5.93
0.90 0.36
300
0,13 1 0.05
0.13 1 0:05
6.30
13.83
9,36 3.97
3,70 2: 2001
0.55 0.99
0.27 0.30
0.11::0:0
9:23
9.31
0.43
30
1.90 1.19
1.2 1:20
13.33 03333.2200 22.51
7.32 10.00
11.10
10.55
15.13
3.91 6.55
1.36 2,36
Microphone
T00
W.76 0.35
1.4 0.90
16.34
9.70
14:00
3399 7.72
0194
300
0.22 0.16
2.13
6.77
9.81
9.92
14.17
2.00 7.11
1000
0.07 1 0.04
0.64 0.91
1.51 2
9.61 7.55
9.8
14.01
1.72:2..5
0.92 1.67
1.00
1.33
3.09
Motoreyele
100
1.21 0.30
1.03 1: 0.38
0.39 0.16
5.02 2: 1.15
2.08 2: 1.23
1.22 1: 0.60
0.81 0.38
300
0.40
4 0.10
0.52 0.19
0.33 0.14
4:98 = 1.08
1.92 1.12
1.03 0.49
0.72 0.38
0.13
4.79
1,11
50
1.93 1 0.78
1.77 0.61
0:43:200:0
10.19 2.98
7.43 7.01
0.59 (139:200 0.44
0.32 0.24
100
0.95 1 0.38
0.37 1 0.25
0.14 1 0.17
10.10
0:00=1: 2.84
6.35 1: 3.36
0.38 1: 0.28
0.19 0.20
0.06
9.77
2.59
3.53
1000
0.09 =: 0.003
0.12 0.07
0.133 0.07
2.67 9.76
3:78 3.47
0.25 =: 0.25
0.14 0.14
2390
1.08
1XT 0.94
16.70
14.58
6.55 3.74
1.68 1.68
0.58 0.65
0.30
Remote Control
100
1,48 1 0.50
0.33 0.110
0.98:23
5.30 3.44
1.52 1.57
0.21 0.26
0.13 0.17
300
0.46 1 0.16
0.19
0.06 0.02
4,13 2:.02.200 2.93
1.57 1.58
0.10 = 0.09
0.08 0.14
0.14
0.04
0.06 0.05
3.91
1.59
0.09
50
1.51 4 0.46
0.89 0.41
8.69 1.699:111 12.50
6.34 3.37
4.76 2.59
0.4 2 0.97
0.39 0.24
Skcateboard
100
0.70 + 1223
11443 + 0.24
1.36 6.12
6:03 1.12
4.27 2.57
1138 + 3.22
0.29 1.29-:0 0.28
300
0008
0.18 0.10
2 3.14
6.40
3.87 2.21
0.41
0.23:0.33
1000
0.08 1: 0.02
0.14 1: 0.09
0.31 1.71
7.13 a. 3.69
3.64 1: 1.95
0.45 1: 0.33
0.21 1: 0.19
15

---

Under review as a conference paper at ICLR 2021
PCN
Dechet
DeetO
West
Ocunctuth
Figure 7: Shape completion results ShapeNet for Basket (training category). Our method starts conservatively and improves as more points are added (it also correctly infers that there is no lid).
Ropeoed
Prepoct
PCN
Figure 8: Shape completion results ShapeNet for Bathtub (training category). Our proposed method captures the large cavity with only 100 observations, while both DeepSDF and OccNet produce artifacts at this level of sparsity.
16

---

Under review as a conference paper at ICLR 2021
FCK
Ouzhet
DespS
Poppaed Nuthad
Prepoued Merved
Goundtuch
Figure Shape completion results on ShapeNet for Car (training category). Our method produces smooth car-like surface with only 50 observations, while both Dee and OccNet generate large artifacts this level of sparsity. Additionally, PCN fails produce car-like point cloud from only 50 observations.
Insu
PCN
Dctlet
Deestor
Prpessd Nathed
Prepoued Merved W'opt
Ocundiuth
Shape completion results ShapeNet for Lamp (training category). All methods struggle with predictions for this category, irrespective of the number of observations. Particularly, IGR consistently assumes that the target object is rectangular, even at higher density levels. At these levels, our method succeeds in capturing the approximate geometry of the lamp.
Figure 10:
17

---

Under review as a conference paper at ICLR 2021
Poppeee Nethad
Prepored Harred
Figure 1I:
Shape completion results on ShapeNet for Mug (training category). Our method is the only one that succeeds in capturing the cavity of the mug from sparse observations, while other methods assume there is lid. As expected, PCN only produces what looks like a densified version of the original point cloud.
Nathod
Prepaued
Figure 12: Shape completion results on ShapeNet for Table (training category). Our method correctly captures the rectangular portion of the table when only 50 observations available, and gradually completes other parts as the number of observations increase. IGR, on the other hand, assumes that the object box-shaped at lower densities and maintains this assumption even higher density levels.
18

---

Under review as a conference paper at ICLR 2021
FCN
Duzhet
DeapSof
Poppaed Nathod
Propoued lerved
Goundirch
Figure 13: Shape completion results on ShapeNet for Bed (novel category). Our proposed method correctly predicts the overall box shape of the object, and gradually improves this prediction as more observations points become available. Other methods, such as OceNet and IGR, fail to separate the top and lower portions of the object even at higher density levels.
Inuu
PCN
Octlet
Deestor
Prpesed Nuthod
Preposed Merved W'ogt
Ocuedluth
Figure 14:
Shape completion results on ShapeNet for Bed (novel category). With only 50 points our proposed method is already capable of predicting the correct shape, that is then further refined with more observations. OccNet requires higher number of observations before settles on the correct shape, while IGR converges to the wrong object.
19

---

Under review as a conference paper at ICLR 2021
Prppored Nuttod
WESSOIN Perved
3000
Figure 15: Shape completion results on ShapeNet for Guitar (novel category). Here, PCN the only method that seems succeed shape completion, mostly because of the compact nature of the object that enables accurate densification. Interestingly, as the number of observations increase, our proposed method performs better without post-encoder optimization, producing reasonable output with 300 observations.
1O2 poins
PCN
Octhat
DeASGOF
GR
Ropeoed Nothll AS
Prepoct wist
Figure 16: Shape completion results ShapeNet for Motorcycle (novel category). Similar the guitar category (Figure 10), our proposed method performs better without post-encoder optimization. We attribute this smoothing effect generated by this step, that enables better generalization the expense of finer details.
20

---

Under review as a conference paper at ICLR 2021
Prppeeed Nuttod
Prepored Parred
Figure 17: Shape completion results on ShapeNet for Pistol (novel category). Interestingly, IGR seems to change the assumed category the number of observation points increase. Again, due to the presence of finer details our method performs better without post-encoder optimization, achieving reasonable results with as few as 100 observations.
Nathod
Prepaued
Figure 18: ICL-NUIM shape completion results for office rooms. With as few as 50 points, our proposed method is able recover most details of the room, including furniture arrangement, wall distribution and floor. As more observations are introduced, these details are further refined without diverging. PCN results are mostly densification of the observed points, IGR requires higher densities for proper convergence, and DeepSDF produces a large amount of artifacts throughout the scene.
21

---

Under review as a conference paper at ICLR 2021
Prppand Nathed
Prepored
Figure 19: ICL-NUIM shape completion results for office rooms. With as few as 50 points, our proposed method is able to recover most details of the room, including furniture arrangement, wall distribution and floor. As more observations are introduced, these details are further refined without diverging. PCN results mostly densification of the observed points, IGR requires higher densities for proper convergence, and DeepSDF produces a large amount of artifacts throughout the scene.
Popesee Nathed
Prepored Mereed
Figure 20: ICL-NUIM shape completion results for living rooms. With as few as 50 points, our proposed method is able to recover most details of the room, including furniture arrangement, wall distribution and floor. As more observations are introduced, these details are further refined without diverging. PCN results mostly densification of the observed points, IGR requires higher densities for a proper convergence, and DeepSDF produces a large amount of artifacts throughout the scene.
22


Reviews:
[{"review_id": "b1Vd19hVzjU", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a meta-learning approach for the neural implicit representation of 3D shapes. The main idea, in my understanding, is to consider the points in the input point cloud as few-shot examples of the object so that each of them can be encoded in a way to best approximate the entire object information. The experiments show that the network can reconstruct the entire shape well even with a very small number of the input points, such as 50 and 100. For better reconstruction, the authors also proposed to use some implicit function regularizations, which are introduced in a previous work (Gropp et al., 2020).\n\n\n*** Strengths ***\nTo my knowledge, this idea of using the meta-learning technique in the neural implicit representation is novel, except for the following concurrent work (I hope the authors discuss this work in the revision):\nSitzmann et al., MetaSDF: Meta-learning Signed Distance Functions, arXiv:2006.09662.\n\nAlso, the experiments show impressive results both quantitatively and qualitatively. It’s interesting to see that the proposed method generates a plausible output even with 50 input points.\n\n\n*** Weaknesses ***\nAll the formulations directly come from previous work; Maeda et al., 2020 for the meta-learning formulation, and Gropp et al., 2020 for the implicit function regularizers. Although, I think it’s still worth showing how the meta-learning technique can be applied in this implicit function learning problem.\n\nThere are many recent works about 3D shape completion and point cloud upsampling, but the authors only referred to and compared with PCN (Yuan et al., 2018): To name a few:\n\n[Completion]\nHuang et al., PF-Net: Point Fractal Network for 3D Point Cloud Completion, CVPR 2020.\nLiu et al., Morphing and Sampling Network for Dense Point Cloud Completion, AAAI 2020\n\n[Upsampling]\nYu et al., PU-Net: Point Cloud Upsampling Network, CVPR 2018\n(This work has 120 citations, meaning that there are many other recent works about deep upsampling).\nLi et al., PU-GAN: a Point Cloud Upsampling Adversarial Network, ICCV 2019.\n\n\n*** Justification ***\nI like this idea of using meta-learning in shape completion/upsampling. But the lack of literature review on this problem and comparisons looks like a big weak point. I hope the authors address this concern in the revision.\n\n\n*** Discussion / Comments ***\nI guess the proposed idea can be applied to the other types of input data. For instance, one can consider a multi-view or sketch-based 3D reconstruction network which, however, can still reconstruct the shape well even with a single image or a few strokes.\n\nIn Figure 1, ShapeNet40 -> ShapeNet?\nIt would be clearer to show one encoder with variable ‘m’ for the input dimension of the last MLP layer and mention in the caption that the ‘m’ is 512 for ShapeNet and 256 for ICL-NUIM.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2UvmIuzcdp", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a way of reconstructing a surface from sparse point clouds via a \"meta learning\" approach. Specifically, the authors view each shape in a collection as a \"domain\", and predicting the SDF values of points in R^3 to reconstruct a given shape (the reconstructed surface is the isosurface of the SDF field) as the \"task\" for that domain. Then, they use a network to predict a distribution over \"task-specific\" latent vectors that characterizes the reconstruction task for a given shape. Given a latent-vector sampled for one shape, they pass it to a decoder that predicts the SDF value at any point in R^3 for that shape.\n\nThe basic decoder setup follows that of IGR. Specifically, by adding Eikonal (and normal vector) regularization, the authors can train purely on point clouds, without ground truth surfaces or SDF values. (This feature, which is clearly stated in the IGR paper, is quite obfuscated in this paper and should be stated much more clearly, though it is not a contribution of the current work per se.)\n\nI am both intrigued and confused by the meta-learning aspect of this work. While I agree the formal mathematical structure of meta-learning applies to the interpretation that each shape is a task/domain, it seems quite non-standard. I would have expected, say, a collection of shapes from one category to be a \"domain\", or the tasks to be much less related e.g. one task could be reconstruction and another could be classification. Further, given the \"each-shape-is-a-domain\" interpretation, the final network architecture ends up looking suspiciously like a VAE, where the encoder (say with a PointNet or similar architecture) takes a point cloud of arbitrary size as input and outputs a local gaussian distribution over latent vectors, and an implicit field decoder that acts exactly as in this paper, taking as input a latent vector sampled from the encoder output. Any difference from a standard-issue VAE, then, boils down to the exact way the encoder's gaussian is computed (and possibly also the training loss, but here the usual ELBO loss appears to be used). In this paper, Eq (2) describes the gaussian computation, drawing from prior work [Maeda et al. 2020].\n\nI am certainly not an expert on meta-learning of any sort, so I cannot comment on the advantages of the proposed approach over a standard VAE encoder. However, the similarities do seem to warrant an explicit comparison, where the overall architecture is as similar as possible (the current encoder with parallel MLPs with shared weights followed by an aggregator appears very similar    to PointNet, so something on those lines could be used), but the encoder explicitly outputs the \\mu and \\sigma of a gaussian. Indeed, OccNet mentions a generative variant along these lines. I suggest posing this as an ablation study of the proposed method, though, to eliminate any bias due to other architectural differences.\n\nThe following paper published at the same time as OccNet is also closely related (generative implicit field model) and should be cited and possibly compared to:\n\nChen and Zhang, \"Learning implicit fields for generative shape modeling\", CVPR 2019\n\nThe proposed method also allows sampling several completion candidates, since the encoder outputs a distribution. But such sampling does not seem to be demonstrated anywhere in the paper. What would happen if one only passed the distribution mean to the decoder? What do different candidates look like?\n\nTo summarize, the proposed method involves several design decisions that distinguish it from prior work. Some of these are novel though somewhat obfuscated, e.g. the \"meta learning\" solution; some have been used elsewhere, e.g. learning from point clouds only; and some are not relevant to the stated contributions, e.g. architectural differences other than the bare minimum to get the meta learning formulation to work. I request the authors to more clearly state and justify their contributions, especially via the ablation suggested above.\n\n=======================================================\n\nUPDATE: I thank the authors for their responses to our questions. However, I still cannot recommend acceptance, because of the issue both R3 and I mentioned: the comparison to a more traditional VAE-style encoder, which replaces the \"aggregator\" of this paper with max-pooling plus some number of FC layers to output a mean and variance (as I mentioned earlier, this is almost exactly PointNet with the last layer outputting mean+variance instead of a single feature vector). I definitely do not buy the assertion that VAEs cannot support variable sized point contexts. Just about any shape encoder can be converted to a VAE encoder by modifying the last layer (and training with an appropriate decoder and loss) -- this includes encoders that handle point clouds of varying size like PointNet. In the exchange with the authors, it was not clear that this point was fully appreciated. Hence, while I appreciate the many interesting aspects of this paper, and the additional results provided by the authors during the discussion session, I remain negative and am slightly lowering my rating to make this clear (if this was a journal submission I would mark it as \"major revisions\" and ask for additional experiments vs VAE baselines which share the bulk of the proposed architecture other than the different aggregator).", "extra_scores": {}, "extra_text": {}}, {"review_id": "-cCpKHZfCLh", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the task of point cloud completion, aiming to infer an implicit occupancy representation given a sparse input point cloud. Following recent practices, the approach represents the shape via a latent-variable conditioned occupancy function $f_{\\phi}(x, h)$ that infers occupancy of a point $x$ given latent $h$. The central task addressed here is to be able to infer a posterior distribution over the latent variable given some observed points $D$ i.e. $p_{\\theta}(h|D)$.\n\nFollowing Maeda etal., this paper uses a specific form of this distribution and trains $(\\phi,\\theta)$ by optimizing the loss on held out surface points and using additional regularization. In particular, each data-point $x_n$ infers a gaussian distribution over $h$ parametrized by $f_n, g_n$ and these are aggregated to get the full posterior. This compositional nature of prediction (each data point infers a distribution over $h$) in particular helps the approach robustly scale to use more/less points during inference.\n\n**Strengths**\n- A requirement for the posterior distribution is that given more point samples, the shape should ideally become more refined and closer to the true shape. The empirical and qualitative results clearly show that this is a property of the proposed approach.\n\n- The empirical results reported are encouraging, and in particular the proposed approach seems to yield more accurate reconstructions compared to vanilla encoders used in prior approaches e.g. OccNet/PCN (although I have some concerns regarding this).\n\n- The paper presents results on Object and Scene reconstruction datasets and shows the ability to recover shapes from just a few (10) to many (1000) samples. \n\n\n**Concerns/Comments**\n\n- While this approach uses a specific factored form of $p_{\\theta}(h|D)$, the more 'typical' way is learn a non-factored encoder (e.g. a PointNet) that directly predicts a variational distribution $q_{\\theta}(h|D)$. Two of the baselines reported (PCN, OccNet) do use this approach of using an encoder, but I am concerned the comparison is unfair.\nIn particular, these baselines are trained using a fixed number of input points in set $D$ (around 5000), but tested using a different number of input points (from 10 to 1000) that are far fewer than the size of $D$ seen in training. Such a discrepancy between training and testing settings can obviously adversely affect the results and might explain why methods like OccNet perform so badly. This is not as much of a concern for the proposed method because of the factored and per-datapoint prediction of $h$. I would therefore really like to see results with the encoders of these baselines trained using a variable number of input points, similar to what would be used at testing. In the current form, the results merely show that encoders trained using many points work poorly when using fewer points for inference.\n\n\n- The proposed approach parametrizes $p(h|x_n)$ as a gaussian for tractability. I would assume this is a limiting choice (e.g. given a single point, say on a chair back, the possible shape space is arguably multi-modal). I would appreciate some discussion on why the method still works despite this choice.\n\n- Despite being novel in context of shape completion, I am slightly concerned regarding the technical novelty as the overall approach is a direct application of the model proposed in Maeda et. al. That said, the insight that implicit occupancy prediction from sparse point clouds can be viewed as a meta-learning task is still interesting and I might still be in favor of acceptance if all the empirical concerns were addressed.\n\n\n- (minor concern) I don't think the normalized metrics add much to the evaluation. I would instead encourage the authors to report alternate metrics such as F-score.\n\n----\nOverall, while I think the approach is interesting and novel in context of shape completion, I cannot recommend acceptance until the concerns regarding the fair comparisons to baseline methods are addressed.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission generated a lot of discussion.

The main strengths of the paper:
* It is an interesting application of meta-learning (to 3D shape completion), and a novel one.
* It appears to work well: it is remarkable that the proposed model can reconstruct shapes as well as it does given a point cloud with only 50 input points.

The main concern (raised by two reviewers) is that while the method is described using the language of meta-learning, the proposed architecture ends up looking very similar to a variational autoencoder: a point cloud encoder that outputs some distribution in a latent space, which is then sampled from to produce a code which drives an implicit surface decoder. The only difference appears to be that the proposed method uses a factored distribution in latent space, whereas a traditional VAE uses a non-factored one (i.e. a single multivariate Gaussian over all dimensions of the latent code).

One reviewer engaged the authors in a discussion about this point, but the resulting conversation was not satisfactory. One interpretation of the authors' response is that they are simply not aware of how a VAE could be trained using variable-sized point clouds as input (which is quite possible using many standard point cloud processing networks). However, at other points, they do seem to grasp this, when they write that "This flexible representation of the uncertainty [i.e. the one proposed by the authors] cannot be attained by the mere average or max pooling aggregation [what a PointNet encoder would do]." They even go on to provide an additional ablation study where they replace their factored probabilistic encoder with a deterministic mean/max pool encoder, and show worse results. Unfortunately, they never compare against *probabilistic* variants of such encoders (i.e. where the mean/max pool output is then used to compute a mean and variance).

Without seeing this comparison, the reviewers believe that this paper cannot be accepted, and I am inclined to agree.

On a related note: one reviewer pointed out an issue with unfair comparisons, in that baselines were trained on high density point clouds and evaluated on low density ones. The reviewer noted that these methods could be (and should have been) trained on point clouds of varying density. Perhaps this relates to my hypothesis that the authors initially did not understand that training such encoders on variable-sized point clouds was possible. In any event, in their rebuttals, they have reported some preliminary results from experiments which do this type of training, but these results are not conclusive. Complete, conclusive results from these experiments would also need to be presented before this paper could be accepted.
Decision: Reject