Year: 2021

Abstract:
We propose a novel 3d shape representation for 3d shape reconstruction from a single image. Rather than predicting a shape directly, we train a network to generate a training set which will be fed into another learning algorithm to define the shape. The nested optimization problem can be modeled by bi-level optimization. Specifically, the algorithms for bi-level optimization are also being used in meta learning approaches for few-shot learning. Our framework establishes a link between 3D shape analysis and few-shot learning. We combine training data generating networks with bi-level optimization algorithms to obtain a complete framework for which all components can be jointly trained. We improve upon recent work on standard benchmarks for 3d shape reconstruction.

heading: 1 INTRODUCTION
text: Neural networks have shown promising results for shape reconstruction (Wang et al., 2018; Groueix et al., 2018; Mescheder et al., 2019; Genova et al., 2019). Different from the image domain, there is no universally agreed upon way to represent 3d shapes. There exist many explicit and implicit representations. Explicit representations include point clouds (Qi et al., 2017a;b; Lin et al., 2017), grids (Wu et al., 2015; Choy et al., 2016; Riegler et al., 2017; Wang et al., 2017; Tatarchenko et al., 2017), and meshes (Wang et al., 2018; Hanocka et al., 2019). Implicit representations (Mescheder et al., 2019; Michalkiewicz et al., 2019; Park et al., 2019) define shapes as iso-surfaces of functions. Both types of representations are important as they have different advantages.
In this work, we set out to investigate how meta-learning can be used to learn shape representations. To employ meta-learning, we need to split the learning framework into two (or more) coupled learning problems. There are multiple design choices for exploring this idea. One related solution is to use a hypernetwork as the first learning algorithm that produces the weights for a second network. This approach (Littwin & Wolf, 2019), is very inspiring, but the results can still be improved by our work. The value of our work are not mainly the state-of-the-art results, but the new framework that enables the flow of new ideas, techniques, and methods developed in the context of few-shot learning and meta-optimization to be applied to shape reconstruction.
In order to derive our proposed solution, we draw inspiration from few-shot learning. The formulation of few-shot learning we use here is similar to Lee et al. (2019); Bertinetto et al. (2019), but also other formulations of few shot learning exist, e.g., MAML (Finn et al., 2017). In few-shot learning for image classification, the learning problem is split into multiple tasks (See Fig. 1). For each task, the learning framework has to learn a decision boundary to distinguish between a smaller set of classes. In order to do so, a first network is used to embed images into a high-dimensional feature space. A second learner (often called base learner in the literature of meta learning approaches for few-shot learning) is then used to learn decision boundaries in this high-dimensional feature space. A key point is that each task has separate decision boundaries. In order to build a bridge from fewshot learning to shape representations, we identify each shape as a separate task. For each task, there are two classes to distinguish: inside and outside. The link might seem a bit unintuitive at first, because in the image classification case, each task is defined by multiple images already divided into multiple classes. However, in our case, a task is only associated with a single image. We therefore introduce an additional component, a training data generating network, to build the bridge to fewshot learning (See Fig. 2). This training data generating network takes a single image as input and outputs multiple 3D points with an inside and outside label. These points define the training dataset
for one task (shape). We adopt this data set format to describe a task, because points are a natural shape representation. Similar to the image classification setting, we also employ an embedding network that maps the original training dataset (a set of 3D points) to another space where it is easier to find a decision boundary (a distorted 3D space). In contrast to the image classification setting, the input and output spaces of the embedding network have a lot fewer dimensions, i.e. only three dimensions. Next, we have to tackle the base learner that takes a set of points in embedding space as input. Here, we can draw from multiple options. In particular, we experimented with kernel SVMs and kernel ridge regression. As both give similar results (In meta learning (Lee et al., 2019) we can also find the same conclusion), we opted for ridge regression in our solution due to the much faster processing time. The goal of ridge regression is to learn a decision boundary (shape surface) in 3D space describing the shape.
Our intermediate shape representation is a set of points, but these points cannot be chosen arbitrarily. The set of points are specifically generated so that the second learning algorithm can process them to find a decision boundary. The second learning algorithm functions similar to other implicit shape representations discussed above. The result of the second learning algorithm (SVM or ridge regression) can be queried as an implicit function. Still the second learning algorithm cannot be replaced by other existing networks employed in previous work. Therefore, among existing works, only our representation can directly use few-shot learning. It does not make sense to train our representation without a few-shot learning framework and no other representation can be directly trained with a few-shot learning framework. Therefore, all components in our framework are intrinsically linked and were designed together.

heading: Contributions.
text: • We propose a new type of shape representation, where we train a network to output a training set (a set of labeled points) for another machine learning algorithm (Kernel Ridge Regression or Kernel-SVM).
• We found an elegant way to map the problem of shape reconstruction from a single image to the problem of few-shot classification by introducing a network to generate training data. We combined training data generating networks with two meta learning approaches (R2D2 (Bertinetto et al., 2019) and MetaOptNet (Lee et al., 2019)) in our framework. Our work enables the application of ideas and techniques developed in the few-shot learning literature to the problem of shape reconstruction.
• We validate our model using the problem of 3d shape reconstruction from a single image and improve upon the state of the art.

heading: 2 RELATED WORK
text: Neural implicit shape representations. We can identify two major categories of shape representations: explicit representations, where a shape can be explicitly defined; and implicit representations, where a shape can be defined as iso-surface of a function (signed distance function or indicator function). In the past decade, we have seen great success with neural network based explicit shape representations: voxel representations (Wu et al., 2015; Choy et al., 2016; Riegler et al., 2017; Wang et al., 2017; Tatarchenko et al., 2017), point representations (Qi et al., 2017a;b; Fan et al., 2017; Lin et al., 2017), and mesh representations (Wang et al., 2018; Hanocka et al., 2019)). On the other hand, modeling implicit representations with neural networks has been a current trend, where usually a signed distance function or indicator function is parameterized by a neural network (Mescheder et al., 2019; Chen & Zhang, 2019; Michalkiewicz et al., 2019; Park et al., 2019). More recent works learn a network that outputs intermediate parameters, e.g. CvxNet (Deng et al., 2019) and BSP-Net (Chen et al., 2019) learns to output half-spaces. We propose a novel type of shape representation, where the model outputs a training set of labeled points.
Few-shot learning. There are two common meta learning approaches for few-shot learning: metric-based (Koch et al., 2015; Vinyals et al., 2016; Snell et al., 2017), which aims to learn a metric for each task; optimization-based (Ravi & Larochelle, 2017; Finn et al., 2017; Nichol et al., 2018), which is designed to learn with a few training samples by adjusting optimization algorithms. These approaches commonly have two parts, an embedding model for mapping an input to an embedding space, and a base learner for prediction. Qiao et al. (2018); Gidaris & Komodakis (2018) train a hypernetwork (Ha et al., 2016) to output weights of another network (base learner). R2D2 (Bertinetto et al., 2019) showed that using a light-weight and differentiable base learner (e.g. ridge regression) leads to better results. To further develop the idea, MetaOptNet (Lee et al., 2019) used multi-class support vector machines (Crammer & Singer, 2001) as base learner and incorporated differentiable optimization (Amos & Kolter, 2017; Gould et al., 2016) into the framework. Lee et al. (2019) also shows it can outperform hypernetwork-based methods. In our work, we propose a shape representation that is compatible with a few-shot classification framework so that we can utilize existing meta learning approaches. Specifically, we will use ridge regression and SVM as the base learner. The most relevant method to ours is Littwin & Wolf (2019) which adapts hypernetworks. However, as we discussed above, differentiable optimization methods (Bertinetto et al., 2019; Lee et al., 2019) are generally better than hypernetworks. Besides that, meta learning has been applied to other fields in shape analysis, e.g., both Sitzmann et al. (2020) and Tancik et al. (2021) propose to use MAML-like algorithms (Finn et al., 2017) to learn a weight initialization over an implicit shape spaces.

heading: 3 METHOD
text: The framework is shown in Fig. 2. The network is mainly composed of 3 sub-networks. The Feature Network maps an input image to feature space. The resulting feature vector λ is then decoded by the Point Generation Network to a labeled point set {xi, yi}Ni=1. The point set will be used as training data in our base learner later. After that the Embedding Network projects the point set into embedding space. The projected points ei and the labels are taken as the input of a binary classifer (ridge regression or SVM) parameterized by α. Finally, the framework is able to output the inside/outside label y of a query point x by projecting it into the embedding space and feeding it to the binary classifier.
In the following subsections, we describe our method in more detail. First, we introduce the background of meta learning approaches for few-show learning (Sec. 3.1) and establish a link between single image 3D reconstruction and few-shot learning (Sec. 3.2). We propose a problem formulation inspired by few-shot learning (Sec. 3.3) and propose a solution in the following subsections. Specifically, we apply recently developed differentiable optimization.

heading: 3.1 BACKGROUND
text: Supervised learning. Given training set Dtrain = {xi, yi}Ni=1, supervised learning learns a predictor y = P(x) which is able to predict the labels of test set Dtest = {xi, yi}Mi=1 (assuming both Dtrain and Dtest are sampled from the same distribution).
Few-shot learning. In few-shot learning, the size N of the training set is typically small. The common learning algorithms on a single task usually cause problems like overfitting. However, we are given a collection of tasks, the meta-training set Dmeta−train = {Dtraint ,Dtestt }Tt=1, on which we train a meta-learner which produces a predictor on every task {Dtraint ,Dtestt } and generalizes well on the meta-testing Dmeta−test = {Dtrains ,Dtests }Ss=1. Consider a K-class classification task, each training set Dtrain consists of N/K labelled examples for each of K classes. The meta-training task is often referred to as N/K-shot-K-way. Refer to Figure 1 for an example visualization of 2-shot-3-way few-shot image classification.
Meta learning approaches for few-shot learning often involve an embedding network, and a base learner (learning algorithm). The embedding network maps training samples to an embedding space. We explain in later subsections how the 3d reconstruction is connected to meta learning in these two aspects.

heading: 3.2 SINGLE IMAGE 3D RECONSTRUCTION
text: A watertight shape can be represented by an indicator (or occupancy) function O : R3 → {0, 1}. We define O(x) = 1 if x ∈ R3 is inside the object, O(x) = 0 otherwise. We can sample a set of points in R3 and evaluate the indicator O, then we have the labeled point set {xi, yi}Mi=1 where yi ∈ {0, 1}. The number M needs to be large enough to approximate the shape. In this way, we rewrite the target ground-truth as a point set. This strategy is also used by Mescheder et al. (2019) and Deng et al. (2019). Also see Figure 1 for an illustration.
The goal of single image 3D reconstruction is to convert an input image I to the indicator function O. Previous work either directly learns O (Mescheder et al., 2019))or trains a network to predict an intermediate parametric representation (e.g. collection of convex primitives (Deng et al., 2019), half-spaces (Chen et al., 2019)). Different from any existing methods, our shape representation is to generate training data for a few-shot classification problem. In order to make the connection clear, we denote the ground-truth {xi, yi}Mi=1 as Dtest. The training data of single image 3D reconstruction are a collection of images It and their corresponding shapes Dtestt which we denote as Dmeta−train = {It,Dtestt }Tt=1. The goal is to learn a network which takes as input an image I and outputs a functional (predictor) P(x) which works on Dtest. We summarize the notation and the mapping of few-shot classification to 3D shape reconstruction in Table 1. Using the proposed mapping, we need to find a data generating network (Fig. 2 left) to
convert the input I to a set of labeled pointsDtrain = {xi, yi}Ni=1 (usuallyN is far smaller thanM ). Then the Dmeta−train can be rewritten as {Dtraint ,Dtestt }Tt=1. It can be seen that, this formulation has a high resemblance to few-shot learning. Also see Figure 1 for a visualization. As a result, we can leverage techniques from the literature of few-shot learning to jointly train the data generation and the classification components.

heading: 3.3 FORMULATION
text: Similar to few-shot learning, the problem can be written as a bi-level optimization. The inner optimization is to train the predictor P(x) to estimate the inside/outside label of a point,
min P
E(x,y)∈Dtrain [L (y,P(x))] , (1)
where L(·, ·) is a loss function such as cross entropy. While in few-shot learning Dtrain is provided or sampled, here Dtrain is generated by a network f , Dtrain = f(I). To reconstruct the shape (I,Dtest), the predictor P should work as an approximation of the indicator O and is expected to minimize the term, E(x,y)∈Dtest [L (y,P(x))] . (2) This process is done by a base learner (machine learning algorithm) in some meta learning methods (Bertinetto et al., 2019; Lee et al., 2019). The final objective across all shapes (tasks) is
min E(I,Dtest)∈Dmeta−train [ E(x,y)∈Dtest [L (y,P(x))] ] ,
s.t. P = min P
E(x,y)∈Dtrain [L (y,P(x))] , Dtrain = f(I), (3)
which is exactly the same as for meta learning algorithms if we remove the constraintDtrain = f(I).
Point Embedding. In meta learning approaches for few-shot classification, an embedding network is used to map the training samples to an embedding space, g(x) = e, where e is the embedding vector of the input x. We also migrate the idea to 3d shape representations, g(x|I) = e, where the embedding network is also conditioned on the task input I.
In later sections, we use ei and e to denote the embeddings of point xi and x, respectively. In addition to the objective Eq equation 3, we add a regularizer,
w · Ex ‖e− x‖22 , (4) where w is the weight for the regularizer. There are multiple solutions for the Embedding Network, thus we want to use the regularizer to shrink potential solutions, which is to find a similar embedding space to the original one. The w is set to 0.01 in all experiments if not specified.

heading: 3.4 DIFFERENTIABLE LEARNER
text: The Eq. equation 1 is the inner loop of the final objective Eq. equation 3. Recent meta learning approaches use differentiable learners, e.g., R2D2 (Bertinetto et al., 2019) uses Ridge Regression and MetaOptNet (Lee et al., 2019) uses Support Vector Machines (SVM). We describe both cases here. Different than the learner in R2D2 and MetaOptNet, we use kernelized algorithms.
Ridge Regression. Given a training set Dtrain = {xi, yi}Ni=1, The kernel ridge regression (Murphy, 2012) is formulated as follows,
minimize α
λ 2 αᵀKα + 1 2 ‖y −Kα‖22 , (5)
where K ∈ RN×N is the data kernel matrix in which each entry Ki,j is the Gaussian kernel K(ei, ej) = exp(−‖ei − ej‖22 /(2σ
2)). The solution is given by an analytic form α = (K + λI)−1y. By using automatic differentiation of existing deep learning libraries, we can differentiate through α with respect to each xi in Dtrain. We obtain the prediction for a query point x via
RR(e;Dtrain) = N∑ i=1 αiK(ei, e). (6)
SVM. We use the dual form of kernel SVM,
minimize α
1
2 N∑ j=1 N∑ i=1 αiαjyiyjK(ei, ej)− N∑ i=1 αi
subject to N∑ i=1 αiyi = 0, 0 ≤ αi ≤ C, i = 1, . . . , N,
(7)
where K(·, ·) is the Gaussian kernel. . The discriminant function becomes,
SVM(e;Dtrain) = N∑ i=1 αiyiK(ei, e) + b. (8)
Using recent advances in differentiable optimization by Amos & Kolter (2017), the discriminant function is differentiable with respect to each xi in Dtrain. While the definitions of both learners are given here, we only show results of ridge regression in our main paper. Additional results of SVM are provided in the supplementary material. In traditional machine learning, SVM is better than ridge regression in many aspects. However, we do not find SVM shows significant improvement over ridge regression in our experiments. The conclusion is also consistent with Lee et al. (2019).

heading: 3.5 INDICATOR APPROXIMATION
text: We clip the outputs of the ridge regression to the range [0, 1] as an approximation ÔRR of the indicator O. For the SVM, the predictor outputs a positive value if x is inside the shape otherwise a negative value. So we apply a sigmoid function to convert it to the range [0, 1], ÔSVM (x) = Sigmoid(βSVM(e;Dtrain)), where β is a learned scale. Then Eq. equation 2 is written as follows:
E(x,y)∈Dtest [∥∥∥Ô(x)− y∥∥∥2
2
] , (9)
where the minimum squared error (MSE) loss is also used in CvxNet (Deng et al., 2019).

heading: 4 IMPLEMENTATION
text: Base learner. We use λ = 0.005 for ridge regression and C = 1 for SVM in all experiments. The parameter σ in the kernel function K(·, ·) is learned during training and is shape-specific, i.e., each shape has its own σ.
Similar to recent works on instance segmentation (Liang et al., 2017; Kendall et al., 2018; Novotny et al., 2018; Zhang & Wonka, 2019), we also find that a simple R3 spatial embedding works well, i.e., x ∈ R3 and σ ∈ R3. Another reason for choosing R3 embedding is that we want to visualize the relationship between the original and embedding space in later sections.
† Re-implemeneted version.
Networks. Our framework is composed of three sub-networks: Feature Network, Point Generation Network and Embedding Network (see Fig. 2). For the Feature Network, we use EfficientNetB1 (Tan & Le, 2019) to generate a 256-dimensional feature vector. This architecture provides a good tradeoff between performance and simplicity. Both the Point Generation Network and Embedding Network are implemented with MLPs (see the supplementary material for the detailed architectures). The Point Generation Network outputs σ and points {xi}Ni=1 (half of which have +1 inside label and the other half have −1 outside label) where N = 64. The Embedding Network takes as input the concatenation of both the point x and the feature vector λ. Instead of outputting e directly, we predict the offset o = e − x and apply the tanh activation to restrict the output to lie inside a bounding box. The training batch size is 32. We use Adam (Kingma & Ba, 2014) with learning rate 2e− 4 as our optimizer. The learning rate is decayed with a factor of 0.1 after 500 epochs.
Data. We perform single image 3d reconstruction on the ShapeNet (Chang et al., 2015) dataset. The rendered RGB images and data split are taken from (Choy et al., 2016). We sample 100k points uniformly from the shape bounding box as in OccNet (Mescheder et al., 2019) and also 100k “near-surface” points as in CvxNets (Deng et al., 2019) and SIF (Genova et al., 2019). Along with the corresponding inside/outside labels, we construct Dtest for each shape offline to increase the training speed. At training time, 1024 points are drawn from the bounding box and 1024 ”nearsurface”. This is the sampling strategy proposed by CvxNet.

heading: 5 RESULTS
text: Evaluation metrics. We use the volumetric IoU, the Chamfer-L1 distance and FScore (Tatarchenko et al., 2019) for evaluation. Volumetric IoU is obtained from 100k uniformly sampled points. The Chamfer-L1 distance is estimated by randomly sampling 100k points from the ground-truth mesh and predicted mesh which is generated by Marching Cubes (Lorensen & Cline, 1987). F-Score is calculated with d = 2% of the side length of the reconstructed volume. Note that following the discussions by Tatarchenko et al. (2019), F-Score is a more robust and important metric for 3d reconstruction compared to IoU and Chamfer. All three metrics are used in CvxNet (Deng et al., 2019).
Competing methods. The list of competing methods includes Pixel2Mesh (Wang et al., 2018), AtlasNet (Groueix et al., 2018), SIF (Genova et al., 2019), OccNet (Mescheder et al., 2019), CvxNet (Deng et al., 2019) and Hypernetwork (Littwin & Wolf, 2019). Results are taken from these works except for (Littwin & Wolf, 2019) which we provide re-implemented results.
Quantitative results. We compare our method with a list of state-of-the-art methods quantitatively in Table 2. We improve the most important metric, F-score, from 51.75% to 59.66% compared to the previous state of the art OccNet (Mescheder et al., 2019). We also improve upon OccNet
in the two other metrics. According to the L1-Chamfer metric, AtlasNet (Groueix et al., 2018) has a slight edge, but we would like to reiterate that this metric is less important and we list it mainly for completeness. Besides that, we re-implemented OccNet (Mescheder et al., 2019) using our embedding network and Hypernetwork (Littwin & Wolf, 2019) under our training setup. For OccNet, the re-implemented version has better results than the original one. For Hypernetwork, we carefully choose the hyper-parameters as the paper suggested. Overall, we can observe that our method has the best results compared to all other methods.
Qualitative results. We show qualitative reconstruction results in Fig. 3. We compare our results with re-implemented OccNet. We also show shape interpolation results in Fig. 4. Interpolation is done by acquiring two features λ1 and λ2 from two different images. Then we reconstruct meshes while linearly interpolating the two features.
Ablation study. We show the metrics under different hyper-parameter choices in Table 3, including different N and w. We find that generally smaller w and larger N gives better results. However when w is small enough (w = 0.01, 0.1), the metrics are very close no matter how large N is.
Visualization of embedding space. In Fig. 5, we first show Dtrain along with the corresponding meshes. Since our embedding space is also R3, we can visualize the embedding space. When the number of pointsN inDtrain is small, the embedded shape looks more like an ellipsoid. In this case, the Embedding Network maps inside points into an ellipsoid-like shape. When N is getting larger, the embedded mesh has a shape more similar to the original one. Thus, the Embedding Network only shift points by a small distance. This shows how the Embedding Network and the Point Generation Network collaborate. Sometimes the Embedding Network does most of the work when points are difficult to classify, while sometimes the Point Generation Network needs to generate structured point sets. In Fig. 6, we show more examples of Dtrain given N = 64 and w = 0.01.
Relation to hypernetworks. The approach of hypernetworks (Ha et al., 2016), uses a (hyper)network to generate parameters of another network. Our method has a relation to hypernetworks. The final (decision) surface is decided by ei = g(xi), i = 1, . . . , N where xi is a point in Dtrain. According to Eq. equation 6, the output is ∑N i=1 αiK(g(xi), g(x)). The points Dtrain = {xi, yi}Ni=1 is obtained by the Point Generation Network, which we can treat as a hypernetwork, and the points are generated parameters. Thus our method can also be interpreted as a hypernetwork. Littwin & Wolf (2019) proposes an approach for shape reconstruction with hypernetworks but with a very high-dimensional latent vector (1024) and generated parameters (3394), while we only use a 256 dimensional latent vector and 3×64 = 192 parameters. We still have better results according to Table 2.
Table 3: Ablation study. We show (a) IoU ↑, (b) Chamfer ↓ and (c) F-Score ↑ for different N and w.
(a) IoU ↑
IoU 256 128 64 32 16 8 w = 10 0.582 0.578 0.587 0.583 0.568 0.553 w = 5 0.599 0.600 0.595 0.588 0.580 0.574 w = 2 0.600 0.604 0.600 0.595 0.589 0.584 w = 1 0.603 0.607 0.604 0.596 0.595 0.592 w = 0.1 0.610 0.606 0.608 0.609 0.611 0.607 w = 0.01 0.606 0.610 0.608 0.608 0.613 0.610 w = 0 0.607 0.606 0.608 0.605 0.609 0.611
(b) Chamfer ↓
Chamfer 256 128 64 32 16 8 w = 10 0.208 0.224 0.235 0.273 0.282 0.317 w = 5 0.198 0.212 0.226 0.242 0.270 0.285 w = 2 0.189 0.196 0.210 0.223 0.230 0.252 w = 1 0.186 0.191 0.199 0.211 0.216 0.220 w = 0.1 0.180 0.179 0.179 0.176 0.175 0.178 w = 0.01 0.182 0.175 0.177 0.175 0.171 0.173 w = 0 0.178 0.182 0.176 0.178 0.173 0.170
N = 32 N = 64 N = 128
O bj
ec t
E m
be dd ed O bj ec
t E m be dd ed
Figure 5: We show Dtrain and Dtrain in embedding space given N = 32, 64, 128 and w = 0.01. Positive points are shown in red and negatives points in blue. The shapes in both spaces are shown as transparent sur-
faces.

heading: 6 CONCLUSION
text: In this paper, we presented a shape representation for deep neural networks. Training data generating networks establish a connection between few-shot learning and shape representation by converting an image of a shape into a collection of points as training set for a supervised task. Training can be solved with meta-learning approaches for few-shot learning. While our solution is inspired by few-shot learning, it is different in: 1) our training datasets are generated by a separate network and not given directly; 2) our embedding network is conditioned on the task but traditional few-shot learning employs unconditional embedding networks; 3) our test dataset is generated by sampling and not directly given. The experiments are evaluated on a single image 3D reconstruction dataset and improve over the SOTA.

heading: A APPENDIX

heading: A.1 NETWORKS
text: We show the architecture of the embedding network and the points generator in Fig. 7. The embedding network is similar to DeepSDF (Park et al., 2019).
A.2 VISUALIZATION OF FEATURE SPACE
We visualize the feature space in Fig. 8.

heading: A.3 STATISTICS OF METRICS
text: Here we show more detailed statistics in Fig. 9. Comparing to IoU and F-score, most of the values of Chamfer concentrate around the mean value and rare values very far from the mean. The “peakedness” property of Chamfer implies it is unstable to a certain extent. The conclusion is also consistent with Tatarchenko et al. (2019).

heading: A.4 SHAPE INTERPOLATION
text: In Fig. 10, we show reconstructed shapes with bilinear interpolation. Given 4 shape latent feature vectors λ0,0, λ0,1, λ1,0 and λ1,1, we show interpolated shapes with the bilinear interpolation equation λv,h = (1− v)(1− h)λ0,0 + (1− v)hλ0,1 + v(1− h)λ1,0 + vhλ1,1 where v ∈ [0, 1] and h ∈ [0, 1]
A.5 EFFECTS OF w
In the main paper, we have a regularizer to force the embedding space to be similar to the original space, w · Ex ‖e− x‖22 . (10) When w is very large, the embedding space is almost the same as the original space, especially when N is large. The visualization is shown in Fig. 11. We set w = 10. This is much higher than w = 0.01 recommended in the paper. In this case, we expect the embedding network to do less work and as reported for w = 10 the method mainly works for a large number of points. We can observe that indeed the embedding space (bottom) is similar to the original space (top) especially when N is large. Positive points are shown in red and negative points in blue. Top row: meshes are reconstructed with our regular pipeline. Generated points are fed into the Embedding Network g first. Visualized points are in the original space. Middle row: meshes are reconstructed by removing
the embedding network without re-training. This isn’t expected to work well. Generated points are directly used in generating classification boundaries. Visualized points are in the original space. We can see that even without the embedding network the results are surprisingly reasonable for a large number of points. Bottom row: meshes in the embedding space. Visualized points are in the embedding space. We can observe that the meshes look similar to the correct chair, especially for large N . This also validates that the embedding network does not have a large influence on the result for large w. If the regularizing weight w is set to 0, the shape in embedding space will be more ellipsoid like.
Recall that, with the kernel ridge regression as our base learner, the final shape (decision) surface is decided by,
N∑ i=1 αiK(g(xi), g(x)).
If the original and embedding space are close, we can even reconstruct meshes without the Embedding Network g,
N∑ i=1 αiK(xi,x).
The results are shown in Fig. 11.

heading: A.6 THE CHOICE OF BASE LEARNERS
text: In Table 4, we show results of different base learners, SVM and ridge regression. The difference between the two methods is very small.

heading: A.7 RESULTS ON REAL WORLD IMAGES
text: Note that our model is trained on a synthetic dataset (ShapeNet). Here we test how our model generalizes to real world datasets. Without retraining and fine-tuning, we apply our trained model to Stanford Online Products. The qualitative results are shown in Fig. 12. We believe that the results are good, but there is no ground truth data for evaluation.
A.8 INTER-CATEGORIES INTERPOLATION
Similar to Fig. 4, we also show inter-categories interpolation in Fig. 13. We take samples from two different categories, and reconstruct shapes with interpolated latents. Although the interpo-
lated shapes are not as high quality than the originals, they still show some characteristics of both categories, e.g., sofas which are as tall as tables, and cars with chair-leg-like tires.

heading: A.9 TRAINING AND INFERENCE
text: We describe the detailed training process in Algorithm 1. This algorithm can be viewed along with Fig. 2. We also show the full process how we generate a triangular mesh given an input image in Algorithm 2.
Algorithm 1 Training for number of training iterations do • Sample a minibatch of m training sample pairs {Ib,Mb}Bb=1, where Ib is an image and Mb
is the target ground-truth mesh. • Get the latent representations for images with a feature backbone
Feature ( I ∈ RB×3×H×W ) = L ∈ RB×256,
• Get the labeled point sets (the first half are marked as positive and the other half are marked as negative)
PointGen(L) = X ∈ RB×N×3, y ∈ RB×N
• Embed the point sets X to another space
Embedding(X) = E ∈ RB×N×3,
• Optimize differentiable base learner P (·|α) which defines binary classification boundaries,
OptBaseLearner(P,E,y) = α ∈ RB×N ,
• Sample labeled point sets from ground-truth meshes {Mi}Bb=1,
Xt ∈ RB×M×3, yt ∈ RB×M ,
• Convert Xt to the embedding space
Embedding(Xt) = Et ∈ RB×N×3,
• Predict labels by using base learner P (·|α)
P (Et|α) = ȳt,
• Update Feature(·), PointGen(·) and Embedding(·) by descending stochastic gradients
∇MSE(ȳt,yt),
end for
Algorithm 2 Inference
• Prepare an input image I ∈ R3×H×W , • Get the latent representations for the image
Feature (I) = L ∈ R256,
• Get the labeled point sets (the first half are marked as postive and the other half are marked as negative)
PointGen(L) = X ∈ RN×3, y ∈ RN
• Embed points to the embedding space
Embedding(X) = E ∈ RN×3,
• Optimize the differentiable base learner P (·|α),
OptBaseLearner(P,E,y) = α ∈ RN ,
• Prepare a point set on a 3D grid with arbitrary resolution G×G×G,
Xt ∈ RG 3×3
• Convert Xt to embedding space
Embedding(Xt) = Et ∈ RG 3×3,
• Predict labels by using base learner P (·|α)
P (Et|α) = ȳt ∈ RG×G×G,
• Apply marching cubes algorithm to ȳt,
MarchingCubes(ȳt) = (V,F).

heading: A.10 CHOICE OF BACKBONES
text: In Table 5, we show the performance of our method with different backbones. We keep all of the setup the same except for the feature backbone.

heading: A.11 EMBEDDING DIMENSIONS
text: In Table 6, we show the performance of our method with different embeddings. Note that different from Table 5, the results shown here are trained with N = 256.

Review summaries:
The paper introduce a few shot learning method for single view reconstruction task. Different from traditional few shot learning methods, data used for training are generated by a network which takes an image as input. Both qualitative and quantitative reconstruction results show that this method gives a better performance than current SOTA methods at less training data(occupancy or distance field ground truth).  

In this paper, the authors provide a novel idea for shape reconstruction from a single image. They  enable the idea and techniques developed in the few-shot learning literature to the problem of shape reconstruction. They construct the meta-train data by a point generation network, and utilize a differential learner to learn the decision boundary.  They model the reconstruct problem by bi-level optimization.  It is an interesting work. The results are also good. 

The paper proposes a method for 3D shape reconstruction from image using a bi-level optimization technique. Specifically, the first part of the proposed network extract features from the image and creates a set of 3D points sampled near the surface of the shape to be reconstructed with labels corresponding to the points being inside or outside the 3D shape boundary. Then, the produced sets of points are fed to the second part of the network which is trained to approximate an implicit representation of the shape. At the inference time, a 2D surface of the shape can be extracted from the predicted implicit representation.