Training Data Generating Networks: Shape Reconstruction via Bi-level Optimization
Authors: Biao Zhang
Peter Wonka
Venue: ICLR 2022 Poster (2022)
Keywords: shape reconstruction single image
meta learning
few-shot learning
differentiable optimization
bi-level optimization

Abstract: We propose a novel 3d shape representation for 3d shape reconstruction from a single image. Rather than predicting a shape directly, we train a network to generate a training set which will be fed into another learning algorithm to define the shape. The nested optimization problem can be modeled by bi-level optimization. Specifically, the algorithms for bi-level optimization are also being used in meta learning approaches for few-shot learning. Our framework establishes a link between 3D shape analysis and few-shot learning. We combine training data generating networks with bi-level optimization algorithms to obtain a complete framework for which all components can be jointly trained. We improve upon recent work on standard benchmarks for 3d shape reconstruction.

Published as a conference paper at ICLR 2022
TRAINING DATA GENERATING NETWORKS: SHAPE RECONSTRUCTION VIA BI-LEVEL OPTIMIZATION
Biao Zhang& & Peter Wonka
KAUST
(biao. (.lao.zhaan zhang, peter. werka..wsnkaekkausste edu. sa
ABSTRACT
We propose novel 3d shape representation for 3d shape reconstruction from a single image. Rather than predicting shape directly, we train network to gen- erate a training set which will be fed into another learning algorithm to define the shape. The nested optimization problem can be modeled by bi-level optimiza- tion. Specifically, the algorithms for bi-level optimization are also being used in meta learning approaches for few-shot learning. Our framework establishes link between 3D shape analysis and few-shot leamning. We combine training data generating networks with bi-level optimization algorithms to obtain complete framework for which all components can be jointly trained. We improve upon recent work standard benchmarks for 3d shape reconstruction.
INTRODUCTION
Neural networks have shown promising results for shape reconstruction (Wang et al. 2018 Groucix et al. 2018 Mescheder et al. 2019 Genova et al.) 2019). Different from the image domain, there universally agreed upon way to represent 3d shapes. There exist many explicit and implicit
representations. Explicit representations include point clouds (Qi et al.) 2017a b Lin et al.) 2017). grids (Wu al. 2015 Choy et al. 2016 Riegler al.
2017:
Wang et al. 2017]] Tatarchenko al.,
2017) and meshes (Wang et al. 2018: Hanocka et al. 2019 Implicit representations (Mescheder
al. 2019; Michalkiewicz al. 2019; Park et al.) 2019) define shapes as iso-surfaces of functions.
Both types of representations are important as they have different advantages.
In this work, we set out investigate how meta-learning can be used to learn shape representations. To employ meta-learning, we need to split the learning framework into two (or more) coupled learn- ing problems. There are multiple design choices for exploring this idea. One related solution is to use hypernetwork as the first learning algorithm that produces the weights for second network. This approach (Littwin & Wolf, 2019), is very inspiring, but the results can still be improved by work. The value of work are not mainly the state-of-the-ar results, but the new framework that enables the flow of new ideas, techniques, and methods developed in the context of few-shot learning and meta-optimization to be applied shape reconstruction.
In order to derive our proposed solution, we draw inspiration from few-shot learning. The formula- tion of few-shot learning we use here is similar to Lee al. (2019); Bertinetto al. (2019). but also other formulations of few shot learning exist, e.g., MAML (Finn et al., 2017). In few-shot learning for image classification, the learning problem is split into multiple tasks (See Fig.[]. D.. For each task, the learning framework has to learn a decision boundary to distinguish between smaller set of classes. In order to do so, a first network is used to embed images into high-dimensional feature space. A second learner (often called base learner in the literature of meta learning approaches for few-shot learning) is then used learn decision boundaries this high-dimensional feature space. A key point is that each task has separate decision boundaries. In order to build bridge from few- shot learning to shape representations, we identify each shape as a separate task. For each task, there are two classes to distinguish: inside and outside. The link might seem bit unintuitive at first, because in the image classification case, each task is defined by multiple images already divided into multiple classes. However, in our case, task is only associated with single image. We therefore introduce an additional component, a training data generating network, to build the bridge to few- shot learning (See Fig. 2 This training data generating network takes single image as input and

---

Published as a conference paper at ICLR 2022
Task
Task 2
Task
Shape
Shape 2
Shape T
Derain
Diraan
Dirain
Dirain
Dirain
Dirain
Diesd
Diuet
D Diest
D
Dinst
Dieat
Figure I: Few-shot learning v.s. shape representation. Left (gray): few-shot classification (im- ages taken from minilmageNet). We show 2-shot-3-way few-shot classification in which every training task contains 3 categories (shown in blue, orange and red bounding boxes) and 2 training samples in each category. Thus we use 2 x 3 6 training samples in Dirasn to build classifiers which are expected to work well on Drest. Right (yellow): shape representation. Shape surfaces are shown as dotted lines. Our proposed training data generating networks can encode a shape as labeled set of points with labels blue or red to provide training set Dirai, The learning frame- work uses the training sets to build the surfaces (classification boundaries) which can be evaluated by densely sampling the space, i.e., Drest.
outputs multiple 3D points with an inside and outside label. These points define the training dataset for one task (shape). We adopt this data set format to describe task, because points are a natural shape representation. Similar to the image classification setting, we also employ an embedding net- work that maps the original training dataset set of 3D points) another space where it is easier to find decision boundary (a distorted 3D space). In contrast to the image classification setting, the input and output spaces of the embedding network have lot fewer dimensions, i.e. only three dimensions. Next, we have to tackle the base learner that takes set of points in embedding space as input. Here, we can draw from multiple options. particular, we experimented with kernel SVMs and kernel ridge regression. As both give similar results (In meta learning (Lee et al. 2019] also find the same conclusion), we opted for ridge regression in our solution due to the much faster processing time. The goal of ridge regression is learn decision boundary (shape surface) in 31) space describing the shape.
Our intermediate shape representation is set of points, but these points cannot be chosen arbitrarily. The set of points are specifically generated so that the second learning algorithm can process them to find decision boundary. The second learning algorithm functions similar to other implicit shape representations discussed above. The result of the second learning algorithm (SVM or ridge regres- sion) can be queried as an implicit function. Still the second learning algorithm cannot be replaced by other existing networks employed in previous work. Therefore, among existing works, only our representation can directly use few-shot learning. It does not make sense to train our representation without a few-shot learning framework and no other representation can be directly trained with few-shot learning framework. Therefore, all components in our framework are intrinsically linked and were designed together.
Contributions.
We propose a new type of shape representation, where we train a network to output a training set (a set of labeled points) for another machine learning algorithm (Kernel Ridge Regression or Kernel-SVM).
We found an elegant way to map the problem of shape reconstruction from a single image to the problem of few-shot classification by introducing network to generate training data. We combined training data generating networks with two meta learning approaches (R2D2 (Bertinetto al. 2019) and MetaOptNet (Lee et al. 2219)) in framework. Our work enables the application of ideas and techniques developed in the few-shot learning literature to the problem of shape reconstruction.
2

---

Published as a conference paper at ICLR 2022
Data Generation
Few-Shot Learning
231
Weights
Feature Network
PointGinn Network
[x.. Mo Jim)
Embedding Network
[e,, Bi
Base
Learner
Query
Embedding
PP
Network
Figure 2: Pipeline. Networks with trainable parameters are shown in red boxes with round corners. Outputs are shown in yellow boxes. The base learner is machine learning algorithm shown in the gray box. Arrows show how the data flows in the network. The sign c means we are concatenating multiple inputs.
We validate our model using the problem of 3d shape reconstruction from single image and improve upon the state of the art.
RELATED WORK
Neural implicit shape representations. We can identify two major categories of shape repre- sentations: explicit representations, where shape can be explicitly defined; and implicit repre- sentations, where a shape can be defined as iso-surface of a function (signed distance function or indicator function). In the past decade, have seen great success with neural network based
plicit shape representations: voxel representations (Wu et al.) [013] Choy et al.) 2016 Riegler et al.]
2017 Wang al.] 20177: Tatarchenko al., 2017), point representations (Qi et al. 2017a b Fan
et al. 2017 Lin et al. (017), and mesh representations (Wang et al. 2018 Hanocka et al. 2019)
the other hand, modeling implicit representations with neural networks has been current trend, where usually signed distance function or indicator function is parameterized by neural net- work (Mescheder et af., 2019; Chen & Zhang 2019, Michalkiewicz et af. 2019 Park et al., 2019). More recent works learn network that outputs intermediate parameters, e.g. CvxNet (Deng et al.) 2019) and BSP-Net (Chen et al. 2019) learns to output half-spaces. We propose a novel type of shape representation, where the model outputs training set of labeled points.
Few-shot learning. There are two common meta learning approaches for few-shot learning: metric-based (Koch et al.) 2015) Vinyals et 2006 (nell et al. (017). which aims to learn metric for each task; optimization- based (Ravi Larochelle| [0017 Finn et al. 2017 Nichol et al. 2018), which is designed to learn with few training samples es by adjusting optimization algorithms. These approaches commonly have two parts, an embedding model for mapping an input to an embedding space, and base learner for prediction. Qiao et al. 2018); Gidaris & Komodakis (2018) train hy- pernetwork (Ha et al. 2016) to output weights of another network (base learner). R2D2 (Bertinetto al. 2019) showed that using light-weight and differentiable base learner (e.g. ridge regression) leads to better results. To further develop the idea, MetaOptNet (Lee er al., 2019) used multi-class support vector machines (Crammer & Singer, (00T) as base learner and incorporated differentiable optimization (Amos & Kolter 2017: Gould et al. (016) into the framework. Lee et al. (2019) also methods. In our work, we propose shape representa-
shows it can outperform hypernetwork- tion that is compatible with few-shot classification framework so that we can utilize existing meta learning approaches. Specifically, we will use ridge regression and SVM as the base learner. The most relevant method to ours is Littwin & Wolf 2019), which adapts hypernetworks. However, discussed above, differentiable optimization methods (Bertinetto et al., 2019 Lee et al. 2019) are generally better than hypernetworks. Besides that, meta learning has been applied to other fields in shape analysis, e.g., both Sitzmann et al. (2020) and Tancik et al. (02T propose to use MAML- algorithms (Finn et al. 2017) to learn weight initialization.
3

---

Published as a conference paper at ICLR 2022
3 METHOD
The framework is shown in Fig. 2 The network is mainly composed of sub-networks. The Feature Network maps an input image to feature space. The resulting feature vector X is then decoded by the Point Generation Network to a labeled point set fx, ) The point set will be used as training data our base learner later. After that the Embedding Network projects the point set into embedding space. The projected points e, and the labels are taken as the input of binary classifer (ridge regression or SVM) parameterized by a. Finally, the framework able to output the insidc/outside label y of query point x by projecting it into the embedding space and feeding it to the binary classifier.
In the following subsections, we describe our method in more detail. First, we introduce the back- ground of meta learning approaches for few-show learning (Sec. 3.1 and establish a link between single image 3D reconstruction and few-shot learning (Sec. 3.22 We propose problem formula- tion inspired by few-shot learning (Sec. 3.3) and propose solution in the following subsections. Specifically, we apply recently developed differentiable optimization.
3.1 BACKGROUND
Supervised learning. Given training set Dirain dictor y = P(x) which is able to predict the labels of test set Diest {x,, 3 (assuming both Dtrain and Diestt are sampled from the same distribution).
ys J yx supervised learning learns pre-
Few-shot learning. In few-shot learning, the size N of the training set is typically small. The common learning algorithms on single task usually cause problems like overfitting. However, we [iDtrain, Drest 1 Jt=1, on which
are given a collection tasks, the meta- training set D train meta-learner which produces predictor on every task (Diram Diest and generalizes well on the meta-testing Dmeea-teat teat = [Dfrain Drestts 1s=1-
Consider K-class classification task, each training set Dirain consists of N/K labelled examples for each of K classes. The meta- vetta-tainnng task is often referred to as Figure for an example visualization of 2-shot-3-way few-shot image classification.
K-way. Refer to
Meta learning approaches for few-shot learning often involve an embedding network, and a base learner (learning algorithm). The embedding network maps training samples to embedding space. We explain in later subsections how the 3d reconstruction is connected to meta learning in these two aspects.
3.2 SINGLE IMAGE 3D RECONSTRUCTION
A watertight shape can be represented by an indicator (or occupancy) function O R + {0, 1}. We define (xx) 1 if x E R is inside the object, O(x) = 0 otherwise. We can sample a set of points in R and evaluate the indicator O, then we have the labeled point set (x,, 1M where y: {0. 1}. The number M needs to be large enough to approximate the shape. In this way, we rewrite the target ground-trutt as a point set. This strategy also used by Mescheder et al. (2019) and Deng et al. (2019). Also see Figure for an illustration.
The goal of single image 3D reconstruction is to convert an input image I to the indicator function O. Previous work either directly learns O (Mescheder et al. 2019))r trains a network to predict an intermediate parametric representation (e.g. collection of convex primitives (Deng et al. 2019), half-spaces (Chen et al.) 2019)). Different from any existing methods, our shape representation is to generate training data for a sew-shot classification problem. In order to make the connection clear, denote the ground-truth (x,, J.M. as D
The training data of single image 3D reconstruction are a collection of images I, and their corre- sponding shapes Diest which we denote network which takes as input an image I and outputs a functional (predictor) P(x) which works on Drest
pymeta-traid = [I, De 11=1- The goal is to learn
We summarize the notation and the mapping of few-shot classification to 3D shape reconstruction in Table Using the proposed mapping, we need to find data generating network (Fig. left) to
4

---

Published as a conference paper at ICLR 2022
Table I: Symbols for few-shot classification and 3D reconstruction. Rows shown in blue are items 3D shape reconstruction has but few-shot classification does not.
Few-shot classification 3D shape reconstruction
I
input images Divoin (I)
J
Dirasn
[x., 9k
Diesi
{x., y4 h
(x,, ye 3
X
images
points
yi predictor P()) D train
categories
inside/outside labels surface boundary
classifier
(Drrai D. 1T
{I, Diest ) =t=1
convert the input to set of labeled points Diroin (x, Y/ M (usually N is far smaller than M). Then the Dmott-ttrin can be rewritten as {Dtrain Diost 17 has high resemblance to few-shot learning. Also Figure can leverage techniques from the literature of few-shot learning to jointly train the data generation and the classification components.
1t=1- can be seen that, this formulation [ for visualization. As result,
3.3 FORMULATION
Similar to few-shot learning, the problem can be written as bi-level optimization. The inner opti- mization train the predictor P(x) estimate the inside/outside label of a point,
min E (x.g) [C (y. P(x))],
(1)
where -(--)) is loss function such as cross entropy. While in few-shot learning Derain is provided or sampled, here Derain is generated by a network f. Drrain = fII. To reconstruct the shape (I, DAest), the predictor P should work as an approximation of the indicator O and is expected to minimize the term,
E [ (v, P(x))]
This process is done by base learner (machine learning algorithm) in some meta learning methods (Bertinetto et al. 2019 Lee et al. 2019). The final objective across all shapes (tasks) is
min E Drett
(X.W)DDD [6 ((x))]]
(3)
s.t. min E (.)) (C.. P(x))], Drrain fII,
p
which is exactly the same as for meta learning algorithms we remove the constraint Diroin = fI).
Point Embedding. In meta learning approaches for few-shot classification, embedding network used to map the training samples to an embedding space, g(x) = e, where e is the embedding vector of the input We also migrate the idea 3d shape representations, g e, where the embedding network is also conditioned on the task input I.
In later sections, use e; and to denote the embeddings of point x; and x, respectively. In addition to the objective Eq equation 3 we add regularizer.
Ex lle - x1[.
(4)
where is the weight for the regularizer. There are multiple solutions for the Embedding Network, thus we want to use the regularizer to shrink potential solutions, which to find a similar embedding space to the original one. The is set to 0.01 in all experiments if not specified.
3.4 DIFFERENTIABLE LEARNER
The Eq. equation is the inner loop of the final objective Eq. equation 3 Recent meta learning approaches use differentiable learners, e.g., R2D2 (Bertinetto et al., 2019) uses Ridge Regression and MetaOptNet (Lee et al. 2019) uses Support Vector Machines (SVM). We describe both cases here. Different than the learner in R2D2 and MetaOptNet, we use kernelized algorithms.
5

---

Published as a conference paper at ICLR 2022
Ridge Regression. Given training set Dtrasn f(x,,331 The kernel ridge regression (uur- phy 2012) is formulated as follows,
A Taa 2
minimize
2 bly Kol
(5)
where K E RNN is the data kernel matrix in which each entry K,j is the Gaussian ker- nel (() exp(-Ile; ey 12 (2o0))) The solution given by an analytic form (K + +AI+ Ay) By using automatic differentiation of existing deep learning libraries, we can differ-
entiate through a with respect each x; in Dirain, We obtain the prediction for query point via
RR(e; Dirain)
(e,,e.,
(6)
1=1
SVM. We use the dual form of kernel SVM,
N
N Σ
minimize
((e,e)
i=1
(7)
N
subject to
Σ
a.y = 0, 0 0 s C, i == I,..,,N,
where ((,,) is the Gaussian kernel..
The discriminant function becomes,
SV Derain) = Σ any K(e +b.
(8)
Using recent advances in differentiable optimization by Amos & Kolter 2017). the discriminant function is differentiable with respect to each x, in Diram
While the definitions of both learners are given here, only show results of ridge regression in main paper. Additional results of SVM are provided the supplementary material. In traditional machine learning, SVM is better than ridge regression in many aspects. However, do not find SVM shows significant improvement over ridge regression in our experiments. The conclusion is also consistent with Lee al. (2019).
3.5 INDICATOR APPROXIMATION
We clip the outputs of the ridge regression to the range [0, 1] as an approximation Ónn of the indicator O. For the SVM, the predictor outputs a positive value is inside the shape otherwise
a negative value. So we apply a sigmoid function to convert to the range [0, 1]. Ósv as (x) =
33SM Dirain)) )), where 3 is learned scale. Then Eq. equation 2 written as follows:
[|00(s) -v1
where the minimum squared emror (MSE) loss also used in CvxNet (Deng et al. 2019
4 IMPLEMENTATION
Base learner. We use À 0.005 for ridge regression and C' 1 for SVM in all experiments. The parameter a in the kernel function K(.,-) is learned during training and is shape-specific, i.c., each shape has its own a.
Similar to recent works on instance segmentation (Liang et al. 2017 Kendall et al. 2018 Novotny et al. 2018 Zhang & Wonka 2019), also find that simple R spatial embedding works well, L..,, X E F and o E R.. Another reason for choosing I embedding is that we want to visualize the relationship between the original and embedding space in later sections.

---

Published as a conference paper at ICLR 2022
Table 2: Reconstruction results on ShapeNet. Pixel2Mesh(P2M) (Wang et al. 2018). At)asNet(AN)
We compare our results with
Groueix et al. 2018). Occ-
Net(ON) (Mescheder et al.; 2019); SIF
(Genova et al.
2019). CvxNet(CN) (Deng et al,
2019) and Hypernetwork(HNN) (Littwin & Wolf Best results are shown in bold.
Iol't
Chamfer ON!
F-Score 1 ON
Careg
Ours
P2M
ON
SIF
HN3
P2M
SIF
HN
HN
plane
11633
(0.420
0.571
0.603
0.530
0.598
0.60%
0.121
0.187
0.104
0.147
0.1
0 1.167
0
0.127
71.15
67.33
62 67
67.25
52.1
68.16
68.
bench
I1.524
0.3333
0.485
0.486
0.333
0.461 0.501
0.132
0.201
0.138
0.155
0.148
0.261
0.133
0.146
69.44
54.50
56.91
64
80
37.31
54.64
65,43
cabines
0.752
0.664
0.733
0.733
0.648
0.209
0.734
0.141
0.196
0.175
0.167
0.142
0.233
0.160
0.145
65.33
46.43
61.79
78
31.68
46.09
63.15
car
0.748
0.737
0.657
0.732
0.121
0.180
0.141
0.125 0.161
51.51
56.91
37.66
61.46
chair
0.532
0:396
0.501
0.515 0.389 0.591 0.517
0.205
0.265
0.209
0,228 0.217 0.380
0.337
0.230
48.83
18.89
42.41
46.38
26.90
38.49
45.56
display
0.:55
0.490
0.471 0.540
0.491 0.576
0,542
0.215
0.239
0.198 0.278
0.213 0.401
0.223
0.217
46.96
42.79
38.96
44.55
27.22
40.69
48,42
Jum
0.383
0.313 0.371 0.395 0.260
0.311 0.389
0.391
0.308
0.305 0.679
0.404
1.096
0.795
0:441
12.99
33.04
38
43.41
20.59
31.41
41.81
speaker
016588
0.617
0.577
0.285
0.261
22.42
rille
00.400 0.402 0.474
0.496
0.463
0.515 0,508
0.111
0.164
0.115 0.141
0.129
0.106
0.120
69.40
64
56
64.34
53
63.74
65.92
sofa
0.707 0.613 0.680 0.692 0.606 0.577 0.692
0.155
0,212 0.177 0.194
0.167
0.272
0.161
0.163
56.40
43:46
48.62
$1.01
30.94
42.11 53.13
table
0.551 0.385 0.506 0.534
0.372 0.473
0.531
0,171
0.218
0.190 0.189 0.179 0:454
0.358
0.189
65
25
44.93
58 58.49
63.67
30.78
48.10 63.35
phone
01.77
0.720
0.658
0.763
0.149
73.56
vessel
[0.567 (0.997 0,530
0.502
0.558
186
0.212
0.151 0.218
0.173
0.199
42.
36.04
45.88
49.03
mean
10.608 0.480 0,571 0,592 0.499 0.567
0.595
00.177
0.217
0.175
0.215
0.187
0.349
0.245
0.190
59.66
48.58
51.75
56.87
34,86
47.36 56.98
Re-implemeneted version.
Networks. Our framework is composed of three sub-networks: Feature Network, Point Genera- tion Network and Embedding Network (see Fig. 2). For the Feature Network, we use EfficientNett- B1 (Tan & Le| (019)) to generate a 256-dimensional feature vector. This architecture provides good tradeoff between performance and simplicity. Both the Point Generation Network and Embedding Network are implemented with MLPs (see the supplementary material for the detailed architec- tures). The Point Generation Network outputs a and points fx: } (half of which have +1 inside label and the other half have -1 1 outside label) where N = 64. The Embedding Network takes as input the concatenation of both the point x and the feature vector A. Instead of outputting e directly, we predict the offset o e x and apply the tanh activation to restrict the output to lie inside a bounding box. The training batch size is 32. We use Adam (Kingma & Ba, 2014) with learning rate 2e 4 as optimizer. The learning rate is decayed with a factor of 0.1 after 500 epochs.
Data. We perform single image 3d reconstruction on the ShapeNet (Chang et al. 2015) dataset. The rendered RGB images and data split are taken from (Choy et al. 2016)) We sample 100k points uniformly from the shape bounding box as in OccNet (Mescheder et al. 2019) and also 100k "near-surface" points as in CvxNets (Deng et al.) 2019) and SIF (Genova al. (209)). Along with
the corresponding inside/outside labels, we construct Dreat for each shape offline to increase the training speed. At training time, 1024 points are drawn from the bounding box and 1024 "near- surface". This is the sampling strategy proposed by CvxNet.
5 RESULTS
Evaluation metrics. We use the volumetric IOU, the Chamfer-LI distance and F- Score (Tatarchenko et al. 2019) for evaluation. Volumetric IoU is obtained from 100k uniformly sampled points. The Chamfer-LI distance is estimated by randomly sampling 100k points from the ground-truth mesh and predicted mesh which is generated by Marching Cubes (Lorensen & Cline 1987]. F-Score is calculated with d = 2% of the side length of the reconstructed volume. Note that following the discussions by Tatarchenko et al. (2019). F-Score a more robust and important met- ric for 3d reconstruction compared to IoU and Chamfer. All three metrics are used in CvxNet (Deng et al. 2019)
Competing methods. The list of competing methods includes Pixe12Mesh (Wang et al. 2018 AtlasNet (Groueix et al.] 2018). SIF (Genova et al., 2019). OceNet (Mescheder et al., 2019). CvxNet (Deng et al.) 2019) and Hypernetwork (Littwin & Wolf 2019). Results are taken from these works except for (Littwin& & Wolf 2019) which we provide re-implemented results.
Quantitative results. We compare our method with list of state-of-the- art methods quantitatively in Table 2 We improve the most important metric, F-score, from 51.75% to 59.66% compared to the previous state of the art OceNet (Mescheder et al. 2019). We also improve upon OceNet
7

---

Published as a conference paper at ICLR 2022
OxeNet Ground Fromh Ispan Images
Propoced
Figure 3: Top: Input images (137 x 137) and ground-truth meshes shown vermilion red. Mid- dle: predicted meshes from (re-implemented) OccNet (Mescheder et al. 2019) shown in yellow. Bottom: predicted meshes from our model shown in blue.
Figure 4: Reconstruction with interpolated feature vectors. Meshes in blue original, green meshes are interpolated.
in the two other metries. According the LI-Chamfer metric, AtlasNet (Groueix al., 2018 has a slight edge, but we would like to reiterate that this metric less important and we list mainly for completeness. Besides that, we re-implemented OceNet (Mescheder et al. 2019) using our embedding network and Hypernetwork (Littwin & Wolf 2019) under our training setup. For OceNet, the re-implemented version has better results than the original one. For Hypernetwork, we carefully choose the hyper-parameters as the paper suggested. Overall, we can observe that our method has the best results compared all other methods.
Qualitative results. We show qualitative reconstruction results in Fig. 3 with re-implemented OceNet. We also show shape interpolation results in Fig. Interpolation is done by acquiring two features A and Ag from two different images. Then we reconstruct meshes while linearly interpolating the two features.
We compare our results
Ablation study. We show the metrics under different hyper-parameter choices in Table includ- ing different N and w. We find that generally smaller w and larger N gives better results. However when small enough (w 0.01, 0.1), the metrics are very close no matter how large N is.
we first show Dtrain along with the corresponding meshes. Since our embedding space is also R we can visualize the embedding space. When the number of points N in Dtrain is small, the embedded shape looks more like ellipsoid. In this case, the Embedding Network maps inside points into an ellipsoid-like shape. When N getting larger, the embedded mesh has shape more similar to the original one. Thus, the Embedding Network only shift points by small distance. This shows how the Embedding Network and the Point Generation Network collaborate. Sometimes the Embedding Network does most of the work when points difficult to classify, while sometimes the Point Generation Network needs to generate structured point sets. In Fig. 6 we show examples of Dirain given N 64 and 0.01.
Visualization of embedding space. In Fig.
8

---

Published as a conference paper at ICLR 2022
(hyper- nnewwrrk to generate parameters of another network. Our method has a relation to hypemet-- works. The final (decision) surface is decided by e, g(x;), = ......N N where x, is point in Dtrain According to Eq. equation 6 the output is Σ (g(x)) g(x)) The points Dtrasn (xi; Vi yi] obtained by the Point Generation Network, which we can treat hyper- network, and the points are generated parameters. Thus our method can also be interpreted hypernetwork. Littwin & Wolf (2019) proposes an approach for shape reconstruction with hyper- networks but with very high- nigh-dimensional latent vector (1024) and generated parameters (3394), while we only use a 256 dimensional latent vector and 3 64 192 parameters. We still have better results according Table 2
Relation to hypernetworks. The approach of hypemetworks (Ha al. 2016).
Table Ablation study. We show (a) IoU 1. bb Chamfer
32
N 64
N 128
4. and (c) F-Score for different N and w.
Eastookeed Olyedt
(a) IoU f
IoU
256
128
64
32
16
8
10
0.582
0.578 0.587
0.583
0.568
0,553
w
0.599
0.600 0.595
0.588
0.580
0.574
u =
0.600 0.601 0.600 0.595 0.589 0.584
s' =
0.603 0.607 0.604
0.596 0.595
0.592
Obfect
0.1
0.610 0.606 0.608 0.609
0.611 0.607
0.01 0.606 0.610 0.608 0.608
0.613 0,610
w
0.607 0.606 0.608 0.605 0.609 0.611
Embeadde
(b) Chamfer J
Chamfer 256 128
64
32
16
w
0.208
0.224
0.235 0.273
0.282 0.317
M = 0.198
0.212 0.226 0.242 0.270 0.285
We show Dtrain and DDirsn in embedding space given N 32, 64, 128 and w 0.01. Positive points are shown in red and negatives points in blue. The shapes in both spaces shown transparent sur- faces.
Figure
w' = 0.189 0.196 0.210 0.223 0.230 0.252
0.186 0.191 0.999 0.211 0.216 0.220
0.1 0.180 0.179 0.179 0.176 0.175 0.178
w = 0.01
0.182
0.175 0.177 0.175 0.171 0.173
W 0.178
0.182 0.176 0.178 0.173 0.170
(c) F-Score
F-Score
256
128
64
32
w = 10
0.542
0.529
0.551
0.544
0.521
0.494
W' = 0.575
0.573 0.565
0.554
0.544
0.533
0.579
0.582 0.578
0.567
0.561 0.552
0.585 0.586 0.585
0.575
0.571 0.570
0.597 0.591 0.598
0.597 0.599 0.593
Figure 6: Visualizations of Drrain when
W 0.01
0.593
0.599 0.597
0.596
0.600 0.598
N 64 and w 0.01. Positive points in red and negative points in blue.
=
0.594 0.591
0.591
0.590
0.597 0.601
6 CONCLUSION
In this paper, we presented shape representation for deep neural networks. Training data generating networks establish a connection between few-shot learning and shape representation by converting an image of a shape into a collection of points as training set for supervised task. Training can be solved with meta-learning approaches for few-shot learning. While our solution is inspired by few-shot learning, is different in: 1) our training datasets are generated by separate network and not given directly; 2) our embedding network is conditioned on the task but traditional few-shot learning employs unconditional embedding networks; 3) our test dataset is generated by sampling and not directly given. The experiments are evaluated on single image 3D reconstruction dataset and improve over the SOTA.
REFERENCES
Brandon Amos and Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 136-

---

Published as a conference paper at ICLR 2022
145. JMLR. org. 2017.
Luca Bertinetto, João F. Henriques, Philip H. S. Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.net, 2019. URL https:
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv: 1512.03511. 2015.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5939-5948, 2019.
Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang, Bsp-net: Generating compact meshes via binary space partitioning. arXiv preprint arXiv: /9/ 1.06971, 2019.
Christopher B Choy, Danfei Xu, Jun Young Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and muet 3d object reconstruction. In European conference computer vision, pp. 628-644- Springer, 2016.
Koby Crammer and Yoram Singer. On the algorithmic implementation of multiclass kernel-based vector machines. Journal of machine learning research, 20De2):265-292, 2001.
Boyang Deng, Kyle Genova, Soroosh Yazdani, Sofien Bouaziz, Geoffrey Hinton, and Andrea Tagliasacchi. Cvxnets: Learnable convex decomposition. arXiv preprint arXiv:1909.057550 2019.
Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object recon- struction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 605-613, 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. arXiv preprint /003.::7770.3444:. 2017.
Kylc Genova, Forrester Cole, Daniel Vlasic, Aaron Sama,, William T Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In Proceedings of the IEEE International Conference on Computer Vision, pp. 7154-7164, 2019.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4367- 4375, 2018
Stephen Gould, Basura Fernando, Anoop Cherian, Peter Anderson, Rodrigo Santa Cruz, and Edison Guo. On differentiating parameterized argmin and argmax problems with application bi-level optimization. arXiv preprint arXiv:1607.05444 2016.
Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. A papier-måché approach to learning 3d surface generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 216-224, 2018.
David Ha, Andrew Dai, and Quoc V Le. Hypernetworss. arXiv preprint 1xXiv:1609.09100 2016.
Rana Hanocka, Amir Hertz, Noa Fish, Raja Giryes, Shachar Fleishman, and Daniel Cohen-Or. Meshcnn: a network with an edge. ACM Transactions on Graphics (TOG), 38(4):1-12, 2019.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semanties. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7482-7491, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2. Lille, 2015.
10

---

Published as a conference paper at ICLR 2022
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 10657-10665, 2019.
Xiaodan Liang, Liang Lin, Yunchao Wei, Xiaohui Shen, Jianchao Yang. and Shuicheng Yan. Proposal-free free network for instance-lever object segmentation. IEEE transactions on pattern anal- ysis and machine intelligence, 40(12):2978-2999 2017.
Chen-Hsuan Lin, Chen Kong, and Simon Lucey. Learning efficient point cloud generation for dense 3d object reconstruction. arXiv preprint
2017.
Gidi Littwin and Lior Wolf. Deep meta functionals for shape representation. In Proceedings the IEEE International Conference on Computer Vision, pp. 1824-1833, 2019.
William Lorensen and Harvey E. Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics, 21(4):163-19, 1987.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4460-4470, 2019.
Mateusz Michalkiewicz, Jhony K Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders Eriks- son. Deep level sets: Implicit surface representations for 3d shape inference. arXiv preprint arXiv:1901.0680) 2019.
Kevin P Murphy. Machine learning: a probabilistic perspective, chapter 14.4.3. The MIT Press, 2012.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint SIIIv:I8033029999 2018.
David Novotny, Samuel Albanic, Diane Larlus, and Andrea Vedaldi. Semi-convolutional opera- tors for instance segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 86- 102, 2018
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 165-174, 2019.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 652-660, 2017a
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical
feature learning on point sets in a metric space. Advances neural information processing systems, pp. 5099-5108, 2017b.
Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan Yuille. Few-shot image recognition by predicting parameters from activations. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 7229-7238, 2018
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In 5th Interna- tional Conference on Leurning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL lattps:// net/
Eorum?id=xJY> Kcll
Gemoo Riegler, Ali Osman Ulusoy, and Andreas Geiger. Octnet: Learning deep 3d representations at high resolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3577-3586, 2017.
Vincent Sitzmann, Eric R. Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein. Metasdf: Meta-leaming signed distance functions, In Proc. NeurIPS, 2020.
11

---

Published as a conference paper at ICLR 2022
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pp. 4077-4087, 2017.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net-
works. In International Conference on Machine Learning, pp. 1105-6114. PMLR, 2019.
Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul Srinivasan, Jonathan Barron, and Ren Ng. Learned initializations for optimizing coordinate-basee neural representa- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2846-2855, 2021.
Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox. Octree generating networks: Efficient convolutional architectures for high-resolution 3d outputs. In Proceedings of the IEEE Interna- tional Conference on Computer Vision, pp. 2088-2096, 2017.
Maxim Tatarchenko, Stephan Richter, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox. What do single-view 3d reconstruction networks learn? In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 3405-3414, 2019.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016.
Nanyang Wang. Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 32-67, 2018.
Peng-Shun Soug-Shai Wang, Yang Liu, Yu-Xiao Guo, Chun-Yu Sun, and Xin Tong. O-enn: Octree-based convolutional neural networks for 3d shape analysis. ACM Transactions on Graphics (TOG), 36 (4):1-11, 2017.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912-1920, 2015.
Biao Zhang and Peter Wonka. Point cloud instance segmentation using probabilistic embeddings. arXiv preprint arXiv: 192.001992200145,,000 2019.
A
APPENDIX
A.1
NETWORKS
We show the architecture of the embedding network and the points generator in Fig. The embed- ding network is similar to DeepSDF (Park et al. 2019).
253
256
3
256
3
512
512
512
512
512
512
1024
1024
1024
(a) Embedding Network
(b) Points Generator
Figure 7: Networks. We use fully connected layers in both networks. In (a) the input is the concatenation of feature À (yellow) and point x (blue). In (b), the input is the feature À (yellow), and the output is the coordinates of points X E R.3 (red).
12

---

Published as a conference paper at ICLR 2022
sofa
airplane
lamp
telephone
vessel
loudspeaker
chair
cabinet
table
display
car
bench
rifle
Figure 8: T-SNE visualization using the extracted features of the shapes.
A.2 VISUALIZATION OF FEATURE SPACE
We visualize the feature space in Fig.
A.3 STATISTICS OF METRICS
Here we show more detailed statistics in Fig. 9
Comparing to IoU and F-score, most of the val- of Chamfer concentrate around the mean value and rare values very far from the mean. The "peakedness" property of Chamfer implies it is unstable to certain extent. The conclusion is also consistent with Tatarchenko al. (2019).
A.4 SHAPE INTERPOLATION
In Fig. 10
we show reconstructed shapes with bilinear interpolation. Given shape latent fea- ture vectors Ao Ao, Åι,0 and ÅI we show interpolated shapes with the bilinear interpolation
equation
Any (1 - (1 h))o,0 (1 (1-v)ha v)h v(1 A)A1. vhA1,1 where v E [0, 1] and [0, 1]
A.5
EFFECTS OF w
In the main paper, we have regularizer to force the embedding space to be similar the original space,
x|12 -
(10)
When is very large, the embedding space almost the same the original space, especially when N is large. The visualization is shown in Fig. [T. We set w 10. This is much higher than the paper. In this case, we expect the embedding network to do less 10 the method mainly works for a large number of points. We can
w 0.01 recommended work and as reported for observe that indeed the embedding space (bottom) similar to the original space (top) especially when N is large. Positive points shown in red and negative points in blue. Top row: meshes are reconstructed with our regular pipeline. Generated points are fed into the Embedding Network first. Visualized points are in the original space. Middle row: meshes reconstructed by removing
13

---

Published as a conference paper at ICLR 2022
1.2
1.0
I0U
0.4
0.2
0.0
-0.2
airplane
bench
cabinet
car
cheir
display
lamp
loudspeaker
rifle
sofa
table
telephone
vessel
(a) loU
2.0
Chamfer
1.5
1.0
0.5
0.0
airplane
bench
cabinet
car
chair
disptay
lamp
loudspeaker
rifle
sofa
table
telephone
vessel
(b) Chamfer-LI
1.2
1.0
0.8
F-score
0.6
0.4
0.0
-02
airplane
bench
cabinet
car
chair
displey
lamp
loudspeaker
rifie
sofa
table
telephone
vessel
(c) F-score
Figure 9: Statisties of metric.
14

---

Published as a conference paper at ICLR 2022
[0, 1]
o
A
A1.0
A
Figure 10: Reconstruction with bilinear interpolated feature vectors. Meshes blue are orig- inal, green meshes are interpolated. The bilinear equation is Xug (1 v) A))o, + (1 v)/ + v(1 A + 1hAr,1
15

---

Published as a conference paper at ICLR 2022
IoU f
Chamfer
F-Score 1
Category
SVM
RR
SVM
RR
SVM
airplane
0.639
0.633
0.117
0.121
71.87
71.15
bench
0.522
0.524
0.144
0.132
67.88
69.44
cabinet
0.731
0.732
0.139
0,141
65.15
65,33
car
0.745
0.748
0.119
0.121
65.77
65.48
chair
0.526
0.532
0.229
0.205
47.73
48.83
display
0.552
0.553
0.213
0.215
47.88
46.96
lamp
0.388
0.383
0.398
0.391
41.94
42.99
loudspeaker
0.662
0.658
0.252
0.247
46.95
46.86
rifle
0.536
0.540
0.112
0.111
69.18
69.40
0,700
0.707
0.158
0.155
55.40
56.40
table
0.538
0.551
0.193
0.171
64.03
telephone
0.776
0.779
0.101
0.106
76.60
75.96
vessel
0.563
0.567
0.177
0.186
51.38
51.57
mean 0.606
0.608
0.181
0.177
59.36
59.66
Table 4: Results of different base learners: SVM Ridge Regression (RR).
the embedding network without re-training. This isn't expected work well. Generated points are directly used in generating classification boundaries. Visualized points are in the original space. We can see that even without the embedding network the results are surprisingly reasonable for large number of points. Bottom row: meshes in the embedding space. Visualized points are in the embedding space. We can observe that the meshes look similar the correct chair, especially for large N. This also validates that the embedding network does not have large influence on the result for large w. If the regularizing weight w is set to O, the shape in embedding space will be more ellipsoid like.
Recall that, with the kernel ridge regression as our base learner, the final shape (decision) surface is decided by,
N
Σ
(g(()
If the original and embedding space are close, reconstruct meshes without the Embed- ding Network g.
Σ
0g (x;,,)
The results are shown in Fig. 11
A.6
THE CHOICR OF BASE LEA.NERSS
In Table 4
we show results of different base learners, SVM and ridge regression. The difference between the two methods is very small.
A.7 RESULTS ON REAL WORLD IMAGES
Note that our model is trained on synthetic dataset (ShapeNet). Here we test how our model generalizes to real world datasets. Without retraining and fine-tuning, we apply our trained model to Stanford Online Products. The qualitative results are shown in Fig. [2] We believe that the results are good, but there no ground truth data for evaluation.
A.8 INTER ANTER-CATERR CATEGORIES INTERPOLATION
Similar to Fig. we also show inter-categories interpolation in Fig. 13 two different categories, and reconstruct shapes with interpolated latents.
We take samples from Although the interpo-
16

---

Published as a conference paper at ICLR 2022
o.K ( (x
N 8
= 16
N = 32
N = 64
N 128
N 256
Σ Ea.( (x. x)
(x.. 1 D
l
Σ CaK (e.
(es. w)
Figure 11: Effects of large
Figure 12: Visual results for Stanford Online Products. The input images are shown in top row. They are resized to 137 x 137 to fit the input requirement of our model.
17

---

Published as a conference paper at ICLR 2022
Figure 13: Reconstruction with interpolated feature vectors. Meshes in blue are original, green meshes interpolated.
lated shapes are not high quality than the originals, they still show some characteristics of both categories, e.g., sofas which are as tall as tables, and cars with chair-leg- -ikee tires.
A.9 TRAINING AND INFERENCE
This algorithm be viewed along with Fig. 2 We also show the full process how we generate triangular mesh given an input image in Algorithm
We describe the detailed training process in Algorithm T
Algorithm Training
for number training iterations do
Sample a minibatch of m training sample pairs (I, M,) is the target ground-truth mesh. Get the latent representations for images with a feature backbone
6=1- where I, is an image and Mg
Feature (I Bx3xxx* L IR Bx256
Get the labeled point sets (the first half are marked as positive and the other half are marked as negative)
PointGen(L) X Bx
y B
Embed the point X another space
Embedding(X). R BxNx3
Optimize differentiable base learner P--lax which defines binary classification boundaries,
OptBaseLearnerrPP E,y) E B
Sample labeled point sets from ground-truth meshes (M;) 1b=1
X, E Bxx33
Yc E P BxM BxM.
Convert X, to the embedding space
Embedding (X,) = E, BBNx3
Predict labels by using base learner P(-Jar
P(E,(a) = Vt:
Update Feature(-). PointGen(-) and Embedding(-) by descending stochastic gradients VMSE(y,,.)),
end for
18

---

Published as a conference paper at ICLR 2022
Algorithm Inference
Prepare an input image I E xx
Get the latent representations for the image
Feature (I) L E R
Get the labeled point sets (the first half are marked as postive and the other half are marked as negative)
PointGen(()) X E Rx3
YERN
Embed points to the embedding space
Embedding X) Nx3
Optimize the differentiable base learner P(-la:.
Opt Oet (P, E,y) = a E R
Prepare point set on 3D grid with arbitrary resolution G G G.
X₁ E x3
Convert X₃ to embedding space
Embedding (X.) E, RO'x3
Predict labels by using base learner P(-aa)
P(Elja) Yr E GxGxG
Apply marching cubes algorithm y.,
MarehingCubes((e) = (V,F).
19

---

Published as a conference paper at ICLR 2022
Table 5: Different backbones. ENB1: EfficientNet-Bl. RN34: ResNet-34. RN50: Resnet-50.
DN121: DenseNet-121
IoU f
Chamfer L
F-Score t
Categ.
ENB1
RN34
RN50
DN121
ENB1 1
RN34
RN50
DN121
ENB1
RN34
RN50
DN121
plane
0.633
0.616
0.623
0.612
0.121
0.129
0.131
0.142
71.15
69.15
69.70
67.77
bench
0.524
0.506
0.500
0.508
0.132
0.135
0.152
0.145
69.44
67.24
65.69
66.32
cabinet
0.732
0.728
0.733
0.742
0.141
0.144
0.141
0.136
65.33
63.44
63.78
65.58
car
0.748
0.746
0.744
0.744
0.121
0.122
0.123
0.123
65.48
64.86
63.98
64.23
chair
0.532
0.520
0.521
0.532
0.205
0.213
0.219
0.216
48.83
47.33
47.14
48.11
display
0.553
0.549
0.534
0.554
0.215
0.209
0.222
0.210
46.96
46.45
43.83
46.85
lamp
0.383
0.378
0.370
0.386
0.391
0.399
0.434
0,410
42.99
41.38
39.49
41.81
speaker
0.658
0.647
0.647
0.659
0.247
0.262
0.265
0.252
46.86
44.47
43.77
46.52
rifle
0.540
0.521
0,515
0,504
0.111
0.124
0.118
0.124
69.40
67.19
66.47
65.09
sofa
0.707
0.691
0.694
0.702
0.155
0.164
0.166
0.155
56,40
54.32
54.18
55.62
table
0.551
0.535
0.532
0.550
0.171
0.177
0.200
0.176
65.25
63.92
63.26
65.22
phone
0.779
0.770
0.769
0.770
0.106
0.105
0.107
0.105
75.96
75.22
73.06
75.30
vessel
0.567
0.562
0.555
0.570
0.186
0.185
0.206
0.192
51.57
50.27
48.74
50.68
mean
0.608
0.598 0.595
0.602
0.177
0.182
0.191 0.183
59.66
58.10
57.16
58.39
Table 6: Different embeddings. Spa: (3D) spatial embeddings. Other columns: trained with linear ridge regression of different different embedding dimensions (32, 64 and 128).
IoU f
Chamfer 4
F-Score ?
Categ.
Spa
32D
64D
128D
Spa
32D
64D
128D
Spa
32D
64D
128D
plane
0.637
0.627
0.630
0.624
0.119
0.125
0.132
0.123
71.54
70.67
70.91
70.30
bench
0.519
0.530
0.532
0.516
0.138
0.135
0.138
0.133
68.42
68.89
69.57
68.23
cabinet
0.744
0.735
0.739
0.736
0.135
0.136
0.138
0.134
66.54
66.92
67.13
66.77
0.747
0.746
0.749
0.746
0.119
0.121
0.118
0.119
65.51
65.53
65.98
65.50
chair
0.527
0.536
0.542
0.532
0.223
0.233
0.210
0.207
47.89
49.05
49.93
48.88
display
0.572
0.562
0.570
0.564
0.201
0,201
0.196
0.201
48.58
48.00
48.73
47.76
lamp
0.387
0.397
0.417
0.408
0.398
0.479
0.456
0.363
42.57
43.78
45.18
44.29
speaker
0.657
0.666
0.666
0.660
0.259
0.244
0.233
0.242
45.83
48.19
48.36
48.21
rifle
0.537
0.540
0.545
0.537
0.113
0.116
0.113
0.109
69.33
69.25
69.81
68.77
sofa
0.705
0.706
0.707
0.707
0.157
0.153
0.153
0.152
55,90
$6.49
56.73
$6.53
table
0.549
0.550
0,553
0.548
0.183
0.177
0.172
0.169
65.08
65.77
66.16
64.96
phone
0.779
0.780
0.786
0.777
0.107
0.100
0.094
0.097
76.95
76.83
76.29
76.06
vessel
0.575
0.572
0.577
0.569
0.189
0.199
0.189
0.183
51.75
52.10
52.03
51.84
mean
0.610
0.611
0.616
0.610
0.180
0.186
0.180
0.172
59.68
60.11
60.52
59.85
A.10
CHOICE OF BACKBONES
In Table 5] we show the performance of our method with different backbones. We keep all of the setup the same except for the feature backbone.
A.11 EMBEDDING DIMENSIONS
In T show the performance of our method with different embeddings. Note that different from Table the results shown here are trained with N = 256.
20


Reviews:
[{"review_id": "vjpnvxb34B", "reviewer": "Reviewer_DXHG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths of the paper:\n- The paper is clearly written, and one can implement the proposed approach using the algorithm description in the paper.\n- It proposes a new technique to reconstruct 3D shapes from 2D images, which obtains results comparable or better (in terms of standard metrics) than state of the art on a subset of the ShapeNet dataset.\n- The proposed reconstruction problem decomposition (described in the previous section) is interesting and novel, albeit not clearly motivated (See paper weaknesses description).\n\nWeaknesses of the paper:\n- The paper does not clearly articulate the motivation for the proposed approach, i.e. why it is beneficial to formulate the reconstruction problem as a meta-learning problem. It should also provide a more in-depth comparison of the proposed method with the existing methods which directly predict the implicit shape representation, in order to explain why the proposed formulation outperforms these methods.\n- I may be missing it - but nowhere in the paper there is a clear description of how the inference is performed, after the network has been trained. Please add it, e.g., at the end of Section 3.2.\n- In the proposed method description at the beginning of Section 3, it is worth clarifying that the output of the Point Generation Network is in fact the training set for the second part of the network, as mentioned in the introduction.\n- It would be interesting to perform an ablation study on dimension of the embedding space.\n- Figure 10 present an example of shape interpolation. Have the authors tried to interpolate shapes with different topologies and were the results plausible?\n\nSmall comments and typos:\n- There is a wrong line break at the beginning of Section 5.\n- The caption for Figure 11 is too long - it should be shortened and the text should be moved to the corresponding appendix.", "summary_of_the_paper": "The paper proposes a method for 3D shape reconstruction from image using a bi-level optimization technique. Specifically, the first part of the proposed network extract features from the image and creates a set of 3D points sampled near the surface of the shape to be reconstructed with labels corresponding to the points being inside or outside the 3D shape boundary. Then, the produced sets of points are fed to the second part of the network which is trained to approximate an implicit representation of the shape. At the inference time, a 2D surface of the shape can be extracted from the predicted implicit representation.\n", "summary_of_the_review": "The paper proposes a novel method, inspired by meta-learning algorithms, for 3D shape reconstruction from images. The proposed method produces results which are comparable or better than the current state of the art approaches. On the other hand, the paper does not provide  a clear explanation why the proposed approach is suitable for the problem of shape reconstruction and why it is expected to perform better than approaches employing a single network. Adding such an explanation, and a more in-depth analysis of the improvements brought by the bi-level optimization will significantly improve the paper. Currently I recommend \"weak accept\".", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "G-oopGhOTx", "reviewer": "Reviewer_7UAm", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Please Bist both the strengths and weaknesses of the paper. When discussing weakness, please provide concrete. actionable feedback on the paper.\n\nTo avoid fully sample points and its distance value to the surface over the 3D space in training, the paper proposes a meta learning approach. It is a novel idea in computer graphics because different from datasets in computer vision or natural language processing, 3D shape dataset is far smaller. However, I still have several concerns. \n\n- The proposed architecture uses a point generation network to generate training data for few shot learning. I am wondering why not directly sample a few points on and near the shape surface to train the few shot network?\n\n- Predicted points are not guaranteed to be exactly those points you have sampled in the offline setting. So how do you get the occupancy value at these points in the training process?\n\n- Even though the author gives the reason for R^3 embedding space, I am wondering if how the embedding space dimension will influence the results.", "summary_of_the_paper": "The paper introduce a few shot learning method for single view reconstruction task. Different from traditional few shot learning methods, data used for training are generated by a network which takes an image as input. Both qualitative and quantitative reconstruction results show that this method gives a better performance than current SOTA methods at less training data(occupancy or distance field ground truth). \n", "summary_of_the_review": "I am generally positive about this paper since it proposes a novel method to single view reconstruction using few shot learning. But it needs a more detailed illustration about the training process and more evaluations on the design choice. If the authors can resolve those issues I mentioned above, I am willing to give a higher score.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1wsHDFUnUlY", "reviewer": "Reviewer_ABfL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a good work. There are several strengths. \n1) The idea that converting shape reconstruction from image into few-shot learning problem is novel.\n2) The authors generalize the differential learner with kernelized algorithms.\n3) The results are better then previous works.\n\nThere are also some problems needed feedback from the authors.\na) The author use EfficientNetB1 as the Feature Network, and how about utilizing other architecture for the Feature Network? Such as ResNet? \nb) The author  generalize the differential learner with kernelized algorithms. There should be experimental result that without kernel skills.\nc) More (visual) results on other dataset are needed, such as KITTI and Online Products datasets. ", "summary_of_the_paper": "In this paper, the authors provide a novel idea for shape reconstruction from a single image. They  enable the idea and techniques developed in the few-shot learning literature to the problem of shape reconstruction. They construct the meta-train data by a point generation network, and utilize a differential learner to learn the decision boundary.  They model the reconstruct problem by bi-level optimization.  It is an interesting work. The results are also good. ", "summary_of_the_review": "This is a good paper with novel idea. More results are needed to make it better.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers recommended accept after author responses. AC doesn't find any reason to overturn this consensus.
Decision: Accept (Poster)