Training Data Generating Networks: Shape Reconstruction via Bi-level Optimization

1 INTRODUCTION . Neural networks have shown promising results for shape reconstruction ( Wang et al. , 2018 ; Groueix et al. , 2018 ; Mescheder et al. , 2019 ; Genova et al. , 2019 ) . Different from the image domain , there is no universally agreed upon way to represent 3d shapes . There exist many explicit and implicit representations . Explicit representations include point clouds ( Qi et al. , 2017a ; b ; Lin et al. , 2017 ) , grids ( Wu et al. , 2015 ; Choy et al. , 2016 ; Riegler et al. , 2017 ; Wang et al. , 2017 ; Tatarchenko et al. , 2017 ) , and meshes ( Wang et al. , 2018 ; Hanocka et al. , 2019 ) . Implicit representations ( Mescheder et al. , 2019 ; Michalkiewicz et al. , 2019 ; Park et al. , 2019 ) define shapes as iso-surfaces of functions . Both types of representations are important as they have different advantages . In this work , we set out to investigate how meta-learning can be used to learn shape representations . To employ meta-learning , we need to split the learning framework into two ( or more ) coupled learning problems . There are multiple design choices for exploring this idea . One related solution is to use a hypernetwork as the first learning algorithm that produces the weights for a second network . This approach ( Littwin & Wolf , 2019 ) , is very inspiring , but the results can still be improved by our work . The value of our work are not mainly the state-of-the-art results , but the new framework that enables the flow of new ideas , techniques , and methods developed in the context of few-shot learning and meta-optimization to be applied to shape reconstruction . In order to derive our proposed solution , we draw inspiration from few-shot learning . The formulation of few-shot learning we use here is similar to Lee et al . ( 2019 ) ; Bertinetto et al . ( 2019 ) , but also other formulations of few shot learning exist , e.g. , MAML ( Finn et al. , 2017 ) . In few-shot learning for image classification , the learning problem is split into multiple tasks ( See Fig . 1 ) . For each task , the learning framework has to learn a decision boundary to distinguish between a smaller set of classes . In order to do so , a first network is used to embed images into a high-dimensional feature space . A second learner ( often called base learner in the literature of meta learning approaches for few-shot learning ) is then used to learn decision boundaries in this high-dimensional feature space . A key point is that each task has separate decision boundaries . In order to build a bridge from fewshot learning to shape representations , we identify each shape as a separate task . For each task , there are two classes to distinguish : inside and outside . The link might seem a bit unintuitive at first , because in the image classification case , each task is defined by multiple images already divided into multiple classes . However , in our case , a task is only associated with a single image . We therefore introduce an additional component , a training data generating network , to build the bridge to fewshot learning ( See Fig . 2 ) . This training data generating network takes a single image as input and outputs multiple 3D points with an inside and outside label . These points define the training dataset for one task ( shape ) . We adopt this data set format to describe a task , because points are a natural shape representation . Similar to the image classification setting , we also employ an embedding network that maps the original training dataset ( a set of 3D points ) to another space where it is easier to find a decision boundary ( a distorted 3D space ) . In contrast to the image classification setting , the input and output spaces of the embedding network have a lot fewer dimensions , i.e . only three dimensions . Next , we have to tackle the base learner that takes a set of points in embedding space as input . Here , we can draw from multiple options . In particular , we experimented with kernel SVMs and kernel ridge regression . As both give similar results ( In meta learning ( Lee et al. , 2019 ) we can also find the same conclusion ) , we opted for ridge regression in our solution due to the much faster processing time . The goal of ridge regression is to learn a decision boundary ( shape surface ) in 3D space describing the shape . Our intermediate shape representation is a set of points , but these points can not be chosen arbitrarily . The set of points are specifically generated so that the second learning algorithm can process them to find a decision boundary . The second learning algorithm functions similar to other implicit shape representations discussed above . The result of the second learning algorithm ( SVM or ridge regression ) can be queried as an implicit function . Still the second learning algorithm can not be replaced by other existing networks employed in previous work . Therefore , among existing works , only our representation can directly use few-shot learning . It does not make sense to train our representation without a few-shot learning framework and no other representation can be directly trained with a few-shot learning framework . Therefore , all components in our framework are intrinsically linked and were designed together . Contributions .. • We propose a new type of shape representation , where we train a network to output a training set ( a set of labeled points ) for another machine learning algorithm ( Kernel Ridge Regression or Kernel-SVM ) . • We found an elegant way to map the problem of shape reconstruction from a single image to the problem of few-shot classification by introducing a network to generate training data . We combined training data generating networks with two meta learning approaches ( R2D2 ( Bertinetto et al. , 2019 ) and MetaOptNet ( Lee et al. , 2019 ) ) in our framework . Our work enables the application of ideas and techniques developed in the few-shot learning literature to the problem of shape reconstruction . • We validate our model using the problem of 3d shape reconstruction from a single image and improve upon the state of the art . 2 RELATED WORK . Neural implicit shape representations . We can identify two major categories of shape representations : explicit representations , where a shape can be explicitly defined ; and implicit representations , where a shape can be defined as iso-surface of a function ( signed distance function or indicator function ) . In the past decade , we have seen great success with neural network based explicit shape representations : voxel representations ( Wu et al. , 2015 ; Choy et al. , 2016 ; Riegler et al. , 2017 ; Wang et al. , 2017 ; Tatarchenko et al. , 2017 ) , point representations ( Qi et al. , 2017a ; b ; Fan et al. , 2017 ; Lin et al. , 2017 ) , and mesh representations ( Wang et al. , 2018 ; Hanocka et al. , 2019 ) ) . On the other hand , modeling implicit representations with neural networks has been a current trend , where usually a signed distance function or indicator function is parameterized by a neural network ( Mescheder et al. , 2019 ; Chen & Zhang , 2019 ; Michalkiewicz et al. , 2019 ; Park et al. , 2019 ) . More recent works learn a network that outputs intermediate parameters , e.g . CvxNet ( Deng et al. , 2019 ) and BSP-Net ( Chen et al. , 2019 ) learns to output half-spaces . We propose a novel type of shape representation , where the model outputs a training set of labeled points . Few-shot learning . There are two common meta learning approaches for few-shot learning : metric-based ( Koch et al. , 2015 ; Vinyals et al. , 2016 ; Snell et al. , 2017 ) , which aims to learn a metric for each task ; optimization-based ( Ravi & Larochelle , 2017 ; Finn et al. , 2017 ; Nichol et al. , 2018 ) , which is designed to learn with a few training samples by adjusting optimization algorithms . These approaches commonly have two parts , an embedding model for mapping an input to an embedding space , and a base learner for prediction . Qiao et al . ( 2018 ) ; Gidaris & Komodakis ( 2018 ) train a hypernetwork ( Ha et al. , 2016 ) to output weights of another network ( base learner ) . R2D2 ( Bertinetto et al. , 2019 ) showed that using a light-weight and differentiable base learner ( e.g . ridge regression ) leads to better results . To further develop the idea , MetaOptNet ( Lee et al. , 2019 ) used multi-class support vector machines ( Crammer & Singer , 2001 ) as base learner and incorporated differentiable optimization ( Amos & Kolter , 2017 ; Gould et al. , 2016 ) into the framework . Lee et al . ( 2019 ) also shows it can outperform hypernetwork-based methods . In our work , we propose a shape representation that is compatible with a few-shot classification framework so that we can utilize existing meta learning approaches . Specifically , we will use ridge regression and SVM as the base learner . The most relevant method to ours is Littwin & Wolf ( 2019 ) which adapts hypernetworks . However , as we discussed above , differentiable optimization methods ( Bertinetto et al. , 2019 ; Lee et al. , 2019 ) are generally better than hypernetworks . Besides that , meta learning has been applied to other fields in shape analysis , e.g. , both Sitzmann et al . ( 2020 ) and Tancik et al . ( 2021 ) propose to use MAML-like algorithms ( Finn et al. , 2017 ) to learn a weight initialization over an implicit shape spaces . 3 METHOD . The framework is shown in Fig . 2 . The network is mainly composed of 3 sub-networks . The Feature Network maps an input image to feature space . The resulting feature vector λ is then decoded by the Point Generation Network to a labeled point set { xi , yi } Ni=1 . The point set will be used as training data in our base learner later . After that the Embedding Network projects the point set into embedding space . The projected points ei and the labels are taken as the input of a binary classifer ( ridge regression or SVM ) parameterized by α . Finally , the framework is able to output the inside/outside label y of a query point x by projecting it into the embedding space and feeding it to the binary classifier . In the following subsections , we describe our method in more detail . First , we introduce the background of meta learning approaches for few-show learning ( Sec . 3.1 ) and establish a link between single image 3D reconstruction and few-shot learning ( Sec . 3.2 ) . We propose a problem formulation inspired by few-shot learning ( Sec . 3.3 ) and propose a solution in the following subsections . Specifically , we apply recently developed differentiable optimization .

Summary: The paper proposes a method for 3D shape reconstruction from image using a bi-level optimization technique. Specifically, the first part of the proposed network extract features from the image and creates a set of 3D points sampled near the surface of the shape to be reconstructed with labels corresponding to the points being inside or outside the 3D shape boundary. Then, the produced sets of points are fed to the second part of the network which is trained to approximate an implicit representation of the shape. At the inference time, a 2D surface of the shape can be extracted from the predicted implicit representation.