Training Data Generating Networks: Shape Reconstruction via Bi-level Optimization

1 INTRODUCTION . Neural networks have shown promising results for shape reconstruction ( Wang et al. , 2018 ; Groueix et al. , 2018 ; Mescheder et al. , 2019 ; Genova et al. , 2019 ) . Different from the image domain , there is no universally agreed upon way to represent 3d shapes . There exist many explicit and implicit representations . Explicit representations include point clouds ( Qi et al. , 2017a ; b ; Lin et al. , 2017 ) , grids ( Wu et al. , 2015 ; Choy et al. , 2016 ; Riegler et al. , 2017 ; Wang et al. , 2017 ; Tatarchenko et al. , 2017 ) , and meshes ( Wang et al. , 2018 ; Hanocka et al. , 2019 ) . Implicit representations ( Mescheder et al. , 2019 ; Michalkiewicz et al. , 2019 ; Park et al. , 2019 ) define shapes as iso-surfaces of functions . Both types of representations are important as they have different advantages . In this work , we set out to investigate how meta-learning can be used to learn shape representations . To employ meta-learning , we need to split the learning framework into two ( or more ) coupled learning problems . There are multiple design choices for exploring this idea . One related solution is to use a hypernetwork as the first learning algorithm that produces the weights for a second network . This approach ( Littwin & Wolf , 2019 ) , is very inspiring , but the results can still be improved by our work . The value of our work are not mainly the state-of-the-art results , but the new framework that enables the flow of new ideas , techniques , and methods developed in the context of few-shot learning and meta-optimization to be applied to shape reconstruction . In order to derive our proposed solution , we draw inspiration from few-shot learning . The formulation of few-shot learning we use here is similar to Lee et al . ( 2019 ) ; Bertinetto et al . ( 2019 ) , but also other formulations of few shot learning exist , e.g. , MAML ( Finn et al. , 2017 ) . In few-shot learning for image classification , the learning problem is split into multiple tasks ( See Fig . 1 ) . For each task , the learning framework has to learn a decision boundary to distinguish between a smaller set of classes . In order to do so , a first network is used to embed images into a high-dimensional feature space . A second learner ( often called base learner in the literature of meta learning approaches for few-shot learning ) is then used to learn decision boundaries in this high-dimensional feature space . A key point is that each task has separate decision boundaries . In order to build a bridge from fewshot learning to shape representations , we identify each shape as a separate task . For each task , there are two classes to distinguish : inside and outside . The link might seem a bit unintuitive at first , because in the image classification case , each task is defined by multiple images already divided into multiple classes . However , in our case , a task is only associated with a single image . We therefore introduce an additional component , a training data generating network , to build the bridge to fewshot learning ( See Fig . 2 ) . This training data generating network takes a single image as input and outputs multiple 3D points with an inside and outside label . These points define the training dataset for one task ( shape ) . We adopt this data set format to describe a task , because points are a natural shape representation . Similar to the image classification setting , we also employ an embedding network that maps the original training dataset ( a set of 3D points ) to another space where it is easier to find a decision boundary ( a distorted 3D space ) . In contrast to the image classification setting , the input and output spaces of the embedding network have a lot fewer dimensions , i.e . only three dimensions . Next , we have to tackle the base learner that takes a set of points in embedding space as input . Here , we can draw from multiple options . In particular , we experimented with kernel SVMs and kernel ridge regression . As both give similar results ( In meta learning ( Lee et al. , 2019 ) we can also find the same conclusion ) , we opted for ridge regression in our solution due to the much faster processing time . The goal of ridge regression is to learn a decision boundary ( shape surface ) in 3D space describing the shape . Our intermediate shape representation is a set of points , but these points can not be chosen arbitrarily . The set of points are specifically generated so that the second learning algorithm can process them to find a decision boundary . The second learning algorithm functions similar to other implicit shape representations discussed above . The result of the second learning algorithm ( SVM or ridge regression ) can be queried as an implicit function . Still the second learning algorithm can not be replaced by other existing networks employed in previous work . Therefore , among existing works , only our representation can directly use few-shot learning . It does not make sense to train our representation without a few-shot learning framework and no other representation can be directly trained with a few-shot learning framework . Therefore , all components in our framework are intrinsically linked and were designed together . Contributions .. • We propose a new type of shape representation , where we train a network to output a training set ( a set of labeled points ) for another machine learning algorithm ( Kernel Ridge Regression or Kernel-SVM ) . • We found an elegant way to map the problem of shape reconstruction from a single image to the problem of few-shot classification by introducing a network to generate training data . We combined training data generating networks with two meta learning approaches ( R2D2 ( Bertinetto et al. , 2019 ) and MetaOptNet ( Lee et al. , 2019 ) ) in our framework . Our work enables the application of ideas and techniques developed in the few-shot learning literature to the problem of shape reconstruction . • We validate our model using the problem of 3d shape reconstruction from a single image and improve upon the state of the art . 2 RELATED WORK . Neural implicit shape representations . We can identify two major categories of shape representations : explicit representations , where a shape can be explicitly defined ; and implicit representations , where a shape can be defined as iso-surface of a function ( signed distance function or indicator function ) . In the past decade , we have seen great success with neural network based explicit shape representations : voxel representations ( Wu et al. , 2015 ; Choy et al. , 2016 ; Riegler et al. , 2017 ; Wang et al. , 2017 ; Tatarchenko et al. , 2017 ) , point representations ( Qi et al. , 2017a ; b ; Fan et al. , 2017 ; Lin et al. , 2017 ) , and mesh representations ( Wang et al. , 2018 ; Hanocka et al. , 2019 ) ) . On the other hand , modeling implicit representations with neural networks has been a current trend , where usually a signed distance function or indicator function is parameterized by a neural network ( Mescheder et al. , 2019 ; Chen & Zhang , 2019 ; Michalkiewicz et al. , 2019 ; Park et al. , 2019 ) . More recent works learn a network that outputs intermediate parameters , e.g . CvxNet ( Deng et al. , 2019 ) and BSP-Net ( Chen et al. , 2019 ) learns to output half-spaces . We propose a novel type of shape representation , where the model outputs a training set of labeled points . Few-shot learning . There are two common meta learning approaches for few-shot learning : metric-based ( Koch et al. , 2015 ; Vinyals et al. , 2016 ; Snell et al. , 2017 ) , which aims to learn a metric for each task ; optimization-based ( Ravi & Larochelle , 2017 ; Finn et al. , 2017 ; Nichol et al. , 2018 ) , which is designed to learn with a few training samples by adjusting optimization algorithms . These approaches commonly have two parts , an embedding model for mapping an input to an embedding space , and a base learner for prediction . Qiao et al . ( 2018 ) ; Gidaris & Komodakis ( 2018 ) train a hypernetwork ( Ha et al. , 2016 ) to output weights of another network ( base learner ) . R2D2 ( Bertinetto et al. , 2019 ) showed that using a light-weight and differentiable base learner ( e.g . ridge regression ) leads to better results . To further develop the idea , MetaOptNet ( Lee et al. , 2019 ) used multi-class support vector machines ( Crammer & Singer , 2001 ) as base learner and incorporated differentiable optimization ( Amos & Kolter , 2017 ; Gould et al. , 2016 ) into the framework . Lee et al . ( 2019 ) also shows it can outperform hypernetwork-based methods . In our work , we propose a shape representation that is compatible with a few-shot classification framework so that we can utilize existing meta learning approaches . Specifically , we will use ridge regression and SVM as the base learner . The most relevant method to ours is Littwin & Wolf ( 2019 ) which adapts hypernetworks . However , as we discussed above , differentiable optimization methods ( Bertinetto et al. , 2019 ; Lee et al. , 2019 ) are generally better than hypernetworks . Besides that , meta learning has been applied to other fields in shape analysis , e.g. , both Sitzmann et al . ( 2020 ) and Tancik et al . ( 2021 ) propose to use MAML-like algorithms ( Finn et al. , 2017 ) to learn a weight initialization over an implicit shape spaces . 3 METHOD . The framework is shown in Fig . 2 . The network is mainly composed of 3 sub-networks . The Feature Network maps an input image to feature space . The resulting feature vector λ is then decoded by the Point Generation Network to a labeled point set { xi , yi } Ni=1 . The point set will be used as training data in our base learner later . After that the Embedding Network projects the point set into embedding space . The projected points ei and the labels are taken as the input of a binary classifer ( ridge regression or SVM ) parameterized by α . Finally , the framework is able to output the inside/outside label y of a query point x by projecting it into the embedding space and feeding it to the binary classifier . In the following subsections , we describe our method in more detail . First , we introduce the background of meta learning approaches for few-show learning ( Sec . 3.1 ) and establish a link between single image 3D reconstruction and few-shot learning ( Sec . 3.2 ) . We propose a problem formulation inspired by few-shot learning ( Sec . 3.3 ) and propose a solution in the following subsections . Specifically , we apply recently developed differentiable optimization .

Summary: In this paper, the authors provide a novel idea for shape reconstruction from a single image. They  enable the idea and techniques developed in the few-shot learning literature to the problem of shape reconstruction. They construct the meta-train data by a point generation network, and utilize a differential learner to learn the decision boundary.  They model the reconstruct problem by bi-level optimization.  It is an interesting work. The results are also good.