Bowtie Networks: Generative Modeling for Joint Few-Shot Recognition and Novel-View Synthesis

1 INTRODUCTION . Given a never-before-seen object ( e.g. , a gadwall in Figure 1 ) , humans are able to generalize even from a single image of this object in different ways , including recognizing new object instances and imagining what the object would look like from different viewpoints . Achieving similar levels of generalization for machines is a fundamental problem in computer vision , and has been actively explored in areas such as few-shot object recognition ( Fei-Fei et al. , 2006 ; Vinyals et al. , 2016 ; Wang & Hebert , 2016 ; Finn et al. , 2017 ; Snell et al. , 2017 ) and novel-view synthesis ( Park et al. , 2017 ; Nguyen-Phuoc et al. , 2018 ; Sitzmann et al. , 2019 ) . However , such exploration is often limited in separate areas with specialized algorithms but not jointly . We argue that synthesizing images and recognizing them are inherently interconnected with each other . Being able to simultaneously address both tasks with a single model is a crucial step toward human-level generalization . This requires learning a richer , shareable internal representation for more comprehensive object understanding than it could be within individual tasks . Such “ cross-task ” knowledge becomes particularly critical in the low-data regime , where identifying 3D geometric structures of input images facilities recognizing their semantic categories , and vice versa . Inspired by this insight , here we propose a novel task of joint few-shot recognition and novel-view synthesis : given only one or few images of a novel object from arbitrary views with only category annotation , we aim to simultaneously learn an object classifier and generate images of that type of object from new viewpoints . This joint task is challenging , because of its ( i ) weak supervision , where we do not have access to any 3D supervision , and ( ii ) few-shot setting , where we need to effectively learn both 3D geometric and semantic representations from minimal data . While existing work copes with two or more tasks mainly by multi-task learning or meta-learning of a shared feature representation ( Yu et al. , 2020 ; Zamir et al. , 2018 ; Lake et al. , 2015 ) , we take a different perspective in this paper . Motivated by the nature of our problem , we focus on the interaction and cooperation between a generative model ( for view synthesis ) and a discriminative model ( for recognition ) , in a way that facilitates knowledge to flow across tasks in complementary directions , thus making the tasks help each other . For example , the synthesized images produced by the generative model provide viewpoint variations and could be used as additional training data to build a better recognition model ; meanwhile , the recognition model ensures the preservation of the desired category information and deals with partial occlusions during the synthesis . To this end , we propose a feedback-based bowtie network ( FBNet ) , as illustrated in Figure 1 . The network consists of a view synthesis module and a recognition module , which are linked through feedback connections in a bowtie fashion . This is a general architecture that can be used on top of any view synthesis model and any recognition model . The view synthesis module explicitly learns a 3D geometric representation from 2D images , which is transformed to target viewpoints , projected to 2D features , and rendered to generate images . The recognition module then leverages these synthesized images from different views together with the original real images to learn a semantic feature representation and produce corresponding classifiers , leading to the feedback from the output of the view synthesis module to the input of the recognition module . The semantic features of real images extracted from the recognition module are further fed into the view synthesis module as conditional inputs , leading to the feedback from the output of the recognition module to the input of the view synthesis module . One potential difficulty , when combining the view synthesis and the recognition modules , lies in the mismatch in their level of image resolutions . Deep recognition models can benefit from highresolution images , and the recognition performance greatly improves with increased resolution ( Wang et al. , 2016 ; Cai et al. , 2019 ; He et al. , 2016 ) . By contrast , it is still challenging for modern generative models to synthesize very high-resolution images ( Regmi & Borji , 2018 ; Nguyen-Phuoc et al. , 2019 ) . To address this challenge , while operating on a resolution consistent with state-of-the-art view synthesis models ( Nguyen-Phuoc et al. , 2019 ) , we further introduce resolution distillation to leverage additional knowledge in a recognition model that is learned from higher-resolution images . Our contributions are three-folds . ( 1 ) We introduce a new problem of simultaneous few-shot recognition and novel-view synthesis , and address it from a novel perspective of cooperating generative and discriminative modeling . ( 2 ) We propose feedback-based bowtie networks that jointly learn 3D geometric and semantic representations with feedback in the loop . We further address the mismatch issue between different modules by leveraging resolution distillation . ( 3 ) Our approach significantly improves both view synthesis and recognition performance , especially in the low-data regime , by enabling direct manipulation of view , shape , appearance , and semantics in generative image modeling . 2 RELATED WORK . Few-Shot Recognition is a classic problem in computer vision ( Thrun , 1996 ; Fei-Fei et al. , 2006 ) . Many algorithms have been proposed to address this problem ( Vinyals et al. , 2016 ; Wang & Hebert , 2016 ; Finn et al. , 2017 ; Snell et al. , 2017 ) , including the recent efforts on leveraging generative models ( Li et al. , 2015 ; Wang et al. , 2018 ; Schwartz et al. , 2018 ; Zhang et al. , 2018 ; Tsutsui et al. , 2019 ; Chen et al. , 2019b ; Li et al. , 2019 ; Zhang et al. , 2019 ; Sun et al. , 2019 ) . A hallucinator is introduced to generate additional examples in a pre-trained feature space as data augmentation to help with low-shot classification ( Wang et al. , 2018 ) . MetaGAN improves few-shot recognition by producing fake images as a new category ( Zhang et al. , 2018 ) . However , these methods either do not synthesize images directly or use a pre-trained generative model that is not optimized towards the downstream task . By contrast , our approach performs joint training of recognition and view synthesis , and enables the two tasks to cooperate through feedback connections . In addition , while there has been work considering both classification and exemplar generation in the few-shot regime , such investigation focuses on simple domains like handwritten characters ( Lake et al. , 2015 ) but we address more realistic scenarios with natural images . Note that our effort is largely orthogonal to designing the best few-shot recognition or novel-view synthesis method ; instead , we show that the joint model outperforms the original methods addressing each task in isolation . Novel-View Synthesis aims to generate a target image with an arbitrary camera pose from one given source image ( Tucker & Snavely , 2020 ) . It is also known as “ multiview synthesis. ” For this task , some approaches are able to synthesize lifelike images ( Park et al. , 2017 ; Yin & Shi , 2018 ; Nguyen-Phuoc et al. , 2018 ; Sitzmann et al. , 2019 ; Iqbal et al. , 2020 ; Yoon et al. , 2020 ; Wiles et al. , 2020 ; Wortsman et al. , 2020 ) . However , they heavily rely on pose supervision or 3D annotation , which is not applicable in our case . An alternative way is to learn a view synthesis model in an unsupervised manner . Pix2Shape learns an implicit 3D scene representation by generating a 2.5D surfel based reconstruction ( Rajeswar et al. , 2020 ) . HoloGAN proposes an unsupervised approach to learn 3D feature representations and render 2D images accordingly ( Nguyen-Phuoc et al. , 2019 ) . Nguyen-Phuoc et al . ( 2020 ) learn scene representations from 2D unlabeled images through foreground-background fragmenting . Different from them , not only can our view synthesis module learn from weakly labeled images , but it also enables conditional synthesis to facilitate recognition . Feedback-Based Architectures , where the full or partial output of a system is routed back into the input as part of an iterative cause-and-effect process ( Ford , 1999 ) , have been recently introduced into neural networks ( Belagiannis & Zisserman , 2017 ; Zamir et al. , 2017 ; Yang et al. , 2018 ) . Compared with prior work , our FBNet contains two complete sub-networks , and the output of each module is fed into the other as one of the inputs . Therefore , FBNet is essentially a bi-directional feedback-based framework which optimizes the two sub-networks jointly . Multi-task Learning focuses on optimizing a collection of tasks jointly ( Misra et al. , 2016 ; Ruder , 2017 ; Kendall et al. , 2018 ; Pal & Balasubramanian , 2019 ; Xiao & Marlet , 2020 ) . Task relationships have also been studied ( Zamir et al. , 2018 ; Standley et al. , 2020 ) . Some recent work investigates the connection between recognition and view synthesis , and makes some attempt to combine them together ( Sun et al. , 2018 ; Wang et al. , 2018 ; Xian et al. , 2019 ; Santurkar et al. , 2019 ; Xiong et al. , 2020 ; Michalkiewicz et al. , 2020 ) . For example , Xiong et al . ( 2020 ) use multiview images to tackle fine-grained recognition tasks . However , their method needs strong pose supervision to train the view synthesis model , while we do not . Also , these approaches do not treat the two tasks of equal importance , i.e. , one task as an auxiliary task to facilitate the other . On the contrary , our approach targets the joint learning of the two tasks and improves both of their performance . Importantly , we focus on learning a shared generative model , rather than a shared feature representation as is normally the case in multi-task learning . Joint Data Augmentation and Task Model Learning leverage generative networks to improve other visual tasks ( Peng et al. , 2018 ; Hu et al. , 2019 ; Luo et al. , 2020 ; Zhang et al. , 2020 ) . A generative network and a discriminative pose estimation network are trained jointly through adversarial loss in Peng et al . ( 2018 ) , where the generative network performs data augmentation to facilitate the downstream pose estimation task . Luo et al . ( 2020 ) design a controllable data augmentation method for robust text recognition , which is achieved by tracking and refining the moving state of the control points . Zhang et al . ( 2020 ) study and make use of the relationship among facial expression recognition , face alignment , and face synthesis to improve training . Mustikovela et al . ( 2020 ) leverage a generative model to boost viewpoint estimation . The main difference is that we focus on the joint task of synthesis and recognition and achieve bi-directional feedback , while existing work only considers optimizing the target discriminative task using adversarial training or with a feedforward network . 3 OUR APPROACH . 3.1 JOINT TASK OF FEW-SHOT RECOGNITION AND NOVEL-VIEW SYNTHESIS . Problem Formulation : Given a dataset D = { ( xi , yi ) } , where xi ∈ X is an image of an object and yi ∈ C is the corresponding category label ( X and C are the image space and label space , respectively ) , we address the following two tasks simultaneously . ( i ) Object recognition : learning a discriminative model R : X → C that takes as input an image xi and predicts its category label . ( ii ) Novel-view synthesis : learning a generative model G : X ×Θ → X that , given an image xi of category yi and an arbitrary 3D viewpoint θj ∈ Θ , synthesizes an image in category yi viewed from θj . Notice that we are more interested in category-level consistency , for which G is able to generate images of not only the instance xi but also other objects of the category yi from different viewpoints . This joint-task scenario requires us to improve the performance of both 2D and 3D tasks under weak supervision without any ground-truth 3D annotations . Hence , we need to exploit the cooperation between them . Few-Shot Setting : The few-shot dataset consists of one or only a few images per category , which makes our problem even more challenging . To this end , following the recent work on knowledge transfer and few-shot learning ( Hariharan & Girshick , 2017 ; Chen et al. , 2019a ) , we leverage a set of “ base ” classes Cbase with a large-sample dataset Dbase = { ( xi , yi ) , yi ∈ Cbase } to train our initial model . We then fine-tune the pre-trained model on our target “ novel ” classes Cnovel ( Cbase∩Cnovel = 0 ) with its small-sample dataset Dnovel = { ( xi , yi ) , yi ∈ Cnovel } ( e.g. , a K-shot setting corresponds to K images per class ) .

Summary: This paper proposes a "feedback-based bowtie network" FBNet for joint generative synthesis via a GAN-based framework (specifically HoloGAN) and few-shot fine-grained recognition. The key idea of this work is to supervise both networks jointly via feedback mechanisms between the two, which helps to improve both tasks: image synthesis and few-shot recognition. The authors propose to use the synthesis network for synthesizing augmented images and additional losses computed by the image classification network along with conditional generation to improve the quality of the synthesized images.