Bowtie Networks: Generative Modeling for Joint Few-Shot Recognition and Novel-View Synthesis

1 INTRODUCTION . Given a never-before-seen object ( e.g. , a gadwall in Figure 1 ) , humans are able to generalize even from a single image of this object in different ways , including recognizing new object instances and imagining what the object would look like from different viewpoints . Achieving similar levels of generalization for machines is a fundamental problem in computer vision , and has been actively explored in areas such as few-shot object recognition ( Fei-Fei et al. , 2006 ; Vinyals et al. , 2016 ; Wang & Hebert , 2016 ; Finn et al. , 2017 ; Snell et al. , 2017 ) and novel-view synthesis ( Park et al. , 2017 ; Nguyen-Phuoc et al. , 2018 ; Sitzmann et al. , 2019 ) . However , such exploration is often limited in separate areas with specialized algorithms but not jointly . We argue that synthesizing images and recognizing them are inherently interconnected with each other . Being able to simultaneously address both tasks with a single model is a crucial step toward human-level generalization . This requires learning a richer , shareable internal representation for more comprehensive object understanding than it could be within individual tasks . Such “ cross-task ” knowledge becomes particularly critical in the low-data regime , where identifying 3D geometric structures of input images facilities recognizing their semantic categories , and vice versa . Inspired by this insight , here we propose a novel task of joint few-shot recognition and novel-view synthesis : given only one or few images of a novel object from arbitrary views with only category annotation , we aim to simultaneously learn an object classifier and generate images of that type of object from new viewpoints . This joint task is challenging , because of its ( i ) weak supervision , where we do not have access to any 3D supervision , and ( ii ) few-shot setting , where we need to effectively learn both 3D geometric and semantic representations from minimal data . While existing work copes with two or more tasks mainly by multi-task learning or meta-learning of a shared feature representation ( Yu et al. , 2020 ; Zamir et al. , 2018 ; Lake et al. , 2015 ) , we take a different perspective in this paper . Motivated by the nature of our problem , we focus on the interaction and cooperation between a generative model ( for view synthesis ) and a discriminative model ( for recognition ) , in a way that facilitates knowledge to flow across tasks in complementary directions , thus making the tasks help each other . For example , the synthesized images produced by the generative model provide viewpoint variations and could be used as additional training data to build a better recognition model ; meanwhile , the recognition model ensures the preservation of the desired category information and deals with partial occlusions during the synthesis . To this end , we propose a feedback-based bowtie network ( FBNet ) , as illustrated in Figure 1 . The network consists of a view synthesis module and a recognition module , which are linked through feedback connections in a bowtie fashion . This is a general architecture that can be used on top of any view synthesis model and any recognition model . The view synthesis module explicitly learns a 3D geometric representation from 2D images , which is transformed to target viewpoints , projected to 2D features , and rendered to generate images . The recognition module then leverages these synthesized images from different views together with the original real images to learn a semantic feature representation and produce corresponding classifiers , leading to the feedback from the output of the view synthesis module to the input of the recognition module . The semantic features of real images extracted from the recognition module are further fed into the view synthesis module as conditional inputs , leading to the feedback from the output of the recognition module to the input of the view synthesis module . One potential difficulty , when combining the view synthesis and the recognition modules , lies in the mismatch in their level of image resolutions . Deep recognition models can benefit from highresolution images , and the recognition performance greatly improves with increased resolution ( Wang et al. , 2016 ; Cai et al. , 2019 ; He et al. , 2016 ) . By contrast , it is still challenging for modern generative models to synthesize very high-resolution images ( Regmi & Borji , 2018 ; Nguyen-Phuoc et al. , 2019 ) . To address this challenge , while operating on a resolution consistent with state-of-the-art view synthesis models ( Nguyen-Phuoc et al. , 2019 ) , we further introduce resolution distillation to leverage additional knowledge in a recognition model that is learned from higher-resolution images . Our contributions are three-folds . ( 1 ) We introduce a new problem of simultaneous few-shot recognition and novel-view synthesis , and address it from a novel perspective of cooperating generative and discriminative modeling . ( 2 ) We propose feedback-based bowtie networks that jointly learn 3D geometric and semantic representations with feedback in the loop . We further address the mismatch issue between different modules by leveraging resolution distillation . ( 3 ) Our approach significantly improves both view synthesis and recognition performance , especially in the low-data regime , by enabling direct manipulation of view , shape , appearance , and semantics in generative image modeling . 2 RELATED WORK . Few-Shot Recognition is a classic problem in computer vision ( Thrun , 1996 ; Fei-Fei et al. , 2006 ) . Many algorithms have been proposed to address this problem ( Vinyals et al. , 2016 ; Wang & Hebert , 2016 ; Finn et al. , 2017 ; Snell et al. , 2017 ) , including the recent efforts on leveraging generative models ( Li et al. , 2015 ; Wang et al. , 2018 ; Schwartz et al. , 2018 ; Zhang et al. , 2018 ; Tsutsui et al. , 2019 ; Chen et al. , 2019b ; Li et al. , 2019 ; Zhang et al. , 2019 ; Sun et al. , 2019 ) . A hallucinator is introduced to generate additional examples in a pre-trained feature space as data augmentation to help with low-shot classification ( Wang et al. , 2018 ) . MetaGAN improves few-shot recognition by producing fake images as a new category ( Zhang et al. , 2018 ) . However , these methods either do not synthesize images directly or use a pre-trained generative model that is not optimized towards the downstream task . By contrast , our approach performs joint training of recognition and view synthesis , and enables the two tasks to cooperate through feedback connections . In addition , while there has been work considering both classification and exemplar generation in the few-shot regime , such investigation focuses on simple domains like handwritten characters ( Lake et al. , 2015 ) but we address more realistic scenarios with natural images . Note that our effort is largely orthogonal to designing the best few-shot recognition or novel-view synthesis method ; instead , we show that the joint model outperforms the original methods addressing each task in isolation . Novel-View Synthesis aims to generate a target image with an arbitrary camera pose from one given source image ( Tucker & Snavely , 2020 ) . It is also known as “ multiview synthesis. ” For this task , some approaches are able to synthesize lifelike images ( Park et al. , 2017 ; Yin & Shi , 2018 ; Nguyen-Phuoc et al. , 2018 ; Sitzmann et al. , 2019 ; Iqbal et al. , 2020 ; Yoon et al. , 2020 ; Wiles et al. , 2020 ; Wortsman et al. , 2020 ) . However , they heavily rely on pose supervision or 3D annotation , which is not applicable in our case . An alternative way is to learn a view synthesis model in an unsupervised manner . Pix2Shape learns an implicit 3D scene representation by generating a 2.5D surfel based reconstruction ( Rajeswar et al. , 2020 ) . HoloGAN proposes an unsupervised approach to learn 3D feature representations and render 2D images accordingly ( Nguyen-Phuoc et al. , 2019 ) . Nguyen-Phuoc et al . ( 2020 ) learn scene representations from 2D unlabeled images through foreground-background fragmenting . Different from them , not only can our view synthesis module learn from weakly labeled images , but it also enables conditional synthesis to facilitate recognition . Feedback-Based Architectures , where the full or partial output of a system is routed back into the input as part of an iterative cause-and-effect process ( Ford , 1999 ) , have been recently introduced into neural networks ( Belagiannis & Zisserman , 2017 ; Zamir et al. , 2017 ; Yang et al. , 2018 ) . Compared with prior work , our FBNet contains two complete sub-networks , and the output of each module is fed into the other as one of the inputs . Therefore , FBNet is essentially a bi-directional feedback-based framework which optimizes the two sub-networks jointly . Multi-task Learning focuses on optimizing a collection of tasks jointly ( Misra et al. , 2016 ; Ruder , 2017 ; Kendall et al. , 2018 ; Pal & Balasubramanian , 2019 ; Xiao & Marlet , 2020 ) . Task relationships have also been studied ( Zamir et al. , 2018 ; Standley et al. , 2020 ) . Some recent work investigates the connection between recognition and view synthesis , and makes some attempt to combine them together ( Sun et al. , 2018 ; Wang et al. , 2018 ; Xian et al. , 2019 ; Santurkar et al. , 2019 ; Xiong et al. , 2020 ; Michalkiewicz et al. , 2020 ) . For example , Xiong et al . ( 2020 ) use multiview images to tackle fine-grained recognition tasks . However , their method needs strong pose supervision to train the view synthesis model , while we do not . Also , these approaches do not treat the two tasks of equal importance , i.e. , one task as an auxiliary task to facilitate the other . On the contrary , our approach targets the joint learning of the two tasks and improves both of their performance . Importantly , we focus on learning a shared generative model , rather than a shared feature representation as is normally the case in multi-task learning . Joint Data Augmentation and Task Model Learning leverage generative networks to improve other visual tasks ( Peng et al. , 2018 ; Hu et al. , 2019 ; Luo et al. , 2020 ; Zhang et al. , 2020 ) . A generative network and a discriminative pose estimation network are trained jointly through adversarial loss in Peng et al . ( 2018 ) , where the generative network performs data augmentation to facilitate the downstream pose estimation task . Luo et al . ( 2020 ) design a controllable data augmentation method for robust text recognition , which is achieved by tracking and refining the moving state of the control points . Zhang et al . ( 2020 ) study and make use of the relationship among facial expression recognition , face alignment , and face synthesis to improve training . Mustikovela et al . ( 2020 ) leverage a generative model to boost viewpoint estimation . The main difference is that we focus on the joint task of synthesis and recognition and achieve bi-directional feedback , while existing work only considers optimizing the target discriminative task using adversarial training or with a feedforward network . 3 OUR APPROACH . 3.1 JOINT TASK OF FEW-SHOT RECOGNITION AND NOVEL-VIEW SYNTHESIS . Problem Formulation : Given a dataset D = { ( xi , yi ) } , where xi ∈ X is an image of an object and yi ∈ C is the corresponding category label ( X and C are the image space and label space , respectively ) , we address the following two tasks simultaneously . ( i ) Object recognition : learning a discriminative model R : X → C that takes as input an image xi and predicts its category label . ( ii ) Novel-view synthesis : learning a generative model G : X ×Θ → X that , given an image xi of category yi and an arbitrary 3D viewpoint θj ∈ Θ , synthesizes an image in category yi viewed from θj . Notice that we are more interested in category-level consistency , for which G is able to generate images of not only the instance xi but also other objects of the category yi from different viewpoints . This joint-task scenario requires us to improve the performance of both 2D and 3D tasks under weak supervision without any ground-truth 3D annotations . Hence , we need to exploit the cooperation between them . Few-Shot Setting : The few-shot dataset consists of one or only a few images per category , which makes our problem even more challenging . To this end , following the recent work on knowledge transfer and few-shot learning ( Hariharan & Girshick , 2017 ; Chen et al. , 2019a ) , we leverage a set of “ base ” classes Cbase with a large-sample dataset Dbase = { ( xi , yi ) , yi ∈ Cbase } to train our initial model . We then fine-tune the pre-trained model on our target “ novel ” classes Cnovel ( Cbase∩Cnovel = 0 ) with its small-sample dataset Dnovel = { ( xi , yi ) , yi ∈ Cnovel } ( e.g. , a K-shot setting corresponds to K images per class ) .

Summary: This paper presents a new dual-task of joint few-shot recognition and novel synthesis. The main idea of this paper is to learn a shared generative model across the dual-task to boost the performances of both tasks. To achieve this, bowtie networks are employed to jointly learn geometric and semantic representations with a feedback loop. The proposed method is evaluated on fine-grained recognition datasets.