Bowtie Networks: Generative Modeling for Joint Few-Shot Recognition and Novel-View Synthesis
Authors: Zhipeng Bao
Yu-Xiong Wang
Martial Hebert
Venue: ICLR 2021 Poster (2021)
Keywords: computer vision
object recognition
few-shot learning
generative models
adversarial training

Abstract: We propose a novel task of joint few-shot recognition and novel-view synthesis: given only one or few images of a novel object from arbitrary views with only category annotation, we aim to simultaneously learn an object classifier and generate images of that type of object from new viewpoints. While existing work copes with two or more tasks mainly by multi-task learning of shareable feature representations, we take a different perspective. We focus on the interaction and cooperation between a generative model and a discriminative model, in a way that facilitates knowledge to flow across tasks in complementary directions. To this end, we propose bowtie networks that jointly learn 3D geometric and semantic representations with a feedback loop. Experimental evaluation on challenging fine-grained recognition datasets demonstrates that our synthesized images are realistic from multiple viewpoints and significantly improve recognition performance as ways of data augmentation, especially in the low-data regime. 

Published as a conference paper at ICLR 2021
BOWTIE NETWORKS: GENERATIVE MODELING FOR JOINT FEW-SHOT RECOGNITION AND NOVEL-VIEW SYNTHESIS
Zhipeng B
Yu-Xiong Wang"
Martial Hebert
Camegii Mellon University University of Illinois at Kibana-Champaar yxw rxweillinois. edu
(zbao, hebert (cs. emu. edu
ABSTRACT
We propose novel task of joint few-shot recognition and novel-view synthesis: given only one or few images of a novel object from arbitrary views with only category annotation, we aim to simultaneously learn object classifier and gen- erate images of that type of object from new viewpoints. While existing work copes with two or more tasks mainly by multi-task learning of shareable feature representations, we take a different perspective. We focus on the interaction and cooperation between a generative model and discriminative model, in way that facilitates knowledge to flow across tasks in complementary directions. To this end, we propose bowtie networks that jointly learn 3D geometric and semantic represen- tations with feedback loop. Experimental evaluation challenging aneiigaieed recognition datasets demonstrates that our synthesized images are realistic from multiple viewpoints and significantly improve recognition performance ways of data augmentation, especially in the low-data regime. Code and pre-trained models are released at https: : //gittutt com/ 2pbao /bwwt ie networks.
INTRODUCTION
Given a never- never-before-sen before-seen object (e.g., a gadwall in Figure 1), humans are able to generalize even from single image of this object in different ways, including recognizing new object instances and imagining what the object would look like from different viewpoints. Achieving similar levels of generalization for machines is a fundamental problem in computer vision, and has been actively explored in areas such as few-shot object recognition (Fei-Fei et al., 2006; Vinyals et al., 2016; Wang & Hebert, 2016; Finn et al., 2017; Snell et al., 2017) and novel-view synthesis (Park et al., 2017; Nguyen-Phun et al., 2018: 8; Sitzmann et al., 2019). However, such exploration is often limited in separate areas with specialized algorithms but not jointly.
We argue that synthesizing images and recognizing them are inherently interconnected with each other. Being able to simultaneously address both tasks with a single model is a crucial step toward human-level generalization. This requires learning richer, shareable internal representation for more comprehensive object understanding than it could be within individual tasks. Such "cross-task" knowledge becomes particularly critical in the low-data regime, where identifying 3D geometric structures of input images facilities recognizing their semantic categories, and vice versa.
Inspired by this insight, here propose novel task of joint few-shot recognition and novel-view synthesis: given only one or few images of a novel object from arbitrary views with only category annotation, we aim to simultancously learn an object classifier and generate images of that type of object from new viewpoints. This joint task is challenging, because of its (i) weak supervision, where we do not have access to any 3D supervision, and (ii) few-shot setting, where we need to effectively learn both 3D geometric and semantic representations from minimal data.
While existing work copes with two or more tasks mainly by multi-task learning or meta-learning of a shared feature representation (Yu al., 2020: Zamir et al., 2018: Lake et al., 2015). we take a different perspective in this paper. Motivated by the nature of our problem, we focus on the interaction and cooperation between generative model (for view synthesis) and a discriminative model (for recognition). in a way that facilitates knowledge to flow across tasks in complementary directions, thus making the tasks help each other. For example, the synthesized images produced by

---

Published as a conference paper at ICLR 2021
View Syntheals
Mullippe
images
Youd
single image
Recongitton
Varpekt
TAGs bind "Gathrrll"
Category
"Gadwall"
Given single image of novel visual concept (e.g., gadwall), person can generalize various ways, including imagining what this gadwall would look like from different viewpoints (top) and recognizing new gadwall instances (bottom). Right: Inspired by this, we introduce a general feedback- bowtie network that facilitates the interaction and cooperation between generative module and a discriminative module, thus simultaneously addressing few-shot recognition and novel-view synthesis in the low-data regime.
Figure 1: Left:
the generative model provide viewpoint variations and could be used as additional training data to build better recognition model; meanwhile, the recognition model ensures the preservation of the desired category information and deals with partial occlusions during the synthesis.
To this end, we propose a feedbuck- bowtie nefwork (FBNer), as illustrated in Figure 1. The network consists of a view synthesis module and recognition module, which are linked through feedback connections in bowtie fashion. This is general architecture that can be used on top of any view synthesis model and any recognition model. The view synthesis module explicitly learns a 3D geometric representation from 2D images, which is transformed to target viewpoints, projected to 2D features, and rendered generate images. The recognition module then leverages these synthesized images from different views together with the original real images to learn a semantic feature representation and produce corresponding classifiers, leading to the feedback from the output of the view synthesis module to the input of the recognition module. The semantic features of real images extracted from the recognition module are further fed into the view synthesis module as conditional inputs, leading to the feedback from the oulput of the recognition module to the imput of the view synthesis module.
One potential difficulty, when combining the view synthesis and the recognition modules, lies in the mismatch in their level of image resolutions. Deep recognition models can benefit from high- resolution images, and the recognition performance greatly improves with increased resolution (Wang et al., 2016; Cai et al., 2019; He et al., 2016). By contrast, it is still challenging for modern generative models to synthesize very high-resolution images (Regmi & Borji, 2018; Nguyen- 2019). To address this challenge, while operating on a resolution consistent with state-of of-thhe-rtttttt view synthesis models (Nguyen- PPuuoe et al., 2019), we further introduce resolution distillation to leverage additional knowledge in a recognition model that is learned from higher- images.
al.,
Our contributions are three-folds. (1) We introduce new problem of simultaneous few-shot recog- nition and novel-view synthesis, and address it from a novel perspective of cooperating generative and discriminative modeling. (2) We propose feedback-
bowtie networks that jointly learn 31) geometric and semantic representations with feedhack in the loop. We further address the mismatch issue between different modules by leveraging resolution distillation. (3) Our approach significantly improves both view synthesis and recognition performance, especially in the low-data regime, by enabling direct manipulation of view, shape, appearance, and semantics in generative image modeling.
2 RELATED WORK
Few-Shot Recognition is classic problem in computer vision (Thrun, 1996; Fei-Fei et al., 2006). Many algorithms have been proposed to address this problem (Vinyals et al., 2016; Wang & Hebert, 2016; Finn et al., 2017; Snell et al., 2017), including the recent efforts
leveraging generative models (Li et al., 2015; Wang et al., 2018; Schwartz ct al., 2018; Zhang et al., 2018; Tsutsui et al., 2019; Chen et al., 2019b; Li et al., 2019; Zhang et al., 2019; Sun et al., 2019). A hallucinator is introduced to generate additional examples in a prentr feature space as data augmentation to help with low-shot classification (Wang et al., 2018 MetaGAN improves few-shot recognition by producing fake images as a new category (Zhang et al., 2018). However, these methods either do not synthesize images directly or use pre-trained generative model that is not optimized towards the downstream task. By contrast, our approach performs joint training of recognition and view synthesis, and enables the two tasks to cooperate through feedback connections. In addition, while there has been work considering both classification and exemplar generation in the few-shot regime, such investigation focuses on simple domains like handwritten characters (Lake et al., 2015) but we address more realistic scenarios with natural images. Note that our effort is largely orthogonal to
2

---

Published as a conference paper at ICLR 2021
designing the best few-shot recognition or novel-view synthesis method; instead, we show that the joint model outperforms the original methods addressing each task in isolation.
View Synthesis aims to generate target image with an arbitrary camera pose from one given source image (Tucker & Snavely, 2020). is also known as "multiview synthesis." For this task, some approaches are able to synthesize lifelike images (Park et al., 2017; Yin & Shi, 2018; Nguyen-Phuoc et al., 2018; Sitzmann al.. 2019; Iqbal et al., 2020; Yoon et al., 2020; Wiles et al., 2020; Wortsman et al., 2020). However, they heavily rely on pose supervision or 3D annotation, which is not applicable in our case. An alternative way is to learn a view synthesis model in an unsupervised manner. Pix2Shape learns an implicit 3D scene representation by generating 2.5D surfel based reconstruction (Rajeswar al., 2020). HoloGAN proposes an unsupervised approach to learn 3D feature representations and render 2D images accordingly (Nguyen- Phuuce et al., 2019). Nguyen-Phrr Phuuc et al. (2020) learn scene representations from 2D unlabeled images through bargrouud fragmenting. Different from them, not only can our view synthesis module learn from weakly labeled images, but it also enables conditional synthesis to facilitate recognition.
Feedback- Based Architectures, where the full or partial output of system is routed back into the input as part of an iterative cause-and-effect process (Ford, 1999). have been recently introduced into neural networks (Belagiannis & Zisserman, 2017: Zamir et al., 2017; Yang et al., 2018). Compared with prior work, our FBNet contains two complete sub-networks, and the output of each module is fed into the other as one of the inputs. Therefore, FBNet is essentially bi-directional feedback- framework which optimizes the two sub-networks jointly.
Multi-task Learning focuses on optimizing collection of tasks jointly (Misra et al., 2016; Ruder, 2017; Kendall et al., 2018; Pal & Balasubramanian, 2019; Xiao & Marlet, 2020). Task relationships have also been studied (Zamir et al., 2018; Standley et al., 2020). Some recent work investigates the connection between recognition and view synthesis, and makes some attempt to combine them together (Sun et al., 2018: Wang et al., 2018; Xian et al., 2019; Santurkar et al., 2019; Xiong et al., 2020; Michalkiewicz et al., 2020). For example, Xiong et al. (2020) use multiview images to tackle fine-grained recognition tasks. However, their method needs strong pose supervision to train the view synthesis model, while we do not. Also, these approaches do not treat the two tasks of equal importance, i.c., one task an auxiliary task to facilitate the other. On the contrary, our approach targets the joint learning of the two tasks and improves both of their performance. Importantly, we focus on learning a shared generative model, rather than a shared feature representation as is normally the case in multi-task learning.
Joint Data Augmentation and Task Model Learning leverage generative networks to improve other visual tasks (Peng et al., 2018; Hu al., 2019; Luo et al., 2020; Zhang et al., 2020). A generative network and a discriminative pose estimation network are trained jointly through adversarial loss facilitate the
in Peng et al. (2018), where the generative network performs data augmentation downstream pose estimation task. Luo et al. (2020) design a controllable data augmentation method for robust text recognition, which is achieved by tracking and refining the moving state of the control points. Zhang et al. (2020) study and make use of the relationship among facial expression recognition, face alignment, and face synthesis to improve training. Mustikovela et al. (2020) leverage a generative model to boost viewpoint estimation. The main difference is that we focus on the joint task of synthesis and recognition and achieve bi-directional feedback, while existing work only considers optimizing the target discriminative task using adversarial training or with a feedforward network.
3 OUR APPROACH
3.1 JOINT TASK OF FEW-SHIOT RECOGNITION AND NOVEL-VIEW SYNTIESIS
Given dataset DD = { y:)}. where x; E X' is an image of an object and V: E C is the corresponding category label (X' and are the image space and label space, respectively), we address the following two tasks simultaneously. (i) Object recognition: learning a discriminative model R X + C that takes as input an image 2; and predicts its category label. (ii) Novel-view synthesis: learning generative model G : X' x θ + X' that, given an image z; of category y; and an arbitrary 3D viewpoint 8 E θ, synthesizes an image in category y/4 viewed from fy. Notice that we are more interested in category-level consistency, for which G is able to generate images of not only the instance 3; but also other objects of the category yi from different viewpoints. This joint-task scenario requires us to improve the performance of both 2D and 3D tasks under weak supervision without any ground-truth 3D annotations. Hence, we need exploit the cooperation between them.
Problem Formulation:
3

---

Published as a conference paper at ICLR 2021
View Synthesis Module
Trunsfomation
DECOBY
Inital Cube
,87,97
Generated images
AdaIN
AdaNN
Lanent Vacice/
Nosse Vester
Images multiple views
Recognition Module
Real Features
Extraction
Prosatypiaa Classfloation Nawwrk
Black foooodo
Real Imagu
Latent Features
Predicted Label
Figure 2:
Architecture our feedback-base based bowtic network. The whole network consists a view synthesis module and recognition module, which are linked through feedback connections in a bowtie fashion.
Few-Shot Setting: The few-shot dataset consists of one only few images per category, which makes our problem even more challenging. To this end, following the recent work on knowledge transfer and few-shot learning (Hariharan & Girshick, 2017; Chen et al., 2019a), we leverage set of "base" classes Cruse with sange dataset D buse {Ir; y/), Vi E Cruse train our initial model. We then fine-tune the pre-trained model on our target "novel" classes Gnovel (Cbuse C = 0) with its amall-sampl dataset D Pnovel {2,,, y:), y: Cnovel } (e.g., g., K-shot setting corresponds K images per class).
3.2 FEEDBACK BASSE BOWTIE NETWORKS
To address the joint task, we are interested in learning a generative model that can synthesize realistic images of different viewpoints, which are also useful for building strong recognition model. We propose a feedback-brid synthesis module and recognition module, trained in joint, end-to-end fashion. Our key insight to explicitly introduce feedback connections between the two modules, so that they cooperate with each other, thus enabling the entire model to simultaneously learn 3D geometric and semantic representations, This general architecture can be used on top of any view synthesis model and any recognition model. Here we focus on a state-of-the the-art view synthesis model HoloGAN (Nguyen- Phuoc al., 2019), and widely adopted few-shot recognition model prototypical network (Snell et al., 2017) (017,, as shown in Figure 2.
bowtie network (FBNet) for this purpose. This model consists of a view
3.2.1 VIEW SYNTHESIS MODULE
The view synthesis module V is shown in the blue shaded region in Figure 2. It is adapted from HoloGAN (Nguyen-Phuoc al., 2019), state-of-the-art model for unsupervised view synthesis. This module consists of a generator G which first generates a 3D feature representation from latent constant tensor (initial cube) through 3D convolutions. The feature representation is then transformed certain pose and projected to 2D with a projector. The final color image is then computed through 2D convolutions. This module takes two inputs: a latent vector input and a view input 0. characterizes the style of the generated image through adaptive instance normalization (AdaIN) (Huang & Belongic, 2017) units. 0 [0 (V, 0=] guides the transformation of the 3D feature representation. This module also contains a discriminator to detect whether an image is real or fake (not shown in Figure 2). We use the standard GAN loss from DC-GAN (Radford et al., 201 6). LGAN (G, D). We make the following important modifications make the architecture applicable to our joint task.
Latent Vector Formulation: To allow the synthesis module get feedback from the recognition module (details are shown in Section 3.2.3), we first change HoloGAN from unconditional to conditional. To this end, model the latent input as: 2; f1 n4, where is the conditional feature input derived from image v; and n; noise vector sampled from Gaussian distribution.
is the combination strategy (e.g., concatenation). By doing so, the synthesis module leverages
4

---

Published as a conference paper at ICLR 2021
additional semantic information, and thus maintains the category- consistency with target image and improves the diversity of the generated images.
Inspired by Chen et al. (2016), we introduce an identity regularizer to ensure that the synthesis module simultaneously satisfies two critical properties: (i) the identity of the generated image remains when we only change the view input 9; (ii) the orientation of the generated image preserves when we only change the latent input 2, and this orientation should be consistent with the view input 0. Specifically, we leverage an encoding network H to predict the reconstructed latent vector z' and the view input 0': (G(, 0)) [z', 0'), where G(z, 0) is the generated image. Then we minimize the difference between the real and the reconstructed inputs as
Identity Regularizer:
Lideniity (G, H) = E 2- - 1/12 + Egl 0112
(I)
Here H shares the majority of the convolution layers of the discriminator D. but uses an additional fully-connected layer. Section A explains the detäiled architecture of the view synthesis module.
3.2.2 RECOGNITION MODULE
The recognition module R (green shaded region in Fig. 2) consists of a feature extraction network F which transforms images to latent features, and a prototypical classification network P (Snell et al., 2017) which performs the final classification. Below we explain the design of these two components, focusing on how to address the technical challenges faced by joint training with view synthesis.
Feature Extraction with Resolution Distillation: We use ResNet (He et al., 2016) as our feature extraction network to transform images into latent features for the recognition module. One of the main obstacles to combining F with the synthesis module is that state-of-ther -frtte-rtttttttte-e synthesis models and recognition models operate on different resolutions. Concretely, to the best of our knowledge, current approaches to unsupervised novel-view synthesis still cannot generate satisfactory high-resolution images (e.g., 224 224) (Nguyen-Phhe Phuoc et al., 2019). By contrast, the performance of current well-performing recogmition models substantially degrades with low-resolution images (Wang et al., 2016; Cai et al., 2019). To reconeile the resolution incompatibility, we introduce a simple distillation technique inspired by the general concept of knowledge distillation (Hinton et al., 2014). Specifically, operate on the resolution of the synthesis module (e.g., 64 64). But benefit from additional auxiliary feature extraction network FiighR that is trained on high-resolution images (e.g., 224 x 224). We first pre-train Frightt following the standard practice with cross-entropy softmax classifier (Liu et al., 2016). We then train our feature extraction network FOOR (the one used in the recognition module), under the guidance of Fiihht through matching their features:
Liesture
( FlowR) Ex FhighR(() (x) F1)wR(=))1
where x is training image, With the help of resolution distillation, the feature extraction network re-captures information in nighhresolution images but potentially missed in low-resolution images.
We use the prototypical network P (Snell et al., 2017) as our classifier. The network assigns class probabilities ý based on distance of the input feature vector from class centers μr; and is calculated by using support images in the latent feature space:
Prototypical Classification Network:
(F
Me E(EnnDeS P(Fiown (z. )) [y, cl
pe(x) E, E(Paa(((huaat).a))
(3)
E(e.nees I [b = -c
where x is real query image, de is the probability of category c, and d is a distance metric (e.g.. Euclidean distance).
support dataset. operates on top the feature extraction network F, and consists of 3 cully-connected layers as additional feature embedding (the classifier is non- parametric). Another benefit of using the prototypical network lies in that it enables the recognition module to explicitly leverage the generated images in a way of data augmentation, i.e., S contains both real and generated images compute the class mean. Notice that, though, the module parameters updated based on the loss calculated on the real query images, which is a cross-entropy loss Liee (R) between their predictions ý and ground-truth labels.
5

---

Published as a conference paper at ICLR 2021
3.2.3 FEEDBACK- BASED BOWTIE MODEL
As shown in Figure 2, we leverage a bowtie architecture for our full model, where the output of each module is fed into the other module as one of its inputs. Through joint training, such connections work as explicit feedback facilitate the communication and cooperation between different modules.
Feedback Connections: We introduce two complementary feedback connections between the view synthesis module and the recognition module: (1) recognition output - synthesis input (green arrow in Figure 2), where the features of the real images extracted from the recognition module are fed into the synthesis module conditional inputs to generate images from different views; (2) synthesis output + recognition input (blue arrow in Figure 2), where the generated images are used to produce an augmented set to train the recognition module.
Categorical Loss for Feedback: The view synthesis module needs to capture the categorical seman- ties in order to further encourage the generated images to benefit the recognition. Therefore, we introduce a categorical loss to update the synthesis module with the prediction results of the generated images:
(G) = E, (o,
(4)
where yi the category label for the generated image G(zi, 0t). This loss also implicitly increases the diversity and quality of the generated images.
Final Loss Function: The final loss function is:
Lioual = LGAN + Liee + Lreatue + Aid Cidentyy + Aea Leat:
(5)
where Xia and Aent are trade-off hyper-parameters.
Training Procedure: We first pre-train FhighR on the high-resolution dataset and save the computed features. These features are used to help train the feature extraction network FOOR through Creature- Then the entire model is first trained on Crase and then fine-tuned on Cnovel- The training on the two sets are similar. During each iteration, we randomly sample some images per class as a support set and one image per class as query set. The images in the support set, together with their computed features via the entire recognition module, are fed into the view synthesis module to generate multiple images from different viewpoints. These synthesized images are used to augment the original support set to compute the prototypes. Then, the query images are used to update the parameters of the recognition module through L the view-synthesis module is updated through LGAN. Lidenity. and Lea The entire model is trained in end-to-end fashion. More details are in Section B.
4 EXPERIMENTAL EVALUATION
Datasets: We focus on two datasets here: the Caltech-UCSD Birds (CUB) dataset which contains 200 classes with 11,788 images (Welinder et al., 2010), and the CompCars dataset which contains 360 classes with 25,519 images (Yang et al., 2015). Please refer to Section C for more details of the datasets. These are challenging fine- mne-graieed recognition datasets for our joint task. The images are resized to 64 x 64. We randomly split the entire dataset into 75% as the training set and 25% as the test set. For CUB, 150 classes are selected as base classes and 50 novel classes. For CompCars, 240 classes are selected as base classes and 120 as novel classes. Note that we focus on simultaneous recognition and synthesis over all base or novel classes, which is significantly more challenging than typical 5-way classification over sampled classes in most of few-shot classification work (Snell et al., 2017; Chen al., 2019a). We also include evaluation on additional datasets in Section D.
Implementation Details: We set AM = 10 and Acat = 1 via cross-validation We use ResNet- 18 (He et al., 2016) as the feature extraction network, unless otherwise specified. To match the resolution of our data, we change the kernel size of the first convolution layer of ResNet from to 5. The training process requires hundreds of examples at each iteration, which may not fit in the memory of our device. Hence, inspired by Wang et al. (2018), 8), we make a trade-off to first train the feature extraction network through resolution distillation. We then freeze its parameters and train the other parts of our model. Section C includes more implementation details.
Our feedback connections enable the two modules to cooperate through joint training. Therefore, evaluate the effectiveness of the feedback connections, we focus the following comparisons. (1) For the novel-view image synthesis task, we compare our approach FBNet with the state-of-the-art method HoloGAN (Nguyen-Phuoe Phuoc al., 2019). We also consider
Compared Methods:

---

Published as a conference paper at ICLR 2021
Model
Base
==
3
FBNet-ree
57.91
47.33
1.144
71.26
0.26
CUB
FBNett-au
58.03
0..9
71.51
+.33
FBNet
59.43
48.39
+ 0.19
72.76
0.24
FBNet-ree
46.05
20.83
+ 0.03
50.52 0.11
CompCars
FBNet-aug
47.41
21.59
0.05
61.07
FBNet
49.63
23.28
0.05
53.12 = 0009
Table 1: Top-1 (%) recognition accuracy on the CUB and CompCars datascts. For base classes: 150
classification on CUB and 240-way classification on CompCars: for K-shot novel classes: 50-way classification on CUB and 120-way classification on CompCars. Our FBNet consistently achieves the best performance for hoth base and novel classes, and joint training significantly outperforms training each module individually.
(
Model
Rase
Nowel-K-1
Novel-K-5
Base
Novel-K-1
Novel-
Rewl Images
1.56
0.30
3.53 = 0.22
3.53
0.22
0
HoloGAN (Nguyen- Phuoc et al., 2019)
Souyen-Phuoceee
3.55 1: 0.09
2.44 0.07
2,58 0.08
79.01
106.56
94.73
CUB
FBNet-viow
3.60
2.53
0.05
75.38
107.36
103:25
FANet
3.69 + 0.17
2.79 = 0.06
2.83 0.12
70.86
104.04
92,97
Real Images
2.36 0.12
2.80 = 0.13
2.80
0
0
CompCars
HoloGAN (Nguyen- Phuoec et 2019)
1.85 1: 0.08
1.41 = 0.04
1.65
$1.49
93.48
83.17
FBNet-viow
2:03 + 0:09
1.11
0.71
49.94
92.01
83.58
FBNet
2.33 0.14
1.89 0.07
0.10
44.70
89.39
78.38
Table 2: Novel-view synthesis results under the FID and IS metrics. f indicates that higher is better, and
indicates that lower is better. As reference, FID and IS of Real Images represent the best results we could
expect. FBNet consistently outperforms the baselines, achieving 18% improvements for FID and 19% for IS.
variant of our approach FBNet-view, which has the same architecture our novel-view synthesis module, but takes the constant features extracted by a pre-trained ResNet-18 as latent input. FBNet- view can be also viewed as conditional version of HoloGAN. (2) For the few-shot recognition task, we compare our full model FBNet with its two variants: FBNet-ree inherits the architecture of our recognition module, which is essentially prototypical network (Snell et al., 2017); FBNet- aug uses the synthesized images from individually trained FBNet-view as data augmentation for the recognition module. Note that, while conducting comparisons with other few-shot recognition (e.g., Chen et al. (2019a); Finn et al. (2017)) or view synthesis models (e.g., Yoon al. (2020); Wiles et al. (2020)) is interesting, it is not the main focus of this paper. We aim to validate that the feedback- bowtie architecture outperforms the single-task models upon which it builds, rather than designing the best few-shot recognition or novel-view synthesis method. In Section F, we show that our framework is general and can be used on top of other single-task models and improve their performance. All the models are trained following the same few-shot setting described in Section 3.1.
View Synthesis Facilitates Recognition: Table 1 presents the top-1 recognition accuracy for the base classes and the novel classes, respectively. We focus on the challenging 1, 5-shot settings, where the number of training examples per novel class K is 1 or 5. For the novel classes, we run five trials for each setting of K, and report the average accuracy and standard deviation for all the approaches. Table 1 shows that our FBNet consistently achieves the best few-shot recognition performance on the two datasets. Moreover, the significant improvement of FBNet over FBNet-aug (where the recognition model uses additional data from the conditional view synthesis model, but they are trained separately) indicates that the ceedback-based joint training is the key to improve the recognition performance.
Recognition Facilitates View Synthesis: We investigate the novel-view synthesis results under two standard metrics. The FID score computes the Fréchet distance between two Gaussians fitted to feature representations of the source (real) images and the target (synthesized) images (Dowson & Landau, 1982). The Inception Score (IS) uses an Inception network pre-trained on ImageNet (Deng et al., 2009) to predict the label of the generated image and calculate the entropy based on the predic- tions. IS seeks to capture both the quality and diversity of a collection of generated images (Salimans et al., 2016). A higher IS lower FID value indicates better realism of the generated images. A larger variance of IS indicates more diversity of the generated images. We generate images of random views in one-to-one correspondence with the training examples for all the models, and compute the IS and FID values based on these images. The results are reported in Table 2. As a reference, we also show the results of real images under the two metrics, which are the best results we could expect from synthesized images. Our FBNet consistently achieves the best performance under both metrics. Compared with HoloGAN, our method brings up to 18% improvement under FID and 19% under IS, Again, the significant performance gap between FBNet and FBNet-view shows that the feedback- joint training substantially improves the synthesis performance.
IS and FID cannot effectively evaluate whether the generated images maintain the category-level identity and capture different viewpoints. Therefore, Figure visualizes the synthesized multiview images. Note that, in our problem setting of limited training data under weak supervision, we could
7

---

Published as a conference paper at ICLR 2021
CUB
CompCars
Base
K=5
K-1
Base
K-S
HoloGAN
FBNet-view
FBNet
Figure 3: Synthesized images from multiple viewpoints. Images in the same row/column are from the same viewpoint/object. Our approach captures the shape and attributes well even in the extremely low-data regime.
Setting
Model
ResNet- 10
ResNet-18
ResNet-34
ResNet-50
FBNet-view
46.28
47.53
46.79
45.68
K=1
FBNet
48.85
48.39
47.65
47.03
FBNet-view
71.66
71.26
70.69
70.00
K=5
FBNet
72.49
72.76
71.28
70.95
Table 3: Few-shot recognition accuracy consistently improves with diffcrent feature extraction networks.
K=1
K=5
Setting
Ace
FID (})
(f)
Acc
FID ({)
[S (f)
Multitask-Feat Feat (Ruder, 2017)
34.71
110.03
2.19
52.54
99.61
2.44 + 0.04
FBNet w/o Dist
22.47
108.73
2.31 + 0.05
34.15
97.64
2.42 0.07
FBNett w/o Proto
44.62
105.81
2.61
1 0.07
70.04
95.15
2.76
0.10
FBNet
48.39
104.04
2.79
1 0.06
72.76
92.97
2.83
0.12
Table 4: Ablation studies on CUB regarding (1) learning a shared feature representation through standard
multi-task learning, (ii) FBNet without resolution distillation, and (iii) FBNet using a regular classification network without prototypical classification. Our full model achieves the best performance.
not expect that the quality of the synthesized images would match those generated based on large amounts of training data, e.g. Brock et al. (2019). This demonstrates the general difficulty of image generation in the few-shot setting, which is worth further exploration in the community.
Notably, even in this challenging setting, our synthesized images are of significantly higher visual quality than the sfate-of-the-ar baselines. Specifically, (1) our FBNet is able to perform controllable conditional generation, while HoloGAN cannot. Such conditional generation enables FBNet to better capture the shape information of different car models on CompCars, which crucial to the recognition task. On CUB, FBNet captures both the shape and attributes well even in the extremely low-data regime (1-shot), thus generating images of higher quality and more diversity. (2) Our FBNet also better maintains the identity of the objects in different viewpoints. For both HoloGAN and FiNet-view, it is hard tell whether they keep the identity, but FBNet synthesizes images well from all the viewpoints while maintaining the main color and shape. (3) In addition, we notice that there is just minor improvement for the visual quality of the synthesis results from HoloGAN to FBNet-view, indicating that simply changing the view synthesis model from unconditional to conditional versions does not improve the performance. However, through our feedback- joint training with recognition, the quality and diversity of the generated images significantly improve.
Shared Generative Model vs. Shared Feature Representation: We further compare with a stan- dard multi-task baseline (Ruder, 2017), which learns a shared feature representation across the joint tasks, denoted 'Multitask-Featt in Table 4. We treat the feature extraction network as shared component between the recognition module and the view synthesis module, and update its parameters using both tasks without feedback connections. Table 4 shows that, through the feedback connec- tions, our shared generative model captures the underlying image generation mechanism for more comprehensive object understanding, outperforming direct task-level shared feature representation.
Ablation Different Recognition Networks: While we used ResNet-18 as the default feature extraction network, our approach is applicable to different recognition models. Table 3 shows that the recognition performance with different feature extraction networks consistently improves. Interestingly, ResNet- 10/18 outperform the deeper models, indicating that the deeper models might suffer from over-fitting in few-shot regimes, consistent with the observation in Chen al. (2019a).
8

---

Published as a conference paper at ICLR 2021
Acat
Acat 0.1
cat
Acaz
Top Acc 60.94 0.29 103.75 IS: 0.10
Ace: 70.93 0.27 FID: 99.50 :5:2.75 a 0.09
Top Acc: 72.76 + 0.24
Top I Acc : 73.09 0.21 FID: 114.85
FID 11.-992.9
18:28334
2.36 + 0.08
Figure 4: Ablation on Aea. Categorical loss trades off the performance between view synthesis and recognition. Novel
Base
Base
Novel
HoloGAN
FBNet
Figure 5: Synthesized images by HoloGAN and FBNet on CelebA-IIQ. Few-shot attributes (left to right): Black
Hair, Gray Hair, Bald, Wearing Hat, and Aging. FBNet synthesizes images of higher quality and diversity.
Ablation - -Categorical Loss: In addition to the feedback connections, our synthesis and recognition modules are linked by the categorical loss. To analyze its effect, we vary Aaxe among 0 (without the categorical loss), 0. 1, and 5. Figure shows the quantitative and qualitative results on CUB. With Asa increasing, the recognition performance improves gradually. Meanwhile, too large Aua reduces the visual quality of the generated images: checkerboard noise appears. While these images are not visually appealing, they still benefit the recognition task. This shows that the categorical loss trades off the performance between the two tasks, and there a "sweet spot" between them.
Ablation - Resolution Distillation and Prototypical Classification: Our proposed resolution dis- tillation reconciles the resolution inconsistency between the synthesis and recognition modules, and further benefits from a recognition model trained on high-resolution images. The prototypical network leverages the synthesized images, which constitutes one of the feedback connections. We evaluate their effect by building two variants of our model without these techniques: 'FBNet w/o Dist' trains the feature extraction network directly from low-resolution images; 'FBNet w/o Proto' uses a regular classification network instead of the prototypical network. Table 4 shows that the performance of full FBNet significantly outperforms these variants, verifying the importance of techniques.
We further show that the visual quality of our synthesized images significantly gets improved on datasets with better aligned poses. For this purpose, we conduct experiments on CelebA-HQ (Lee et al., 2020), which contains 30,000 aligned human face images regarding 40 attributes in total. We randomly select 35 attributes as training attributes and 5 as few-shot test attributes. While CelebA-HQ does not provide pose annotation, the aligned faces mitigate the pose issue some extent. Figure 5 shows that both the visual quality and diversity of our synthesized images substantially improve, while consistently outperforming HoloGAN.
Qualitative Results on the CelebA-HQ Dataset:
Our experimental evaluation has focused on fine-graine categories. mainly because state-of-the-art novel-view synthesis models still cannot address image generation for a wide spectrum of general images (Liu et al., 2019). Mcanwhile, our feedback- -bsed bowtic architecture is general. With the advance in novel-view synthesis. such the recent work of BlockGAN (Nguyen-Phuoc et al., 2020) and RGBD-GAN (Noguchi & Harada, 2020), our framework could be potentially extended deal with broader types of images. Additional further investigation includes exploring more architecture choices and dealing with images with more than one object.
Discussion and Future Work:
5 CONCLUSION
This paper has proposed feedback- based bowtie network for the joint task of few-shot recognition and novel-view synthesis. Our model consistently improves performance for both tasks, especially with extremely limited data. The proposed framework could be potentially extended address more tasks, leading to a generative model useful and shareable across a wide range of tasks.
Acknowledgement: This work was supported in part by ONR MURI N00001 14 16-1-70777 and by AFRL Grant FA23861714660 We also thank NVIDIA for donating GPUs and AWS Cloud Credits for Research program.

---

Published as a conference paper at ICLR 2021
REFERENCES
Vasileios Belagiannis and Andrew Zisserman, Recurrent human pose estimation. In International
Conference Automatic Face & Gesture Recognition (FG), 2017. 3
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity
natural image synthesis. In ICLR, 2019. 8
Dingding Cai, Ke Chen, Yanlin Qian, and Joni-Kristian Kämäräinen. Convolutional low-resolution fine-grained classification. Pattern Recognition Letters, 2019. 2,5
Wei- Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu- Ch.anian Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. In ICLR,, 2019a. 6, 7,8
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets. In NeurlPS, 2016.
Zitian Chen, Yanwei Fu, Yu-Xiong Wang, Lin Ma, Wei Liu, and Martial Hebert. Image deformation meta-networks for one-shot learning. In CVPR, 2019b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009. 7
DC Dowson and BV Landau. The fréchet distance between multivariate normal distributions. Journal of multivariate analysis, 1982. 7
Li Fei-Fei, Rob Fergus, and Pietro Perona. One-shot learning of object categories. PAMI, 2006. 1,2
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In JCML, 2017. 1,
Andrew Ford. Modeling the environment: an introduction to system dynamics models of environmental systems. Island press, 1999. 3
Bharath Hariharan and Ross Girshick. Low-shot visual recognition by shrinking and hallucinating features. In ICCV, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 2, 2,5,6 6, 14
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In NeuriPS Workshops, 2014. 5
Zhiting Hu, Bowen Tan, Russ R Salakhutdinov, Tom M Mitchell, and Eric P Xing. Learning data manipulation for augmentation and weighting. In NeurIPS, 2019. 3
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In ICCV, 2017. 4
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. /CML, 2015. 14
Umar Iqbal, Pavlo Molchanov, and Jan Kautz. Weakly-supervised 3D human pose learning via multi-view images in the wild. CVPR, 2020. 3
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In CVPR, 2018.
Aditya Khosla, Nityananda Jayadevaprakash. Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine-grained image categorization. In CVPR Workshops, 2011. 15
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 2015. 1,2
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In CVPR, 2020.
10

---

Published as a conference paper at ICLR 2021
Aoxue Li, Tiange Luo, Tao Xiang, Weiran Huang, and Liwei Wang. Few-shot learning with global class representations. ICCV, 2019.
Yujia Li, Kevin Swersky, and Rich Zemel. Generative moment matching networks. ICML, 2015.
Ming- Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. Few-shot unsupervised image-to-image translation. In ICCV, 2019.
Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolu- tional neural networks. In ICML, 2016. 5
Canjie Luo, Yuanzhi Zhu, Lianwen Jin, and Yongpan Wang. Learn to augment: Joint data augmenta- tion and network optimization for text recognition. In CVPR, 2020.
Mateusz Michalkiewicz, Sarah Parisot, Stavros Tsogkas, Mahsa Baktashmotlagh, Anders Eriksson, and Eugene Belilovsky. Few-shot single-view 3-D object reconstruction with compositional priors. In ECCV, 2020. 3
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In CVPR, 2016.
Siva Karthik Mustikovela, Varun Jampani, Shalini De Mello, Sifei Liu, Umar Iqbal, Carsten Rother, and Jan Kautz. Self-supervised viewpoint learning from image collections. In CVPR, 2020. 3
Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. HoloGAN: Unsupervised learning of 3D representations from natural images. In ICCV, 2019. 2, 3, 5, 14, 16, 18
Thu Nguyen-Phuos Christian Richardt, Long Mai, Yong-Liang Yang, and Niloy Mitra. BlockGAN: Learning 3D object-awas scene representations from unlabelled images. NeurIPS, 2020. 3,9
Thu Н Nguyen-Phuoc, Chuan Li, Stephen Balaban, and Yongliang Yang. Rendernet: A deep convolutional network for differentiable rendering from 3D shapes. In NeurIPS, 2018. 1,
Atsuhiro Noguchi and Tatsuya Harada. RGBD- GAD: Unsupervised 3D representation learning from
natural image datasets via rgbd image synthesis. In ICLR, 2020. 9
Arghya Pal and Vineeth N Balasubramanian. Zero-shot task transfer. CVPR, 2019.
Eunbyung Park, Jimei Yang, Ersin Yumer, Duygu Ceylan, and Alexander C Berg. Transformation- grounded image generation network for novel 3d view synthesis. In CVPR, 2017. 1,3
Xi Peng, Zhiqiang Tang, Fei Yang, Rogerio S Feris, and Dimitris Metaxas. Jointly optimize data augmentation and network training: Adversarial data augmentation in human pose estimation. In CVPR, 2018.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2016. 4
Sai Rajeswar, Fahim Mannan, Florian Golemo, Jérôme Parent-I éévesqee David Vazquez, Derek Nowrouzezahrai, and Aaron Courville. Pix2Shape: Towards unsupervised learning of 3D scenes from images using a view-based representation. IJCV, 2020.
Krishna Regmi and Ali Borji. Cross-view image synthesis using conditional gans. In CVPR, 2018.
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint 1765v.1706.05098, 2017. 3,8 8
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NeurIPS, 2016. 7
Shibani Santurkar, Andrew Ilyas, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Image synthesis with a single (robust) classifier. In NeurlPS, 2019. 3
Eli Schwartz, Leonid Karlinsky, Joseph Shook, Sivan Harary, Mattias Marder, Abhishek Kumar, Rogerio Feris, Raja Giryes, and Alex Bronstein. Delta-encoder: an effective sample synthesis method for few-shot object recognition. In NewrlPS, 2018. 2

---

Published as a conference paper at ICLR 2021
Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer. Deepvoxels: Learning persistent 3D feature embeddings. CVPR, 2019. 1,3
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NeurlPS, 2017. 1, 2, 4, 5, 6, 7, 17
Trevor Standley, Amir R Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In ICML, 2020.
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. In CVPR, 2019. 2
Shao-Hua Sun, Minyoung Huh, Yuan-Hong Liao, Ning Zhang, and Joseph J Lim. Multi-view to novel view: Synthesizing novel views with self-learned confidence. In ECCV, 2018. 3
Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales.
Learning to compare: Relation network for few-shot learning. In CVPR, 2018. 17
Sebastian Thrun. Is learning the n-th thing any easier than learning the first? In NeurIPS, 1996. 2
Satoshi Tsutsui, Yanwei Fu, and David Crandall. Meta-reinforced synthetic data for one-shot fine-grained visual recognition. NeurlPS, 2019.
Richard Tucker and Noah Snavely. Single-view view synthesis with multiplane images. CVPR, 2020. 3
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint urXiv: 2016.
Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie. Building bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In CVPR, 2015. 15
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In NeurlPS, 2016. 2, 17
Yu-Xiong Wang and Martial Hebert. Learning to learn: Model regression networks for easy small sample learning. ECCV, 2016. 1,
Yu-Xiong Wang, Ross Girshick, Martial Hebert, and Bharath Hariharan. Low-shot learning from imaginary data. In CVPR, 2018. 2, 6, 17
Zhangyang Wang, Shiyu Chang, Yingzhen Yang, Ding Liu, and Thomas S Huang. Studying very low resolution recognition using deep networks. In CVPR, 2016. 2,5
Welinder. Branson, Mita, C. Wah, Schroff, Belongie, and P. Perona. Caltech-UCSD California Institute of Technology, 2010. 6, 14,
Birds 200. Technical Report 17
Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. SynSin: End-to-end view synthesis from a single image. In CVPR, 2020. 3, 7
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. In NeurIPS, 2020.
Yongqin Xian, Saurabh Sharma, Bernt Schiele, and Zeynep Akata. F-VAEGAN-D2: A feature generating framework for any-shot learning. In CVPR, 2019. 3
Yang Xiao and Renaud Marlet. Few-shot object detection and viewpoint estimation for objects in the wild. In ECCV, 2020.
Wei Xiong, Yutong He, Yixuan Zhang, Wenhan Luo, Lin Ma, and Jiebo Luo. Fine-grained image-to- image transformation towards visual recognition. In CVPR, 2020.
Linjie Yang, Ping Luo, Chen Change Loy, and Xiaoou Tang. A large-scale car dataset for fine-grained categorization and verification. In CVPR, 2015. 6, 14, 17
12

---

Published as a conference paper at ICLR 2021
Yibo Yang. Zhisheng Zhong, Tiancheng Shen, and Zhouchen Lin. Convolutional neural networks with alternately updated clique. CVPR, 2018.
Zhichao Yin and Jianping Shi. Geonet: Unsupervised learning of dense depth, optical flow and
camera pose. In CVPR, 2018. 3
Jac Shin Yoon, Kihwan Kim, Orazio Gallo, Hyun Soo Park, and Jan Kautz. Novel view synthesis of dynamic scenes with globally coherent depths from monocular camera. In CVPR, 2020. 3,
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-worte world: A benchmark and evaluation for multi-task and meta reinforcement learning. In CoRL, 2020.
Amir R Zamir, Te-Lin Wu, Lin Sun, William Shen, Bertram E Shi, Jitendra Malik, and Silvio Savarese. Feedback networks. In CVPR, 2017. 3
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. CVPR, 2018. 1,3
Feifci Zhang, Tianzhu Zhang, Qirong Mao, and Changsheng Xu. A unified deep model for joint facial expression recognition, face synthesis, and face alignment. TIP, 2020. 3
Hongguang Zhang, Jing Zhang, and Piotr Koniusz. Few-shot learning via diliency-guideer hallucina- tion of samples. CVPR, 2019.
Ruixiang Zhang, Tong Che, Zoubin Ghahramani, Yoshua Bengio, and Yangqiu Song. MetaGAN: An adversarial approach to few-shot learning. In NeurlPS, 2018. 2, 17
13

---

Published as a conference paper at ICLR 2021
APPENDIX
Input
Conv2D [in, out]
Instance Norm
Conv2D [out, out]
Instance Norm
Output
Conv2D [in, i]
Instance Norm
Figure A: Architecture of ResBlock used in the view synthesis module. The default kernel size is 3 and the stride 1.
A DETAILED ARCHITECTURE OF VIEW SYNTHESIS MODULE
One of the central components in our view synthesis module is the ResBlock adapted from ResNet (He al., 2016), where use Instance Normalization instead of Batch Normalization (Ulyanov et al., 2016; Toffe & Szegedy, 2015). Figure A shows the architecture of a 2D ResBlock. For all the convolution layers in this structure, the kernel size is 3 and the stride is 1. By changing the 2D convolution layers to 3D convolution layers, we will get a 3D ResBlock. Figure B shows the architecture of the discriminator D. The kernel size is 3 and the stride is for all the convolution layers. The structure of the generator G is illustrated in Figure C. Notice that the "Res-Up" module is the structure of ResBlock, followed by an upsampling layer. The kernel size is 3 and the stride is I for the convolution layers.
B PSEUDOCODE OF TRAINING ALGORRITHM
To have a better understanding of the training procedure FBNet, we include Algorithm to show the detailed training process on base classes. The training novel classes follows similar process, except the sample number n (n 1 when we conduct 1-shot training).
C ADDITIONAL EXPERIMENTAL DETAILS
Data Pre-processing: For the CUB (Welinder al., 2010) dataset, we first square crop the im- ages with the given bounding boxes, and then we resize the cropped images to 64-resolution. For the CompCars (Yang al., 2015) dataset, slightly different from the original paper and Holo- GAN (Nguyen-Phuoe et al., 2019), we follow the instructions of the publicly released CompCars dataset for classification tasks, and obtain 366 classes of car models after standard pre-processing. We then manually drop 6 classes which have fewer than 15 images per class, and construct proper dataset for our experiments.
Additional Implementation Details: In main paper, we set Aia 10 and A = via cross- validation, and found that the performance is relatively stable to the setting of these trade-off hyper-parameters. We sample 5 images per class for Chase and I image for Crovet- We use Adam optimizer for all the networks. The learning rate is set to 5e 5. The final dimension of the feature extraction network is 1.000. The hidden size of all the three fully-connected layers 128. and the
hhttp://mml ab. le. cubk edu. wwt.hhhkddtttteeeeee cotpatttasettsccccrraaaas
14

---

Published as a conference paper at ICLR 2021
Res2D [512, 51.21
Ang Pool
FCN [512.
Redt [4,4
Res2r 128]
RedD [1288, 256]
RedDD [256, 512)
[512. 512)
Pool
[512, 1311
Figure B: Architecture of the discriminator in the view synthesis module. The kernel size of all the convolution layers is and the stride is 2.
20
Proger
Rolly 3D
[18,99
Figure C: Architecture of the generator in the view synthesis module. Res-Up module is combination of ResBlock an upsampling layer.
final feature dimension of the prototypical classification network is also 128. The batch size is 64 for the view synthesis module. We train 1,400 iterations for Crase and 100 iterations for Cnovel-
Algorithm 1: Training process of FBNet on base classes.
Initialization:
mar_ it: Maximum iteration for the training;
R: Recognition module, V: View synthesis module;
F: Feature extraction network, Fnigh: Feature extraction network with high-resolution images;
G: Generator of view synthesis module, D: Discriminator of view synthesis module;
n: Number of support images per class, n 5;
for iler &- to manr_ierr do (), Sangmented = ():
Ssapport (), Squery
for c E Crase do
support_ime iim sample images in c;
query ims +- sample image in o;
Support f Suppon' support ims
Squery t SqueryU
end
Snigh +- Foigh (S (Suupon U Squery)
Slow
F(5 (Suuppon U Squery)
for img in Ssupport do
R(ing):
z = f @ N:
0 sample view angle;
imng' + G(z,0);
y D(img):
",,2,,0'=D((mgg) D(img')
LGAN(Y, ing. y1, immg)) update G, D;
Lia(z, 0, =', 00)) update D;
Saugmemted &- Sangmented U sing'
end
Swhnle Sauppon U S Sugmented
Gree (Swhalee Squerry) - update R:
Creature (Jhigh, freat) update F;
Leal ( (Suppon, Saugmented) - update G:
end
D EXPERIMENTS ON ADDITIONAL DATASETS
To show the effectiveness and generality of proposed FBNet, conduct experiments on two additional datasets: the North American Birds (NAB) dataset (Van Horn et al., 2015) and the Stanford Dog dataset (DOG) (Khosla al., 2011). There 555 classes with 48,527 images in NAB dataset.
15

---

Published as a conference paper at ICLR 2021
Model
Base
-ovel-K=1
Novel-K<55
FBNet-ree
44.56
23.97 + 0.05
58.09 0.19
NAB
FBg
44.85
23.69 0.08
58.40 + 0.26
FBNet
45.63
24.15 + 0.07
58.98 + 0.15
FBNet-rec
51.13
53.33
0.09
72.59 0.17
DOG
FBNet-aug
51.46
53.11 + 0.11
72.68 0.15
FBNet
52.25
53.78
0.08
73.21 0.14
Table A: Top-1 (%) recognition accuracy on the NAB and DOG datasets. For base classes: 350-way classification
on NAB and 80-way classification on DOG; for K-shot novel classes: 205-way classification on NAB and way classification on DOG. Again, FBNet consistently achieves the best performance for both base and novel classes.
(D)
Model
Base
Novel-K=1
Novel-X=5
Base
Novel-K=I
Novel-KE5
Images
3.47
3.88
0.19
NAR
HoloGAN (Nguyen-Phusee Phuoc et nl., 2019)
4.06 + 0.08
2.52 + 0.04
2.65 + 0.06
47.52
85.74
76.38
0.12
0.04
0.08
40,00
74.55
Real Images
8.02 + 0.36
4.18
4.91 0.19
0
0
0
DOG
HoloGAN (Nguyen-Phuoc et al., 2019)
6.06 = 0.29
0.11
0.17
53.85
82.64
73.21
FBNet
6, = 0.32
0.14
4.02 0.21
45.17
79.25
69.66
Table B: Quantitative results of novel-view synthesis under the FID and IS metrics on the NAB and DOG
indicates that higher is better, and J. indicates that lower is better. FBNet consistently outperforms the baselines.
datasets. 1
NAB
DOG
Base
K=5
K=1
Basc
K=$
K=1
HoloGAN
FBNei
Figure D: Qualitative comparison of synthesized images from multiple viewpoints between our FBNet and state-of-the-art HolaGAN on the NAB and DOG datasets. Images in the same row/column are from the same viewpoint/object. The overall quality of the synthesized images for both methods indicates the general difficulty of the task, due weak supervision and lack of data. However. FBNet still captures the shape and attributes well, even though the data is scarcely limited. This shows the strong adaptability of FBNet few-shot learning.
We randomly select 350 classes as base classes and 255 classes as novel classes. For the DOG dataset, there are 20,580 images belonging to 120 classes. We randomly select 80 classes as base and 40 classes as novel. For each class of the two datasets, we randomly select 75% as training data and 25% as test data. We pre-process these two datasets in a similar way as the CUB dataset.
Slightly different from the evaluation in the main paper, we only compare our FBNet with Holo- GAN (Nguyen-Phuoc al., 2019) for the view synthesis task. For the recognition task, we compare our method with FBNet-rrec and FBNet-aug. All the experimental setting remains the same as that in the main paper on the CUB and CompCars datasets.
Quantitative Results: Table A shows the recognition performance of the three competing models. Again, our method achieves the best performance for both base and novel classes. Table B shows the results of view synthesis and our method also achieves the best performance.
Qualitative Results: Figure D shows the synthesized images by HoloGAN and our FBNet on the two datasets. First, we note that the overall quality of the synthesized images for both methods becomes substantially worse than one would expect with large amounts of training images. This demonstrates the general difficulty of the task due weak supervision and lack of data, indicating the need for the community to focus on such problems. Second, our FBNet significantly outperforms the state-of-the-arr HoloGAN, especially for the diversity of the synthesized images. Additionally, even though the data is scarcely limited, FBNet still captures the shape and some detailed attributes of images well.
16

---

Published as a conference paper at ICLR 2021
Setting PN
RN
MN
PMN
PN w/
MetaGAN
FBNet
Base
46.05
45.89
46.72
47.10
48.57
39.91
49.63
Novel-K== 20.83
22.14
21.62
22.78
22.71
18.59
23.28
Novel-K =
50.52
50.26
50.59
51.07
52.65
44.20
53.12
Table C: Top-- (%) recognition accuracy for our approach and other slate-o-ttheaae few-shot methods, including
data hallucination-based methods on CompCars. Our FBNet consistently outperforms the other models.
Novel-K=1
Novel
PN (Snell al,, 2017)
57.91
17.53
71.26
FBNet-PN
$9.43
48.39
72.76
CUB
RN (Sung 2018)
47.77
FBNet-RN
59.19
48.46
72.60
PN (Snell et al., 2017)
46.05
20.83
50.52
CompCars
FBNet-PN
49.63
23.28
53.12
RN (Sung 2018)
45.89
22.14
50.26
FBNet-RN RN
48.99
24.83
52.72
Table D: Top-1 1 (%) recognition accuracy for recognition modules with PN (prototypical network) and RN
(Relation Network), respectively, the CUB and CompCars datasets. For base classes: 150-way classification on CUB 240-way classification on CompCars: for K K-shot novel classes: 50-way classification CUB 120-way classification on CompCars. The proposed FBNet consistently improves the performance the single recognition modules for both PN and RN, indicating the generality of our framework.
E COMPARISON WITH OTHER FEW-SHOT RECOGNITION METHODS
We further compare our FBNet with variety of saaeeeof-thhe-ee few-shot recognition methods on the CompCars dataset (Yang al., 2015), including prototypical network (PN) (Snell al., 2017), relation network (RN) (Sung et al., 2018), matching network (MN) (Vinyals et al., 2016), and proto- matching network (PMN) (Wang al., 2018). We also compare with data dallucinationnbasse methods: PN w/ G (Wang et al., 2018) and MetaGAN (Zhang et al., 20.8). Table C shows that FBNet consistently outperforms these methods. Importantly, note that these methods only address few-shot recognition, while our FBNet is able to deal with the joint task.
F ADDITIONAL EXPERIMENTAL RESULTS WITH RELATION NETWORK
The proposed feedback- based framework has great generalization capability. In the main paper, we have shown that it is flexible with different backbone feature extraction networks F. In addition. our FBNet could improve the performance of the recognition module with different classification networks P through feedback connections. To show this, we change the prototypical network (Snell al., 2017) to the relation network (Sung al., 2018) and report the recognition result on the CUB (Welinder et al., 2010) and CompCars (Yang et al., 2015) dataset in Table D. The experimental setting remains the same that in the main paper. From the table, based models consistently outperform the single recognition modules for the prototypical network and the relation network. This indicates that proposed bowtie architecture is general and robust. framework that could improve the performance different types of classification networks.
that the whole feedback-
Basc
K=S
K=1
Figure E: Synthesized 128-resolution images by FBNet the CUB dataset.
17

---

Published as a conference paper at ICLR 2021
im2
iml
Figure F: FBNet with an extended attribute transfer task. 'iml' is the source image corresponding with 3D AdaIN and 'im2' is the attribute image corresponding with 2D AdaIN, Images in the same row/column have the same identitylatriibtts..
G EXPERIMENTS WITH SECHER-RESOLUTION IMAGES
In the main paper, following the state-of-the-att novel- synthesis model HoloGAN (Nguyen- Phuoc et al., 2019), we mainly focused on images of size 64 x 64. Our FBNet can also operate on higher-resolutinn resolutiosss by adding more 2D ResBlocks for the view synthesis module, and adding more convolution layers for the recognition module. Here, we modify our model to operate on images of size 128 128 and show some representative generated images on the CUB dataset in Figure E. We see that the proposed method effectively works with 128-resolution. We also note that higher resolution requires higher-quality training data. The unsatisfactory size and quality of the CUB dataset introduce additional challenges for synthesis, such as missing of details, inconsistency of identity across different viewpoints, and noisy background. However, such problems could be further addressed by improving the quantity and quality of the training data.
H FROM JOINT-TASK TO TRIPLE-TASK: ATTRIBUTE TRANSFER
The proposed bowtie framework could also be extended to address more than two tasks, by either introducing additional feedback connections or changing the architectures of the two modules. Here as an example, we introduce an additional "attribute transfer" task combined with the novel-view synthesis task. That is, instead of seeing one image each time, the view synthesis module sees one source image ('iml') and one target image ('im2') at the same time; then it generates images with the object of the source image and the attributes of the target image. We achieve this by arranging source latent input for 3D AdaIN units and target latent input for 2D AdaIN units in the view-synthesis module. Figure F shows the result of this additional task: images in the same row keep the same identity with iml; images in the same column have the same attributes of im2. Note that, to have better visualization, we use the predicted views from the source images as the view input for all the generated images in Figure F, but the model could still synthesize images of different viewpoints.
18


Reviews:
[{"review_id": "vhrkG7EkXVz", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new dual-task of joint few-shot recognition and novel synthesis. The main idea of this paper is to learn a shared generative model across the dual-task to boost the performances of both tasks. To achieve this, bowtie networks are employed to jointly learn geometric and semantic representations with a feedback loop. The proposed method is evaluated on fine-grained recognition datasets.\n\nPros:\n\n1. This paper is well-written and easy-to-follow.\n2. The idea of jointly learning the generative model and the proposed feedback loop is well-motivated.\n3. The experiments on evaluating the view synthesis module is comprehensive and the results are promising.\n\nCons:\n\n1. I think the contribution of this paper on the technical side is somehow weak. The recognition model and view synthesis modules are both adapted from existing works. The major contribution of this paper should be the bowtie architecture with the feedback loop, but this has also been well-studied in the literature.\n\n2. During the evaluation of the recognition model, the proposed method should also be compared with other state-of-the-art methods on the same datasets other than only the variants of the proposed method. Although the authors claim that this is not the major goal of this paper, I think this is vital since one major benefit claimed by the author is that jointly learning the generative model can improve both tasks.\n\n3. In Table 3, it is interesting to see that there is a performance gap when using simple (ResNet-18) and deeper (ResNet-50) models. Are there any intuitive explanations for this gap?\n\nOverall, I like the main idea of the proposed method which learns a shared generative model across the dual-task. However, there are some concerns about the technical contributions and experiments. I will be happy to increase my rating if these concerns can be addressed in the rebuttal period.\n\n---\nUpdate after author feedback: I thank the authors for their reply. The authors have addressed all of my concerns. Therefore, I increased my final rating.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "btH400nW3zb", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a model for joint few-shot recognition and novel-view synthesis. As shown in Figure 2, the model consists of two modules: view synthesis module and recognition module. The authors claim that these modules can help each other to become better. The two modules are trained jointly on each task alternatively.\n\nPros: \n1. As both the justifications and experiments show, these two modules can help each other to be better.\n2. Given the few-shot setting, the help from another task is meaningful.\n3. The writing is clear, and makes me easy to follow.\n\nCons:\n1. The novel-view synthesis quality is bad, so it can barely used for any other purposes besides assisting few-shot recognition module. In other words, the few-shot recognition result should be the only final output, and the novel-view synthesis output should only be considered as an intermediate result. So I think that the position of novel-view synthesis should not be lifted as high as the few-shot recognition, and this work should be dedicated to the few-shot recognition with a narrowed scope from \"joint few-shot recognition and novel-view synthesis\" to \"few-shot recognition\".\n2. Based on the above point, then I doubt whether it is necessary to synthesize the pixels of other views, because the pixel quality is bad. Synthesizing the intermediate feature maps should be more realistic in this case, because the pixels are mainly for human (reviewers), but the featuremaps are mainly for model (recognition module). Human does not make the final few-shot recognition result better, but the recognition module does.\n3. The experiments only show the fine-grained recognition results, e.g., fine-grained recognition for birds or cars. Given a single category (bird or car), the view synthesis quality is so bad, so I doubt if this module can be used for general recognition task involving many categories simultaneously. In that case, the novel-view synthesis module might make no sense at all. If that's the case, the scope of this work should be further narrowed from \"few-shot recognition\" to \"few-shot fine-grained recognition\".\n\nBased on above points, I suggest rejecting this paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "U7h17EvnuyX", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a \"feedback-based bowtie network\" FBNet for joint generative synthesis via a GAN-based framework (specifically HoloGAN) and few-shot fine-grained recognition. The key idea of this work is to supervise both networks jointly via feedback mechanisms between the two, which helps to improve both tasks: image synthesis and few-shot recognition. The authors propose to use the synthesis network for synthesizing augmented images and additional losses computed by the image classification network along with conditional generation to improve the quality of the synthesized images.\n\nPros:\nThe authors consider a new setting of coupling HoloGAN with a downstream image recognition network and jointly training the two together. They show improvements both in image synthesis quality and image recognition by their approach. The experimental section is fairly thorough with many analyses and results presented.\n\nCons:\n1. The proposed work's idea of coupling the synthesis HoloGAN network with a downstream visual learning task though a feedback mechanism between the two is not novel. This approach was previously introduced in the work Mustikovela et al., \"Self-Supervised Viewpoint Learning From Image Collections\", CVPR 2020, which bears much resemblance in approach to the current work, but trains for a different downstream task of viewpoint estimation versus few-shot categorial classifications. Mustikovela et al. also employ a conditional synthesis and various similar task-specific losses to jointly supervise both networks. The authors of this work should clearly cite this prior work and reframe the novelty of their approach in relation to it. \n\n2. The paper lacks comparisons to existing SOTA few-shot learning techniques that employ strategies for hallucinating additional data/features for the training classes in the few-shot training settings, e.g., Wang et al., 2018 and Zhang et al., 2018. Is the proposed approach better than the previous few-shot learning approached that hallucinate data per class?\n\n3. What dataset was the high resolution recognition network trained on?\n\n------------\nPost Rebuttal:\nI thank the authors for their response. I am mostly satisfied with the authors' response to my (and other reviewers') concerns about properly citing prior works that jointly consider coupled image generation with downstream tasks and reframing the novelty of their work in juxtaposition to them. I would like to point out, however, that the authors' statement in the rebuttal \"(2) we achieve bi-directional feedback while this work only implements the feedback from viewpoint estimation task to the generative network.\" is technically incorrect. The viewpoint estimation network in Mustikovela et al. is directly trained with images generated by the synthesis network under various viewpoints and hence it also achieves bi-directional feedback much like this current work. The authors should clearly re-frame their novelty and make this correction in the final version if accepted.\n\nNevertheless, I do feel that this work adds to the body of literature on joint conditional synthesis coupled with downstream vision tasks by (a) showing improvements in the quality of image synthesis achieved by considering downstream tasks and (b) by showing improvements in few-shot learning versus prior methods where only features are hallucinated, and (c) considering other applications beyond viewpoint estimation. Hence its contribution is above the acceptance threshold. I will maintain my previous rating.", "extra_scores": {}, "extra_text": {}}, {"review_id": "-R9Wu0NIlpF", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nWell written paper with solid experiments on an extension of two prior works. This is likely of interest, good quality and I recommend to accept the paper at ICLR. There are no extensions that I would propose to include in this version.\n\nQuality: \nGood quality, well written paper, easy to follow and sufficiently detailed. Content is on-topic for ICLR and of interest to a general audience. \n\nClarity: \nThe paper is very well written, all details on the model, the training procedures, experiments are included. This paper is well polished and was easy to read and follow. The main assumptions are stated early on, problem definition is well stated, and the goal of the experiments are clearly stated before going into their discussion. The Appendix provides additional results and details. Nice paper to read, thanks for putting in the effort. \n\nOriginality: \nThis paper is a combination of two tasks, combining two prior works to one system. The original part is the research question on whether the two tasks should share a common feature space and whether the results improve by the network model. So this is an interesting paper, I would assume there is quite an audience that is interested in this topic. Without doubt the model is well constructed and trained, so there is also value in the construction. \n\nI have not seen the task of (few shot) recognition and visual reconstruction seen so far. This paper is a good extension of HoloGAN and has some novel points.\n- Conditional version of HoloGAN. This is a simple extension but useful and serves the purpose. \n- Combination of view synthesis and recognition. The flow of the architecture is well explained and leads to empirical improvements over each task in separation. More architecture choices would be possible, an evaluation of different backbones is included but not of other network combinations.\n- Experimental results are sufficient, on established dataset, there is no novelty in the application. \n\nSignificance: \nFor both tasks (view synthesis and reconstruction) there are stronger models. The authors claim that other models could be combined in their setup, I agree, but the empirical results are below state-of-the-art. But this is definitely a step in the right direction and I believe there is an interested audience for this finding and it is likely that the construction inspires future work. There are some extensions that would go beyond the paper, such as more challenging data, images with more than one object, and combination with even more vision tasks.\n\n\n\n----\nUpdate and final recommendation. \nI still recommend acceptance of the submission. The paper is well written, results stand on its own and the numbers improve in the way described. In light of the missing comparisons to other works pointed out by the fellow reviewers I have lowered my score because I think better calibrates with the significance of the work. Combination of downstream tasks is not novel but this combination I have not seen and so even bearing similarity with other approaches the paper still stands on its own. \nThanks to the reviewers and authors for their responses.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper uses an extension of HoloGAN for few shot recognition and novel view synthesis. All but one reviewer gave a final rating of accept. These reviewers were concerned that the submitted version of this work had not adequately placed this work in context with prior art. However, during the discussion these concerns seem to have been addressed sufficiently. The most negative reviewer was not impressed by the quality of the generated images; however these are relatively new methods and the few shot recognition aspect of this work is also part of the contribution. Accounting for all reviews and the discussion the AC recommends accepting this work as a poster. 
Decision: Accept (Poster)