Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary: 
Well written paper with solid experiments on an extension of two prior works. This is likely of interest, good quality and I recommend to accept the paper at ICLR. There are no extensions that I would propose to include in this version.

Quality: 
Good quality, well written paper, easy to follow and sufficiently detailed. Content is on-topic for ICLR and of interest to a general audience. 

Clarity: 
The paper is very well written, all details on the model, the training procedures, experiments are included. This paper is well polished and was easy to read and follow. The main assumptions are stated early on, problem definition is well stated, and the goal of the experiments are clearly stated before going into their discussion. The Appendix provides additional results and details. Nice paper to read, thanks for putting in the effort. 

Originality: 
This paper is a combination of two tasks, combining two prior works to one system. The original part is the research question on whether the two tasks should share a common feature space and whether the results improve by the network model. So this is an interesting paper, I would assume there is quite an audience that is interested in this topic. Without doubt the model is well constructed and trained, so there is also value in the construction. 

I have not seen the task of (few shot) recognition and visual reconstruction seen so far. This paper is a good extension of HoloGAN and has some novel points.
- Conditional version of HoloGAN. This is a simple extension but useful and serves the purpose. 
- Combination of view synthesis and recognition. The flow of the architecture is well explained and leads to empirical improvements over each task in separation. More architecture choices would be possible, an evaluation of different backbones is included but not of other network combinations.
- Experimental results are sufficient, on established dataset, there is no novelty in the application. 

Significance: 
For both tasks (view synthesis and reconstruction) there are stronger models. The authors claim that other models could be combined in their setup, I agree, but the empirical results are below state-of-the-art. But this is definitely a step in the right direction and I believe there is an interested audience for this finding and it is likely that the construction inspires future work. There are some extensions that would go beyond the paper, such as more challenging data, images with more than one object, and combination with even more vision tasks.



----
Update and final recommendation. 
I still recommend acceptance of the submission. The paper is well written, results stand on its own and the numbers improve in the way described. In light of the missing comparisons to other works pointed out by the fellow reviewers I have lowered my score because I think better calibrates with the significance of the work. Combination of downstream tasks is not novel but this combination I have not seen and so even bearing similarity with other approaches the paper still stands on its own. 
Thanks to the reviewers and authors for their responses.
review_title: Well written paper on recognition + view synthesis approach

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose a model for joint few-shot recognition and novel-view synthesis. As shown in Figure 2, the model consists of two modules: view synthesis module and recognition module. The authors claim that these modules can help each other to become better. The two modules are trained jointly on each task alternatively.

Pros: 
1. As both the justifications and experiments show, these two modules can help each other to be better.
2. Given the few-shot setting, the help from another task is meaningful.
3. The writing is clear, and makes me easy to follow.

Cons:
1. The novel-view synthesis quality is bad, so it can barely used for any other purposes besides assisting few-shot recognition module. In other words, the few-shot recognition result should be the only final output, and the novel-view synthesis output should only be considered as an intermediate result. So I think that the position of novel-view synthesis should not be lifted as high as the few-shot recognition, and this work should be dedicated to the few-shot recognition with a narrowed scope from "joint few-shot recognition and novel-view synthesis" to "few-shot recognition".
2. Based on the above point, then I doubt whether it is necessary to synthesize the pixels of other views, because the pixel quality is bad. Synthesizing the intermediate feature maps should be more realistic in this case, because the pixels are mainly for human (reviewers), but the featuremaps are mainly for model (recognition module). Human does not make the final few-shot recognition result better, but the recognition module does.
3. The experiments only show the fine-grained recognition results, e.g., fine-grained recognition for birds or cars. Given a single category (bird or car), the view synthesis quality is so bad, so I doubt if this module can be used for general recognition task involving many categories simultaneously. In that case, the novel-view synthesis module might make no sense at all. If that's the case, the scope of this work should be further narrowed from "few-shot recognition" to "few-shot fine-grained recognition".

Based on above points, I suggest rejecting this paper.
review_title: The novel-view synthesis quality is bad, so the contributions of this paper are weakened.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a "feedback-based bowtie network" FBNet for joint generative synthesis via a GAN-based framework (specifically HoloGAN) and few-shot fine-grained recognition. The key idea of this work is to supervise both networks jointly via feedback mechanisms between the two, which helps to improve both tasks: image synthesis and few-shot recognition. The authors propose to use the synthesis network for synthesizing augmented images and additional losses computed by the image classification network along with conditional generation to improve the quality of the synthesized images.

Pros:
The authors consider a new setting of coupling HoloGAN with a downstream image recognition network and jointly training the two together. They show improvements both in image synthesis quality and image recognition by their approach. The experimental section is fairly thorough with many analyses and results presented.

Cons:
1. The proposed work's idea of coupling the synthesis HoloGAN network with a downstream visual learning task though a feedback mechanism between the two is not novel. This approach was previously introduced in the work Mustikovela et al., "Self-Supervised Viewpoint Learning From Image Collections", CVPR 2020, which bears much resemblance in approach to the current work, but trains for a different downstream task of viewpoint estimation versus few-shot categorial classifications. Mustikovela et al. also employ a conditional synthesis and various similar task-specific losses to jointly supervise both networks. The authors of this work should clearly cite this prior work and reframe the novelty of their approach in relation to it. 

2. The paper lacks comparisons to existing SOTA few-shot learning techniques that employ strategies for hallucinating additional data/features for the training classes in the few-shot training settings, e.g., Wang et al., 2018 and Zhang et al., 2018. Is the proposed approach better than the previous few-shot learning approached that hallucinate data per class?

3. What dataset was the high resolution recognition network trained on?

------------
Post Rebuttal:
I thank the authors for their response. I am mostly satisfied with the authors' response to my (and other reviewers') concerns about properly citing prior works that jointly consider coupled image generation with downstream tasks and reframing the novelty of their work in juxtaposition to them. I would like to point out, however, that the authors' statement in the rebuttal "(2) we achieve bi-directional feedback while this work only implements the feedback from viewpoint estimation task to the generative network." is technically incorrect. The viewpoint estimation network in Mustikovela et al. is directly trained with images generated by the synthesis network under various viewpoints and hence it also achieves bi-directional feedback much like this current work. The authors should clearly re-frame their novelty and make this correction in the final version if accepted.

Nevertheless, I do feel that this work adds to the body of literature on joint conditional synthesis coupled with downstream vision tasks by (a) showing improvements in the quality of image synthesis achieved by considering downstream tasks and (b) by showing improvements in few-shot learning versus prior methods where only features are hallucinated, and (c) considering other applications beyond viewpoint estimation. Hence its contribution is above the acceptance threshold. I will maintain my previous rating.
review_title: Interesting work, but missing important citation

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a new dual-task of joint few-shot recognition and novel synthesis. The main idea of this paper is to learn a shared generative model across the dual-task to boost the performances of both tasks. To achieve this, bowtie networks are employed to jointly learn geometric and semantic representations with a feedback loop. The proposed method is evaluated on fine-grained recognition datasets.

Pros:

1. This paper is well-written and easy-to-follow.
2. The idea of jointly learning the generative model and the proposed feedback loop is well-motivated.
3. The experiments on evaluating the view synthesis module is comprehensive and the results are promising.

Cons:

1. I think the contribution of this paper on the technical side is somehow weak. The recognition model and view synthesis modules are both adapted from existing works. The major contribution of this paper should be the bowtie architecture with the feedback loop, but this has also been well-studied in the literature.

2. During the evaluation of the recognition model, the proposed method should also be compared with other state-of-the-art methods on the same datasets other than only the variants of the proposed method. Although the authors claim that this is not the major goal of this paper, I think this is vital since one major benefit claimed by the author is that jointly learning the generative model can improve both tasks.

3. In Table 3, it is interesting to see that there is a performance gap when using simple (ResNet-18) and deeper (ResNet-50) models. Are there any intuitive explanations for this gap?

Overall, I like the main idea of the proposed method which learns a shared generative model across the dual-task. However, there are some concerns about the technical contributions and experiments. I will be happy to increase my rating if these concerns can be addressed in the rebuttal period.

---
Update after author feedback: I thank the authors for their reply. The authors have addressed all of my concerns. Therefore, I increased my final rating.

review_title: Official Blind Review #3

Meta-review: This paper uses an extension of HoloGAN for few shot recognition and novel view synthesis. All but one reviewer gave a final rating of accept. These reviewers were concerned that the submitted version of this work had not adequately placed this work in context with prior art. However, during the discussion these concerns seem to have been addressed sufficiently. The most negative reviewer was not impressed by the quality of the generated images; however these are relatively new methods and the few shot recognition aspect of this work is also part of the contribution. Accounting for all reviews and the discussion the AC recommends accepting this work as a poster. 
Decision: Accept (Poster)