Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper presents a modular network architecture for few-shot concept learning. The architecture consists of image-to-scene module that maps input RGBD images to 3D scene features and an object-centric disentangling auto-encoder that crops object features to generate shape and style codes, and finally a neural rendering module that put back the reconstructed object and background features to image domain. The proposed network is verified in few-short recognition task and VQA task with comparisons to state-of-the-art methods. 

+This paper is well-written and the core ideas and network design are well illustrated and explained in the paper and supp video. This paper also presents comprehensive experiments including sufficient details in supp to support the claims, which also makes the paper more reproducible. 
+State-of-the-art results on few-shot recognition, especially on one-shot recognition. The object-centric disentangling and 3D shape prototype learning seem to play complementary roles for the results. 

-The success of the proposed method seems to be limited to the data being considered in this paper, rigid 3D objects with different appearances. It is not clear how the proposed methods, e.g. AdaIn disentangling and rotation invariant prototype can be easily generalized to other scenarios including deformation (pointed out in the paper) and part-based composition. 
review_title: Well-designed neural architecture for few-shot concept learning with state-of-the-art results

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
The paper claims that the main contribution is "to identify the importance of using disentangled 3D feature representation for few-show learning". This is a great goal, but in my view the paper does not deliver on that front. There are several issues from my perspective

1) The model formulation aims to disentangle shape from style in a particular formulation. To achieve training of this disentanglement it requires large amounts of multi-view data of highly simplified (and in this case simulated) scenes of at most few objects at the time. I am not convinced that we will ever have such data except for highly specialized situations. Unfortunately, the paper does not discuss this major limitation and does not make any attempt to convince the reader that this is a sensible starting point for further work. The only attempt to use real data is a custom dataset called "real world veggie" that has never been used before and it's characteristics are very unclear (number of classes, number of images, number of scenes, etc - also no sample images are given)

2) In order to show the "importance of using disentangled 3D feature representation" one would expect a set of experiments that shows comparable models with and without some form of disentanglement. However, the paper does not seem to contain any such experiments as far as I understood.

3) While there are several experiments given, it is unclear how valuable the results and comparisons are. It seems that most if not all of the explored settings have not been addressed in the other works that this paper compares to. E.g. the paper [Mao et al 2019] does not seem to report any of the results given in Table 2. Similarly, the referenced papers that I checked from table 1 did not seem to report those results or this respective setting. This makes it very hard to understand if the proposed approach really has any benefit when it is used only in non-published and non-standard settings

So while the model formulation is quite sensible and combines in a meaningful way previous ideas and approaches, it remains unclear what can be taken away from this paper beyond the rather specific experiments mostly on simulated data (and some unclear real world veggie experiments)


========= post rebuttal ========

Thanks for adding the experiments for Replica - these at least seem to suggest that the approach can work on more complex scenes than shown initially in the paper. 

I still think the training scheme and required data is relatively specific for this approach that it is hard to see that it will generalize beyond - but that is probably fine for a research paper.  

that addresses Q1

for Q2 - please include some information in the main paper - not everyone will check the supplement and the paper should be self-contained. 

I still find the comparisons and ablations weak as the particular training setup and model are the key contribution for the paper and thus without a proper ablation it is hard to know what exactly to take away. Therefore I still remain somewhat skeptical but have raised my score somewhat. To me the paper is still borderline and thus somewhere between 5 and 6 really. 
review_title: Unclear what can be really learned from the proposed method and experiments

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper describes an approach that learns a disentangled shape and style representation of objects in a self-supervised way from RGB-D images.  The approach is based on various components, like a 3D feature volume, a bounding box detector, and a disentanglement network. Neural rendering (e.g. recomposing the various disentangled parts into an image) is used as the learning signal for self-supervision. Various applications of this representation  are shown, examples are few-shot shape learning and Visual Question Answering.

--- Strengths ---

Some applications that are enabled by the representations are very interesting. Disentangling style and shape for example allows to detect object independently of style (something easily done by humans) and generation of scenes from language utterances. Overall, it seems like a good direction to try to go for a full 3D(+style) representation to get more flexible and general models. The experiments indicate that the main concepts help in the down-stream tasks.

--- Weaknesses ---

Only synthetic samples are shown. Since the approach is fully self-supervised, it should be possible to apply it to real images. The authors show quantitative results on one, apparently, real dataset "Real veggies", but no qualitative samples are shown to give the reader an indication on how realistic this dataset is. Is it realistic? If not, what is preventing applying this approach to more realistic images and objects?

The other propose a very elaborate, modular pipeline. While this modularity does explicitely enable some of the applications that are shown, it also could be a weakness, as some modules present single points of failure which might be hard to recover from (for example when the 3D object detector fails). This could be alleviated by either showing results on realistic data and/or a controlled studies of how robust the approach is the to failures of individual parts (e.g. what happens when the detector is inaccurate?). 

The paper is a bit hard to read because it is crowded. I suspect that it due to he many concepts that are introduced. The  main contribution clearly is the self-supervised learning of the representation, but a sizeable part of the paper discusses down-stream applications (which are non-trivial and are thus not discussed in sufficient detail), while other parts (for example the feature selection) would require a bit more space to be easier to understand. 

--- Other ---

The example for one-shot scene generation in Figure 1 is exceptionally clean. Is this a real example that was rendered by the proposed approach?

Are you planning to release the code? An elaborate system like this would benefit strongly from this.

I suspect that the submission is severely over length. Starting from page 5, the font size is much smaller than the default font size.

Typos: page 3: "operation operation"

-- Summary ---

The proposed representation is interesting, but the huge number of applications makes the paper inaccessible. I'd argue it would be beneficial to explore and understand the representation and pipeline on a more fundamental level (i.e. can it be learned on real images? what about individual inaccuracies? when does the approach fail? what is the influence of the granularity of the 3D feature map? ....), than to jump right into elaborate higher level tasks.


--- Post rebuttal ---
I'd like to thank the authors for the response. I've updated my score in light of these additional results.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents a framework that combines 1) multi-view prediction for 3D reconstruction from a single image and 2) content-style disentangled representation learning using instance-norm-based auto-encoders. It aims at learning disentangled 3D representation of input images. The authors show that these methods can be applied to few-shot recognition, visual question-answering (VQA), view prediction, and image generation from texts.

The paper is well-written. It's impressive that the authors were able to build such an integrated system with so many modules. The experiments are quite extensive.

My concerns are mostly about experimental evaluation. There are some important studies missing, and other results not well-justified.

First, some important ablation studies are missing, especially in Section 4.1. The authors have used three different techniques to improve the few-shot recognition performance: 1) multi-view predictive learning, 2) content-style disentanglement, and 3) optimization-based prototypical network (the authors used gradient descent to find the rotation of the observation w.r.t. the prototype). However, the only ablation study available in Table 1 is the 3DP-Net, which uses only (1) multi-view predictive learning. Moreover, I suggest authors test the performance of other algorithms for few-shot recognition, for example, a very simple baseline algorithm from Chen (2020, https://arxiv.org/abs/2003.04390 ). Is the proposed representation algorithm only working with prototypical networks?

Second, Table 2 also misses an important baseline. Looking at the in-domain test set, we have seen that NSCL-2D has a better performance than D3DP -- the authors explained that this is because the NSCL-2D uses ImageNet pretraining. What will be the performance of NSCL without ImageNet pretraining? On the one-shot generalization test split: what will be the performance of D3DP without 3D shape primitive and without shape/style disentanglement? It seems that even with one of these two features, the proposed D3DP still has worse performance than the 2D network. It'd be great to hear from the authors if there is any intuition on the result.

I also have serious concerns about the generality and applicability of the claimed shape-style disentanglement. From the results in Table 1 (comparing D3DP and 3DP) and Table 2 (comparing D3DP w/. and w/o. style/content disentanglement), it seems that the shape-style disentanglement has shown great importance in few-shot learning settings. But, in "style features", how should materials and colors be disentangled; in "content features", how should object textures and shapes be disentangled? The limitation on generality will greatly restrict the application of this framework in real-world images. 

This is also related to another minor comment about the setup of few-shot style recognition in Table 1. In CLEVR, there are 8 colors and 2 materials. So the "16 style classes" in the paper indicates that the authors are treating pairs of (color, material) as the label for objects. Such a design will not scale up wrt the type of attributes. I'd love to hear more justifications from authors. And this should be more clearly indicated in the main paper as well.

Update: I appreciate the response and have adjusted my score accordingly.
review_title: Interesting submission, but experiments are not well justified

Meta-review: This meta-review is written after considering the reviews, the authors’ responses, the discussion, and the paper itself.

The paper proposes a system for learning disentangled object-centric 3D-based representations of scenes and shows that the proposed model works well on several tasks, including few-shot classification and VQA. 

The reviewers point out that the direction is important (R1, R3), the model is sensible (R2), and the reported results are good (R1, R4); on the downside, they note that the system is complicated (R1), the considered datasets are relatively simplistic (R1, R3, R4), some ablations are missing (R2, R3), and comparisons with baselines are not necessarily convincing (R2, R3). The authors did a good job of addressing the concerns in the rebuttal, by reporting additional ablation results, baselines, and experiments on the realistic Replica dataset.

All in all, I recommend acceptance. The direction of the work is important and complex, the experimental evaluation presented in the paper is extensive, and the results are good relative to relevant baselines. On the downside, the proposed system and the paper are somewhat complicated and overwhelming, which may limit the benefit for the readers. I hope the authors will take this into account in the future.

Decision: Accept (Poster)