Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Strong Points:
1) The paper was fairly well written and easy to follow.
2) The results are quite promising.

Questions to the author:
1) It is unclear how the model learns the category without any supervision. It is recommended to provide more explanations as it is the major contribution.
2) It is unclear how \pi_c in equation 2 is determined. Does it follow the same distribution of the dataset?
3) For figure 2, why was the proposed CIGMO not compared to GVAE plus k-means or IIC?
4) For figure 3A, are the numbers of examples balanced in the 3 categories?
5) For figure 3B (swapping), based on data preparation, shape should capture identity, but it is not obvious from the figure. Any thoughts?
6) For the experiment with MultiPIE, grouping by emotion category might reveal more about the model.
7) The model is C(category) times larger than GVAE. Is it possible that the performance gain comes from using larger model?

review_title: This paper proposed a generative method, which learns to disentangle the category, shape, and view features of an image using weak supervision. The weak supervision is provided by grouping the images of same shape together and enforcing them to converge in the feature space. The authors performed two tasks: invariant clustering and one-shot classification, to demonstrate the effectiveness of the proposed method.

final_score: rating: 7: Good paper, accept
review_content: ============================
Summary

This paper proposes a categorical invariant generative model (CIGMO) from a set of 2D images that tries to disentangle the factors of data category, intra-category geometry, and rendering viewpoint. CIGMO trains a VAE with a hierarchical graphical model that explicitly factors out the three components by design. Experiments on two datasets (ShapeNet rendered images, MultiPie face images) show that the proposed method can discover the concept of data category without using explicit supervision. It also supports feature manipulation over the geometry and viewpoint factors.


============================
Pros

1) The proposed task and approach are valid, reasonable and technically sound. Also, the paper is very easy to follow.
2) This paper brings in a novel approach that explicitly models data category as a crucial factor for learning disentangled representations. 
3) Experiments show that given no supervision, the clustering based on the category latent variables makes sense. And, with 1-shot supervision, the classification performance is better than baseline methods.
4) The learned geometry and viewpoint factors are well disentangled, as shown in the manipulation tasks.


============================
Cons

1) Can you show some quantitative evaluations or quantitative user studies to show the quality of the learned geometry and viewpoint disentanglement? How would you compare to previous works? I understand that your primary goal is to introduce data category factor into the game, but it is helpful for readers to know how much your model perform in terms of disentangling the standard two factors: geometry and viewpoint, and how do you compare to previous works?
2) In the manipulation experimental figures you show in the paper, the reconstruction quality seems pretty bad. Especially for the experiments on face dataset (Fig 3, B), it's really hard for me to tell the person identity from the reconstructed face images. Can you provide more discussions/explanations on this?

============================
Overall Rating

I like the novel task/method in this paper that brings in explicit modeling for the object category factor. Experiments comparing to baseline methods on unsupervised object category clustering and 1-shot object classification is quite convincing to me.
review_title: Official Blind Review #4

final_score: rating: 5: Marginally below acceptance threshold
review_content: Strength:
- The proposed model can simultaneously model the category, shape, and view factors of general object images.
- The supervision is only from the view-grouping information, which can be easily obtained. 

Weakness:
- The proposed model is a mixture of disentangling models, which is not in an end-to-end manner.
- The authors only compare the proposed model with other methods in two down-stream tasks. The authors should compare the generated results directly in quantitative and qualitative ways.
- Experimental results are not convincing.  The quantitative comparisons of invariant clustering and one-shot classification are interesting settings and clearly show good performance of the proposed method. However, it does not seem very fair to compare methods with differently structured input. As the GVAE baseline does not model category (C=1) and the mixture of VAEs are trained with ungrouped data (K=1), it is easy to see why the proposed method performs better in such tasks. Adding a stronger baseline with the same level of supervision (grouped input) would make the comparison more convincing. In Figure 3 (B), the qualitative results of different views are not clearly shape-invariant. It would be clearer if the authors can show the swapping results on the ShapeNet objects (and possibly of other baseline methods). 

While the authors have validated the proposed model on the ShapeNet and MutliPIE datasets, the variations in these images are limited with simple backgrounds (in well-controlled lab setups). It is important to demonstrate whether the proposed method performs well on real-world datasets (e.g., LFW face, Pascal3D+). 

Novelty: The novelty of this work is generally good. The proposed model can discover the category, shape, and view factor at once, so as to generate general object images in a much larger space. The model capacity is much stronger than previous generative models which try to model general object images. What’s more, the model is trained in a weakly-supervised manner. The supervision information can be easily obtained. 

Presentation: The mathematical representation is easy to understand theoretically. On the other hand, the authors should give more initiative representations when describing the workflow of the proposed model and training procedure, which can make the paper much easier to understand. In addition, there should be more implementation details in the paper.

Evaluation: As discussed above, the authors only compare their method with other methods in two downstream tasks. As such, the experimental results are not convincing. Could the authors directly show the comparison results between the proposed method and other approaches quantitively and qualitatively?  For example, as the authors claimed, the proposed method can model the category, shape, and viewpoint of images at once, but other previous models cannot achieve this goal. Can the authors directly compare their results with the results from other models to support this claim?  The authors should provide such results to support their claim more directly.

Contribution: The contributions of this paper are good, which can simultaneously discover those factors of general object images.
Nevertheless, the experimental results are lacking to clearly demonstrate the merits of this work.
review_title: Interesting paper with some novelty but the experimental results are lacking to clearly demonstrate the merits.

final_score: rating: 4: Ok but not good enough - rejection
review_content: This work proposes a probabilistic model which disentangles view, class and shape attributes explicitly (it does not rely on an emergent phenomena of disentangled factors in the latent space). In comparison to a similar approach, GVAE, CIGMO additionally disentangles the content factor into category and shape factors. Obtained results are reasonable and show advantage of the proposed method over related approaches.

Strengths:
- I believe the idea, while not being novel compared to GVAE, shows that one can scale VAEs into disentangling more factors. I imagine the model that can additionally incorporate the data where additional factors are present such as explicit camera viewpoint.
- Theoretical grounding is well formulated.
- Baselines are well chosen and show superiority over the basic baseline that came to my mind while reading this work - mixture of VAEs.


Weaknesses:
- The authors mentioned that their approach is heavily influenced by GVAE but do not mention how these two differ in any principal way. I perceive CIGMO as an incremental improvement over GVAE. To mitigate this issue, it would be great to see this work as a meta approach over GVAE where the authors include more factors that are easily obtainable (since ShapeNet is rendered either way, hence the authors can extract illumination, camera position and other factors) and how including these factors influence the results. With the presented model formulation, it seems to be possible to include more such factors. Finally, I would argue that such a meta approach would increase the applicability and extensibility of the work significantly.
- Choosing 10 classes is not well supported. What classes are chosen? Does the final dataset include classes that are rare in the ShapeNet (besides, it would show how the model is able to handle such objects thanks to the new class factor)
- The main part lacks a good diagram of the method. The plate model is not sufficient since details that make the model converge in the first place cannot be presented in such a diagram. The one presented in the Appendix does more harm than good and I spent more time to understand it than to follow equations directly. 

Other comments:
- Overall, the paper is well written, however the authors should focus on typos present in the paper. Also, fixing or reformulating the sentence “our choice [ … ] is orthogonal to our goal” may reveal the main difference with GVAE. 
The authors refer to biological motivation which is not mentioned later in the work. In this case, the whole paragraph is highly irrelevant to me. It is also formulated in such a way that one would argue that mammals are limited in terms of how many objects they can recognize by shape.
review_title: Incremental improvement over GVAE

Meta-review: Description:
The paper presents a weakly-supervised model CICGMO for disentangling category, shape and view information from images. Label information is not need as  the weak supervision is done by grouping together different views of the same object. They show that this outperforms other techniques on tasks such as invariant clustering and one-shot classification.

Strengths:
- Paper is well written
- Data category is explicitly modeled 
- The weakly supervision approach is appealing,  since the grouped data used as supervision information is easy to obtain
- Invariant clustering and one-shot classification results outperforms other methods significantly, showing CIGMO is doing a decent job at those tasks. This could be explained by CIGMO ability to better  disentangle category-shape-view.

Weaknesses:
- It is unclear how well (quality) the generative model is able to disentangle shape from view
- The reconstruction quality is quite low, such that it is difficult often times, in the MULTIPIE example, to clearly identify a face geometry.
- Generated results are not evaluated directly, but rather evaluation is done through down-stream tasks such as invariant clustering. This makes it difficult to show the quality of shape and view information.
Decision: Reject