Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a novel unsupervised learning method called COMET that discovers global and local factors of variation in image datasets by representing these factors as energy functions. The experiments show that the proposed approach performs better than beta-VAE in terms of disentanglement evaluation. Qualitative examples on various datasets show that the proposed approach captures global and local object-based factors of variation and is able to generate images from various combinations of such factors well.
main_review: Strengths:
- The proposed approach allows combination of factors of variation in a non-linear manner, enabling more complex relationships to be modeled.
- The authors show how the proposed approach is more expressive than the previous work that combines segmentation masks.
- Experiments on multiple datasets show the generalization capacity of the proposed approach.
- The paper is well written.


Weaknesses:
- Can the authors give examples where beta-VAE “is unable to compose multiple instances of the same component or a larger number of components than seen during training”?
- The evaluation is lacking:
    - How do the authors tag the inferred energy functions with semantic concepts? Manual inspection by the authors? In many cases, they do not seem intuitive — how did the authors determine that the first energy function is camera position? Looking at Figure 3, the third energy function seems to be focussing on blue-colored objects and the fourth energy function seems to be highlighting the complete image. Like other disentangled representation learning works such as beta-VAE, can they generate sets of examples where only this factor of variation is changing while everything else remains the same? Do they observe the camera position changing clearly? Same study for all the factors of variation that have been tagged with semantic concepts.
    - The proposed approach has only been compared against beta-VAE? What’s the reasoning behind this? Why not with MONET? Why not with other approaches mentioned in the related work as well as other methods that learn disentangled representations such as InfoGAN? Although [12] requires concept labels, it would still be informative to compare how energy functions perform in the presence vs absence of concept labels.
    - The shown beta-VAE reconstructions look odd. The reconstructions for CelebA-HQ seem too reddish. What implementation of beta-VAE is being used? 
    - Factor recombination can also be done with other models such as beta-VAE, right? How do the results shown in Figures 4 and 6 look for beta-VAE?
    - Figure 5: I don’t see any change by adding the “Lighting” energy function, what should be noticed here? Can the authors show the generations when these energy functions are added in the opposite order — lighting, then floor and then objects?
    - Table 1: what is the variance across different seeds? Also, please change the answer to 3(c) in the checklist to [No] since that’s clearly not true.
    - The resolution of images is so low that it’s hard to make out the shapes in CLEVR images and understand the semantics of KITTI images. Why is that? Can the resolution be increased to mitigate these problems?
    - How does the proposed approach compare to other methods in terms of computational time/resources?
    - What is the use of cross-dataset recombinations? What might be an example where this might be needed?
- Minor:
    - Since the code is currently not provided, the answer to 3(a) in the checklist should be [No].


------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
I thank the authors for their response to my questions/concerns and the additional qualitative and quantitative results. The response answers most of my concerns. Therefore, I still recommend acceptance for the paper. However, I still have some concerns that, when fixed, would make the paper much stronger in my opinion --

1. There is something wrong with Beta-VAE implementations used in the paper, resulting in weird qualitative results and perhaps affecting quantitative comparisons.
2. Many qualitative results are hard to understand and appreciate because of the low resolution of images.
3. The interpretability of the decomposition of natural images (Falcor3D, CelebA-HQ, KITTI) is a bit subjective.
limitations_and_societal_impact: Yes, the authors discuss their work’s limitations and societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Authors propose a model that decomposes images in terms of a set of 'components', which are energy terms whose minimisation gives rise to the observed pixels. Unlike previous works on compositional EBMs, the different components are learnt/discovered without supervision, just by learning to reconstruct a dataset of images. Results on various datasets show the method has significant potential, though there is considerable room for improvement. In particular, the method discovers concepts corresponding to different objects in multi-object scenes, and different attributes of faces.
main_review: Model & theory:

++ The model is novel and elegant, and a significant extension of earlier works on energy composition

++ The approach is clearly motivated and carefully described

++ The theoretical results are reassuring (although unsurprising)

-- It should be made clearer early on that there is no probabilistic interpretation attributed to the energy functions here (in contrast to typical EBMs)

-- The need for deciding a priori which component 'slots' will be local and which will be global is awkward, and somewhat limiting where this is unknown

-- 171: the analysis here does not seem so relevant -- surely what is important in practice is expressivity when the argmin is calculated only approximately

Experiments:

++ Decomposition & recombination results where components correspond to objects (CLEVR & Tetrominos) are qualitatively good

++ Quantitative disentanglement results are better than beta-VAE

-- Segmentation is measured using ARI, which is rather insensitive to certain failure cases. Mean segmentation covering should be given instead (as in most other works on object-centric modelling)

-- Disentanglement & recombination where components correspond to attributes (faces & KITTI) are not so good qualitatively

-- Reconstructions are not very high fidelity compared with other recent models

-- 299: "cross-dataset" seems rather a strong designation, given the similarity of the datasets in question

---
## Post-rebuttal

The authors have addressed my concerns sufficiently in their response, and also (in my opinion) those of the other reviewers. Therefore, I still favor acceptance, provided the promised evaluations etc. are added in the camera-ready.
limitations_and_societal_impact: Discussion is brief but adequate.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a novel method to composite the energies (statistical description of the images by the neural network). The author extracts the component energy by different latent codes for the image. The component latent code z controls the energy with the input image together in a conditional energy manner. The learned model can generate the image with the specified component codes inferred from the encoder network.
main_review: 1. Originality: The work is a novel combination of the well-know techniques, such as compositional energy and generation technique in “Compositional Visual Generation and Inference with Energy Based Models” and conditional generation technique in style transfer and the general conditioning layer.

2. Quality: The work is technically sound. I have several questions.

 (1). The author says at page2, line 74, “Our underlying energy optimization procedure is …Langevin sampling”, but in Eq. (2), the random term N(0, $\lambda$) of the Langevin dynamics is missed. Without the random diffusion, Eq. (2) degenerates to the gradient descent and may not guarantee the sampling x converging to the real distribution by the MCMC chain. Did author compare the performances of the inferring x between the Langevin dynamics and gradient descent?

  (2). The author claims to “capture independent factors without additional supervision” in the abstract, but from the main contribution of the paper, such as Eq. (1), (2) and Algorithm 1, I cannot find any guarantee for capturing the independent factors. The paper proposes method to compose the component energy, but does not design the method to learn the independent component factors from the encoder. Since all the component factors compete each other to explain the generated image, it is suggested to study the algorithm to guarantee each component factor is independent. As a consequence, the experimental results are less interpretable, that is, I need to guess the meaning of each component factor from the figures, such as Fig. 3, the experimental results about the interpretability can be further improved. 

  (3). The quality of the generated images in all the experiments are not sharp and can be improved in a large part. It is suggested to use and generate high resolution images, which is a mature technology nowadays.  

  (4). In the section 4 of Complexity Analysis, from the proof of remark 1, I can not find that the $\sum_z m_k(z)f(z)$ is equivalent to $\mathop{\arg\min}_x \sum_z m_k(z)(x-f(z))^2$, could you explain it?

3. Clarity: The paper is well written, but the results heavily depends on several techniques such as FILM, conditional EBMs, spatial attention, and the details are all missed. I know this is a conference paper and suffers space limitation, but the details is suggested to supply in the supplementary material. The current supplementary material still miss lots of details.

4. Significance: The result has referential significance. The other researchers could build on the idea to further develop other applications.  


limitations_and_societal_impact: Yes.
review_title: null

Meta-review: There is a reviewer consensus that the paper is just above the bar. I concur with that.
Decision: Accept (Poster)