Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes to combine energy functions to realize compositionality. This is interesting, and different from previous methods, which use either an explicit vector of factors that is input to a generator function, or object slots that are blended to form an image.
Specifically, three operators (logical conjunction, disjunction, and negation) are realized and empirically evaluated through combining energy functions in three different ways.
Extrapolating concept combinations, continually learning, and concept inference are also evaluated.

This paper is well motivated, showing compositional generation and inference for images. However, I have some concerns:

1. The experiments on the CelebA dataset are mainly subjective.

2. The equal sign in Eq.(4) should be \proto.

3. The most serious concern is that although empirical results are promising, I have concern about the correctness that Eq.(6) realizes disjunciton, and Eq.(8) realizes negation.

It is sensible that Eq.(4) realizes conjuction, according to the idea of Product-of-Expert. Multiplying several energy-based densities reduces to summation of the energies. 

For Eq.(6), the authors ignore the influence of normalizing constants when adding several energy-based densities. The authors seem to assume that the normalizing constants for p(x|c_i) are equal. Justification is needed.

Note that we cannot have :
log [0.6* exp(-E(x|c1)) + 0.4* exp(-E(x|c2)) ] will output c1 with probability 0.6 and c2 probability 0.4.

Fig. 1 seems to illustrate ideas at first sight, but is not so convinced at second thought.

4. No discussion for the \alpha in Eq.(8) for concept negation.

5. The description of the baseline joint model in Section 3.4 is missing.

6. For learning EBMs, the following reference is missed, besides (Kim & Bengio, 2016)
Yunfu Song, Zhijian Ou. Learning Neural Random Fields with Inclusive Auxiliary Generators. arxiv 1806.00271, 2018.

--------update after reading the response-----------
I appreciate the authors' response, but the paper still lacks in sound justification of assuming equal normalizing constants in concept disjunction and concept negation.
The claim that the partition functions are similar across dataset and models is mainly empirical (hardly hold in general). The authors' comment (under Figure 16 in A.6) on scaling the model by a suitable temperature to force histogram match in practice makes their reasoning further complicated.

Convincing quantitative experiments on disjunction and negation lacks. It would be better to focus on conjunction by EBMs, which already can makes a good paper, instead of claiming skeptical disjunction and negation by EBMs. A mixture of distribution is more natural to realize disjunction (Review #1 also comments on this).

Therefore, I tend to keep my original score.

Minor: in the updated paper, "The equal sign in Eq.(4) should be \proto" is not fixed.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The paper talks about training multiple energy-based models for each concept and then combining them in different ways to construct a composite model. For each composition of concepts, the samples from the composite energy-based model follow the logic of composition. 
The main novelty of the paper is to define an energy-based model for each logical composition of the concepts based on the energy-based model of each concept. Regarding the usage of energy-based models, the idea is interesting. However, I am not sure about the existing works using alternative approaches. 

My main question: the experiments only combine the energy models trained for the same domain. What does happen if you combine the energy models from different domains, for example, Celebs and color from the objects? What if you combine models with different architectures?
 
-- The qualitative images show the method is working, but the outputs are not great!!!! 

-- I really want to see the outputs for old + male + smiling + heavy hair in contrast to Figure 3?


-- The "concept inference is not clear from the text. The goal is to infer a concept from several given images of the same concept.
It is not clear from the text, but I assume you compute an approximate likelihood (by estimating the partition function with a single sample using Langevin dynamics) of images for each concept and pick the concept with the highest value.  
So what do you mean by this: "We can then obtain maximum a posteriori (MAP) estimates of concepts by minimizing the energy of the above expression." 

--I had a hard time following the experiments of sections 3.4 and 3.5, mostly because of unclear writing. 

--Some sentences are not clear, for example:
"To test this, we construct a sphere dataset consisting of sphere of all sizes at a specified percentage of the rightmost positions and large spheres remaining positions, with size/position annotations."

--Equations 4 and 10 need negation for the energy term to be consistent with Equation 1.
--The citation for Langevin dynamics should be Welling and Teh, 2011. 
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: Update: In light of the quality of the author response, I decide to raise my score to 6: Weak Accept. I encourage the authors to keep improving their writing of section 3.4. The description of the dataset is much clearer than the previous version, but I still think it is worthwhile to use more space for a better description.

There are some other issues with this paper that I recommend the authors to address in the revisions.

- Currently all quantitative experiments are based on conjunctions. It would be nice to have quantitative results for disjunctions and negations as well to make the authors' argument more convincing.

- I would suggest removing the result of PixelCNN in the paper. Of course sampling with Langevin dynamics for PixelCNN is hard. I think a more straightforward way is to do Langevin dynamics for each conditional distribution. The current result of PixelCNN is not done in this way and can be misleading for future work.

That being said, I think the idea proposed by this paper is of novel interest, and even though the execution is not ideal, I feel the novelty itself should warrant an acceptance at ICLR.


============ Original Review ===============

This paper proposes several rules to compose different energy-based models for compositional image generation. The authors propose empirical rules for conjunction, disjunction, and negation, which in combination can be used to derive arbitrary logic expressions. The authors demonstrate that using these composition rules, energy-based models can be used for compositional image generation, continual learning, and compositional concept inference.

I agree with the authors that the composition of energy-based models is a very desirable property and a very important direction to explore, and I believe that the work of this paper can have great potential in the future. However, I feel that the current paper is not very mature, and many questions are left unanswered. Here are some detailed comments:

- What's the justification for the formula of concept disjunction? For disjunction, it makes sense to me to form a mixture of distribution where each mixture has the same weight, ie, \sum_i p(x | c_i). However, \sum_i p(x | c_i) is not proportional to \sum e^{-E(x | c_i)} because of the unknown partition functions. Therefore, when the partition functions of two energy-based models differ by a lot, the disjunction will not be a mixture with equal weights for each component. Instead, it will collapse to a single distribution where all weights are put on the component with the smallest partition function. This does not match the intuition of "disjunction".

- One way of solving the disjunction problem might be abandoning the Langevin dynamics sampling procedure (7). Sampling from a mixture of distribution is doable for arbitrary generative models as long as it is possible to sample from each component of the mixture. Why not just apply Langevin dynamics to each energy-based model in the disjunction and return the samples from each model with the same probability? This procedure cannot be trivially composed with other rules such as conjunction and negation, but it suffices to produce all results in this paper and should be more theoretically sound.

- In Table 1, it seems that energy-based models are more robust in concept inference compared to a ResNet. Why is this the case? I would envisage that the energy-based model is also a ResNet and should have the same inductive bias? Is this a property of generative classifiers? Since the results in Table 1 do not involve composition of energies it is desirable to compare the results with other generative models as well, such as conditional PixelCNN.

- Since more steps of negative sampling can give better samples and generalize better, why not use 400 steps in all experiments throughout the paper?

- In the original OpenAI paper for energy-based models [1], the authors tune the coefficients of the gradient and noise terms in Langevin dynamics for better performance. This is equivalent to doing the ordinary Langevin dynamics for a different energy function re-scaled by some temperature parameter. Did you tune the scales of Langevin dynamics as well? If so, since the energies are scaled by some temperature, the original composition rules will be problematic. Did you take account of this temperature scaling in doing the experiments? 

- There is no detailed description of model architecture or hyperparameter in the paper. The author provide no detailed information on how the model was trained, for example, the various hyperparameters in Langevin dynamics and replay buffer. No code is available. There is even no appendix. The paper is very hard to reproduce, which hurts the reliability of the experimental results.

- Writing can be improved.  The description of the dataset in section 3.4 is barely readable. The second paragraph in section 3.4 needs to be paraphrased or significantly expanded. At least there should be some more detailed description of the settings in the appendix.

- At the end of the day, why energy-based models? I can imagine that an autoregressive model can be perfectly used for all tasks described in this paper. It does not even have the problem I mentioned in disjunction, since all densities are normalized. No Langevin dynamics sampling is needed. Since autoregressive models decompose the density using the chain rule, all compositions of the densities can also nicely decompose, and sequential sampling still works.

References:
[1] Du, Yilun, and Igor Mordatch. "Implicit generation and generalization in energy-based models." arXiv preprint arXiv:1903.08689 (2019).
review_title: Official Blind Review #1

Meta-review: This submission proposes an image generation technique for composing concepts by combining their associated distributions. 

Strengths:
-The approach is interesting and novel.

Weaknesses:
-Several reviewers were not convinced about the correctness of the formulations for negation and disjunction.
-The experimental validation of the disjunction and negation approaches is insufficient.
-The paper clarity and exposition could be improved. The authors addressed this in the discussion but concerns remain.

Given the weaknesses, AC shares R3’s recommendation to reject.
Decision: Reject