Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: Summary:
This paper claims to achieve disentanglement by encouraging an orthogonal latent space.

Decision: Reject. I found the paper difficult to read and the theoretical claims problematic. 

Issue 1: The Theorem
Can the authors explain how they got from Eq 5 to Eq 6? It seems that the authors claim that:
p(x | z1 z2 … zn) = p(x | z1) … p(x | zn) / p(x)**(n - 1)
I have difficulty understanding why this is true. It would suggest that
p(x | a b) = p(x | a) p(x | b) / p(x). 
Suppose a and b are fair coin flips and x = a XOR b. Then
p(x=1 | a=1 b=1) = 0
p(x=1 | a=1) = 0.5
p(x=1 | b=1) = 0.5
p(x=1) = 0.5
Can the authors please address this issue?

Even if Equation 8 is somehow correct, can the authors explain why BasisVAE provably maximizes the RHS expression in Eq 8? In particular the object p(x | z_i) is the integral of p(x, z_not_i | z_i) d z_not_i, which is quite non-trivial. 

Issue 2: The Model
The notation is a bit confusing, but it looks like the proposed model is basically a standard VAE, but where the last layer of the mean-encoder is an orthogonal matrix. I do not think the authors provided a sufficient justification for how this model relates back to Theorem 1. 

Furthermore, it is unclear to me why an orthogonal last-layer is of any significance theoretically. Suppose f is a highly expressive encoder. Let f(x) = M.T g(x) where g is itself a highly expressive neural network. Then M f(x) = g(x), which reduces to training a beta-VAE (if using Eq 12). From a theoretical standpoint, it is difficult to assess what last-layer orthogonality is really contributing.

Issue 3: The Experiments
Experimentally, the main question is whether the authors convincingly demonstrate that BasisVAE achieves better disentanglement (independent of whether BasisVAE is theoretically well-understood). 

The only experiment that explicitly compares BasisVAE with previous models is Table 3. What strikes me as curious about the table is the standard deviation results. They are surprisingly small. Did the authors do multiple runs for each model? Furthermore, the classification result is not equivalent to measuring disentanglement. There exists examples of perfectly entangled representation spaces can still achieve perfect performance on the classification task (any rotation applied to the space is enough to break disentanglement if disentanglement is defined as each dimension corresponding to a single factor of variation).
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: [updated rating due to supervision of $c_i$, which was not made clear enough and would require other baseline models]

This paper proposes a modification of the usual parameterization of the encoder in VAEs, to more allow representing an embedding $z$ through an explicit basis $M_B$, which will be pushed to be orthogonal (and hence could correspond to a fully factorised disentangled representation). It is however possible for different samples $x$ to use different dimensions in the basis if that is beneficial (i.e. x is mapped to $z = f(x) \cdot M_B$, where f(x) = (c_1, ... , c_n) which sums to 1.). This stretches the usual definition of what a “disentangled representation” means, as this disentanglement is usually assumed to be globally consistent, but this is a fair extension.
They show that this formulation can be expressed as a different ELBO which can be maximized as for usual VAEs.

I found this paper interesting, but I have one clarification that may modify my assessment quite strongly (hence I am tentatively putting it on the accept side). Some implementation details seem missing as well. Otherwise the presentation is fair, there are several results on different datasets which demonstrate the model's behaviour appropriately.

1.	The main question I have, which may be rather trivial, is “are the c_i supervised in any way?”. When I first read the paper, and looking at the losses in equations 9-11, I thought that this wasn’t the case (also considering this paper is about unsupervised representation learning), but some sentences and figures make this quite unclear:
	a.	In Section 3.2, you say “We train the encoder so that c_i = 1 and c_j = 0 if the input data has i-feature and no j-feature”. Do you?
	b.	How are the features in Figure 6 attached to each b_i? I.e. how was “5_o_clock_shadow” attached to that particular image at the top-left?
	If the c_i are supervised, this paper is about a completely different type of generative modeling than what it compares against (it would be more comparable to VQ-VAE or other nearest-neighbor conditional density models).
2.	There is not enough details about the architecture, hyperparameter and baselines in the current version of the paper.
	a.	What n_x (i.e. dimensionality of the basis) do you use? How does this affect the results?
	b.	How exactly are f(x), \Sigma_f(x) parametrized? They mention the architecture of the “encoder” in Section 4.1, but this could be much clearer.
	c.	How do you train M_B? I assume they are just a fixed set of embeddings that are back-propagated through?
	d.	What are the details about the architecture of the baselines, and their hyperparameters? E.g. what is the beta you used for Beta-VAE?
3.	The reconstructions seem only partially related to their target inputs (e.g. see Figure 4). This seems to indicate that instead of really reconstructing x, the model chooses to reconstruct “a close-by related \tilde{x}”, or even perhaps a b_i. This would make it behave closer to VQ-VAE, which explicitly does that. How related are reconstructions/samples to the b_i?
4.	Could you show the distribution of c_i that the model learns, and how much they vary for several example images?  How “peaky” is this distribution for a given image (this feeds into to the previous question as well)? The promise of the proposed model is that different images pick and choose different combinations of b_i, which hopefully one should see reflected in the distributions of c_i per sample, across clusters, or across the whole dataset.
5.	What happens when L_B is removed? I.e. what is the effect of removing the constraint on M_B being a basis, and instead allow it to be anything? This seems to make it closer to a continuous approximation to VQ-VAE?
6.	Is Equation 10 correct? Should the KL use N(f(x) \cdot M_B, \Sigma_f(x)), as in equation 9 above?
7.	Similarly, in Section 4.2.3, did you mean “c_i = 1 and c_j = 0 for i != j”?

If the model happens to be fully unsupervised, I think that these results are quite interesting, and provide a good modification to the usual VAE framework, I find that having access to the M_B basis explicitly could be very valuable.

There is still an interesting philosophical discussion to be had about when one would like to obtain a “global basis” for the latent space (i.e. Figure 3 (b)), or when one would prefer more local ones. I can see clear advantages for a non-local basis, in terms of generalisation and compositionality, which your choice (i.e. Figure 3 (c) ) would prohibit.

References:
[1] VQ-VAE: Aaron van den Oord, Oriol Vinyals, Koray Kavukcuoglu, “Neural Discrete Representation Learning”, https://arxiv.org/abs/1711.00937
review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: This paper proposes BasisVAE for acquiring a disentangled representation of VAE. 
Though the topic is much of interest for this conference, I cannot support its acceptance because the paper leaves many aspects unexplained in the model design. 

In particular, the following points need justified and clarified.
1) Theorem 1 is difficult to follow. 
The claim of the theorem is unclear. 
I suppose it says ELBO can be written as a sum with respect to z_i given p(z)=\prod_i p(z_i), but the statement is not clear enough from the text. 
Proof of Lemma 1 is logically incomplete. Discuss the cases n>2.
Derivation of equation (6) from (5) seems erroneous: p(x|z_1, ..., z_n) = \prod_{i=1}^n p(x|z_i) / p^{n-1}(x) does not hold in general even if z_i's are independent p(z_1, ..., z_n)=\prod_{i=1}^n p(z_i).

2) Connection between the objective function and Theorem 1 is unclear. 
BasisVAE uses a linear combination of Eqs. (9,10,11) as its objective function. 
How Theorem 1 motivates this formulation?

3) Reconstruction error (9). 
The text says \ell of Eq. (9) is the binary function and configured as in (Bojanowski et al. 2017). 
However, Bojanowski et al. used a weighted l1 error Laplacian Pyramid representation. 
Furthermore, the original VAE formulation uses a conditional log-likelihood log p(x|z) for the reconstruction term. 
How is binary function \ell related the likelihood?

4) KL regularization term (10).
For computing this term, the output of encoder c=f(x) should be converted into z. 
Notation of N(f(x), \Sigma) is confusing. 

5) Figure 6 shows diversity in many factors. 
Figure 6 is not as impressive for disentangled images since many factors change by varying a single basis. 
Is this an expected result?
review_title: Official Blind Review #3

Meta-review: The paper proposes a new way to learn a disentangled representation by embedding the latent representation z into an explicit learnt orthogonal basis M. While the paper proposes an interesting new approach to disentangling, the reviewers agreed that it would benefit from further work in order to be accepted. In particular, after an extensive discussion it was still not clear whether the assumptions of Theorem 1 applied to VAEs, and whether Theorem 1 was necessary at all. In terms of experimental results, the discussions revealed that the method used supervision during training, while the baselines in the paper are all unsupervised. The authors are encouraged to add supervised baselines in the next iteration of the manuscript. For these reasons I recommend rejection.
Decision: Reject