Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents a novel method for introducing inductive biases into variational auto-encoders (VAEs). The inductive biases are specified through a deterministic transformation in the latent space, which is shared between the encoder and the decoder. Through experiments on toy data, the authors demonstrate how to design appropriate inductive biases by matching the topological properties of the transformed representations to those of the real data distributions. They further quantify the performance of the model's ability to learn informative representations through experiments on MNIST, Fashion-MNIST, and CelebA, and show that it outperforms competitors both in terms of the generation quality with and without sparsity constraints as well as terms of downstream classification based on the representations.
main_review: Firstly, I'd like to thank the authors for an exceptionally well-written paper - it was a joy to read! The well-made figures and plots really help to show the benefits of the proposed model.

The problem of how to effectively introduce biases in VAEs is nicely introduced and discussed. In particular, I found the discussion on the shortcomings of prior modification enlightening and convincing. The proposed model is both simple and intuitive, and I like the connection to normalising flows that the authors make a couple of times, since it gives a nice, mental picture. I would have liked, however, to see some of the equations relating to the inductive biases from the supplementary in the main paper, for instance equations (C.3) through (C.5). I think they are quite important for understanding how the model would work in practice.

I am also not entirely convinced that the transformation should be seen as being shared between the encoder and decoder. I understand why the authors would like to push this view, since the introduced inductive bias, by definition, should improve the learnt representations, which may then be used for downstream tasks. However, my understanding of the model is that the transformation is simply the first layer of the decoder, which then happens to be handcrafted to introduce the appropriate inductive bias. I don't think this view reduces the significance of the proposed model; on the contrary, I find it cleaner to separate the encoder and the decoder this way. (Indeed, some VAE fundamentalists would argue that only the decoder is the model, so decoupling the encoder and the decoder would also please them :) ).
In this light, I don't think theorem 1 is necessary to have in the main paper and, personally, I find the discussion of how to construct inductive biases for different problems both more interesting and practically useful. Of course, if I have misunderstood the model I'm happy to be corrected by the authors.

The experiments are well-thought-out and show increasingly complex aspects of the model - from learning to represent toy data to learning sparse representations of CelebA images. I like the progression of the section, even though the supplementary material is needed to make it completely reproducible.
The point that the authors make that only the topological properties of the data distribution need to be matched by the inductive bias is particularly fascinating. I would very much like to see this discussed further somewhere, though it may be outside the scope of the paper.

I didn't find any technical or mathematical issues in the paper, but there aren't many equations either. This isn't meant as a criticism - I like the simplicity of the model, and it highlights how little is needed to improve the decoder. In that sense, it's a nice reminder of how ineffective neural networks can be.


**Questions**

In the discussion of the links to normalising flows, you mention that a normalising flow would be an unlikely choice for the transformation. I assume that you are referring to the architectural constraints and computational costs of normalising flows, but are there other reasons for not using a normalising flow as the transformation?


**Typos**

* Page 5, fourth paragraph from below: "in start contrast" -> "in stark contrast"
* Page 9, first paragraph from above: "DD significantly degrades generation ~generation~ quality"



summary_of_the_review: The proposed model is clearly of interest to the ICLR community. Both the discussion of the issues with modifying the prior and the simple and effective solution of introducing a latent transformation should provoke some interesting discussions and further work. The paper is exceptionally well-written and presented, and while it would be a stretch to call it a groundbreaking paper, it is a solid "accept".
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose to introduce some transformation of the Gaussian latent variable of VAEs in order to incorporate specific inductive biases on the semantics of the latent representation. They showcase several examples of such transformation and empirically examine the performance with comparison to baseline methods.
main_review: The technique is reasonable, and the experiments are with a decent comparison with baseline methods. Overall, I think this is solid work. That being said, some reasoning around the proposed framework sounds awkward, due to which I think the technical contribution of the paper is not clearly presented.

My concern is on the novelty of the abstract proposed framework, InteL-VAE. Basically, the transformation of the Gaussian latent variable, $g_\psi$, is just a part of a specific design of the decoder. The authors emphasize that $g_\psi$ is "shared" by both decoder and encoder, and I do understand the intention of such a stance (i.e., the output of $g_\psi$ can be regarded as a transformed latent variable). However, I do not think it makes much sense because in a technical point of view, $g_\psi$ is nothing more than a part of the decoder. Designing a decoder based on some prior knowledge of data is a common practice, and thus the abstract framework of the proposed method can hardly claim significant technical novelty.

I think the technical contribution of the paper rather lies in the particular definitions of $g_\psi$, some of which are presented in the main text, and more are in the appendix. For example, the multi-modality case in section C.2 is very interesting (yet looks ad-hoc). So, I would suggest pivoting the main claim of the paper on these particular instances of $g_\psi$, rather than on the abstract framework of InteL-VAE. For example, it would be great if the multi-modality $g_\psi$ is enough discussed in the main text, rather than being deferred to the appendix.
summary_of_the_review: While I think this is solid work, the current presentation is not necessarily kind for readers to understand the technical contribution specific to this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to incorporate some inductive biases into VAEs by modifying decoders. The idea is to first transform stochastic latent variables using some specific mapping and then apply a deep neural network. The expected result is that the encoder will be forced to assign z’s in such a way that the mapping can turn them into a manifold of a given form.
main_review: Strengths:
- The idea of modifying decoders is interesting.
- The presented mappings seem to be indeed helpful in most of the cases presented in the experimental section.

Weaknesses:
- I do not agree with the statement that the prior in the VAE framework is about regularizing. First, the VAE is not a Bayesian model (i.e., we still optimize the likelihood function, not the marginal likelihood). Second, the prior (or rather the marginal distribution over z’s) could be trained and then it serves a different purpose, namely, fitting a model to match the aggregated posterior distribution. Therefore, I find the paragraphs on the first page not fully convincing.
- From the presentation perspective, it is rather a peculiar choice to refer to a figure (Fig. 3) that could be found on page 4. It breaks the flow of reading the paper.
- It is unclear to me why the deterministic variable z is obtained by taking the mean value of y, namely, $z = g_{\psi}(\mu_{\phi}(x))$ (page 4). Shouldn’t it be $z = g_{\psi}(y)$? Otherwise, what’s the point in having a stochastic y?
- Following the previous point, in Eq. 2, it is mentioned that $g_{\psi}(y)$ is used. I start doubting whether the ELBO is correctly calculated.
- I do not see how the proposed idea is different from having a more sophisticated decoder in which we focus on its specific part (i.e., first layers). In other words, I do not see why the proposed analysis in Sect. 4.1 and 4.2 cannot be applied to any decoder.
- The whole paper is written in a pretty confusing manner. One can have an impression that z’s are separate random variables (the authors constantly repeat that the resulting distribution over z’s after applying a mapping g is, e.g., approximately spherical) but they are not used anywhere in the objective function explicitly. I fully understand that it is interesting to see how the inductive biases could be used and that the resulting distributions are of a specific form. However, it does not help a reader to follow the paper. Especially that the stochastic latent variables are y and they are used in the ELBO.
- I tried to go through the code but after spending 30min I gave up. The code is not necessarily helpful to figure out some missing parts of the paper.
- I think the paper presents some interesting ideas, however, it is lost by not necessarily a good way of presenting it.

summary_of_the_review: The paper shows that modifying a decoder in a VAE could be helpful and allows to still use a simple marginal distribution over z’s (e.g., a standard Gaussian distribution). However, the presentation of the paper is somehow misleading in many places that making it difficult to follow the flow of thoughts. Therefore, I believe the paper is ready to be accepted.

=== UPDATE ===
I would like to thank the authors for their rebuttal. Indeed, some of my concerns were fully addressed and the paper looks better now. However, I am still not fully convinced how to properly choose the transformation $g$ and what is the recipe for that. Anyway, I decided to increase my score to 6.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work introduces a new method to embed used-designed inductive-biases into VAE architectures. the basic idea is straightforward: a fixed user-defined deterministic transformation is used to map the latent z into a "structured latent" y. The transformation is used to induces topological and/or sparsity constraints. Technically, this is identical to using the transformation as the first layer of the decoder architecture. Besides of the general idea, the authors introduce a variety of simple but clever transformations used to enforce latent constraints. The approach is shown to offer an interpretable latent space and attractive performance in a large class of density estimation tasks.
main_review: Strengths:
- Incorporating human knowledge into deep learning architectures is becoming a very relevant research area as the ML community is integrating with science and engineering. The paper offers a relevant contribution to this booming field of research. 
- The paper is very well written and it contains sound and convincing arguments in favor of using user designed inductive biases and against the use of the VAE prior in order to induce those constraints. 
- While the general approach is not very novel, the specific layers used to induce topological constraints are innovative, clever and useful. In my opinion the main contribution comes from these layers rather than the general technique. 

Weaknesses:
- From a technical point of view, the main contribution is hardly novel as it simply entails the use of a user designed fist encoder layer. However, the approach is interesting from a conceptual and  interpretability standpoint.
- The sparsity application is hardly novel as it is nothing more than a simple soft attention layer. Soft attention is already widely used in the generative modeling literature. 

Suggestions:
- I would recoment the authors to de-emphasize the general approach and to leave more space for the details of the topological layers used as in my opinion they are the main innovation of this work. 
summary_of_the_review: The paper is well written and convincing. The new framework is not novel from a technical point of view but the details of the layers introduced to model complex topologies are. There is a major need of methods for embedding human knowledge into deep learning architectures and this paper is on a good direction.
review_title: null

Meta-review: I am recommending a poster for this paper.  There was considerable discussion and much author response.  The reviews were good (after taking author response and paper revision into account) with one out of three being enthusiastic. There was a concern that the basic idea was technically mis-represented as the inductive bias is being placed in the decoder rather than prior. But I am convinced that it is a reasonable idea to place bias in the decoder and that idea is worth publication.

Personally I think the paper would be much stronger with better empirical evaluation.  I find a focus on MNIST (or fashion MNIST) unconvincing. Results on CelebA should be accompanied by sample image generations.  I would rather see downstream task metrics based on learned features.  This paper cannot be put in the same class as recent results on unsupervised learning of image features for downstream tasks. It remains an open question as to whether this paper provides any contribution in that arena.
Decision: Accept (Poster)