Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper targets richer and higher-quality generation with VAE. Two techniques are adopted to achieve the goal: 1). bijective model to enrich data generation with flexible prior. 2). presenting compressed variants of the input data, i.e. self -supervision as additional condition $y$, for reconstruction. The two techniques interact through a hierarchical sampling process, $... y\sim p(y|u)\rightarrow z\sim p(z|u,y)$, thus benefits VAE generation with data-dependent prior and condition generation. 

The idea novel and reasonable. the paper is clearly presented. Here are some of my concerns.

1.  The author specifically argues the transformation $x \rightarrow y$ to be 'non-trainable', i.e. the mapping between x and y is deterministic.  BUT, will modeling $q(x|z,y)$ with discretized logistic distribution, affect the generation quality, since the likelihood is classicly assumed to be Gaussian distributed? 

2. The HIERARCHICAL SELF-SUPERVISED VAE is presented here to show the model can adopt multi-scaling information to benefit generation step-by-step. However, I  am afraid, in this way, the inference would be much difficult since the flow-based bijective operation is hard to train already. 

3. Is the conditional information, e.g. the sketches, also need in the test phase? or unconditional generational setting is adopted here.

4.  It seems the experiments are not conducted on High-quality datasets. To me, the presented results can not obviously demonstrate the achievements of the model. 

5. Can you please explain the connection between your self-supervised VAE to the general conditional VAE model in [1].

[1] Sohn, Kihyuk, Honglak Lee, and Xinchen Yan. "Learning structured output representation using deep conditional generative models." Advances in neural information processing systems. 2015.
review_title: Interesting work, but some details need to be further clarified.

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
## Summary

The paper presents a self-supervised variational auto-encoder called selfVAE. The work proposes the use of downscaling and edge detection as simpler representations of the input images to be reconstructed. The model should then learn to improve the low dimensional approximations to recover the higher dimensional ones in a hierarchical fashion. 

## Quality & Clarity

The paper is generally quite difficult to follow and the purpose, contributions and experiments are not presented clearly enough. The figures are not discussed in order, and the paper often references figures that are far away.

There are a number of grammatical errors in the paper.

## Outcome

The message of the paper generally was quite unclear and it could do with restructuring to assist readers.
review_title: Difficult to follow

final_score: rating: 4: Ok but not good enough - rejection
review_content: ###################################
Pros:

$\bullet$ VAEs can ignore some dimensions of the latent code. Enforcing the posterior distributions to consider desired factors of variations in the input can be fulfilled by either making it more structured (i.e., quantization as in VQ-VAE-2) or introducing additional constraints. This paper tackles this problem by applying the latter, two self-supervised tasks: edge maps and downscaled versions of inputs.

$\bullet$ The idea of adding self-supervised tasks to improve latent representation is very interesting. When learning a more structured latent representation, image superresolution, or sketch-to-image networks are also trained.

###################################
Cons:

$\bullet$ *Hierarchical self-supervised tasks* In section 3.4, multiple transformations explained. However, none of the experiments are conducted as a consecutive set of transformations. Does 3-level downscale mean a single downscaling three times or generating from $u$ using four different networks and match each of these levels with $z$? If yes, why does not the selfVAE-sketches model apply in a similar way hierarchically?  

$\bullet$ *How to train/balance operations* In Figure 3, there are several modes of operations given. How did you balance these modes during training?

$\bullet$ *Performance of self-supervised tasks:* What is the effect of self-supervised tasks' performance on the quality of latent representations? Considering the literature in image superresolution and sketch-to-image, did you use a pretrained auxiliary generator?

$\bullet$ *What is RE and KL in Table?* Are they the summation of both reconstruction ($RE_x, RE_y$) and KL divergence ($KL_z, KL_u$) terms in the loss (Eq.2)?  The reason why previous methods' RE/KL values were omitted should be stated. Similarly, why were the FID scores on CelebA and ImageNet-64 not given?   

Furthermore, the state-of-the-art FID scores on CIFAR-10 is better than the methods compared in Table 1. For instance, some examples of FID scores on CIFAR-10 are 18.9 in MoML [1], 29.3 in WP-GAN [2], 29.3 in spectrally normalized GAN [3], 26.4 in adversarial score matching [4], and so on.

[1] https://arxiv.org/pdf/1806.11006.pdf
[2] https://arxiv.org/pdf/1706.08500.pdf
[3] https://arxiv.org/pdf/1802.05957.pdf
[4] https://arxiv.org/pdf/2009.05475.pdf

As the results on CelebA and Imagenet-64 were not compared with previous literature, it is difficult to understand whether the contribution w.r.t. vanilla VAE is due to the self-supervised task or merely the use of an additional stochastic variable ($u$) and networks. 

Minor issue: "Imagenette64" might cause confusion, I did not see this dataset name before. I suppose that it is "ImageNet resized to 64x64" as in PixelCNN paper. References for all datasets should be added.

###################################
Reasons for score: 
Overall, I rate towards rejection. Even though the idea of bijective priors and doing this through self-supervised tasks is a novel approach, my major concern is that it is beyond the state-of-the-art in CIFAR-10, not compared to any other method on CelebA and ImageNet-64. Hopefully, the authors address my concerns above in the rebuttal period. 
review_title: This paper describes a framework that combines Variational Autoencoders (VAE) with self-supervised transformations by adding latent variables such as downscaling and edge detection. The main idea is to match the latent distribution of the original and transformed (downscaled or edge detection) data. Experimental results are done on Cifar-10, ImageNet-64, and CelebA datasets.

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper focuses on the task of generating high-quality data with generative models. To be specific, the authors proposed a variant of variational autoencoder (VAE) model, named self-supervised VAE. The intuition behind this model is that by breaking down the complex generation task into simpler/smaller ones, complex models can be trained steadily with the guidance from the simpler-level task. To his end, a hierarchical generative model with multiple-level latent variables is proposed, in which lower-level latent variables are governed by lower-level data features. The lower-level feature is generally obtained by a determined and discrete transformation, like down scaling. In addition, to further the modeling capability, a flow-based prior is proposed to fit the data distribution. Experiments were conducted to evaluate the performance of the proposed generative model.


Strength:
1. The idea of guiding the complex image generation with easer tasks is interesting, and is maybe the right way to accomplish complex tasks.

2. the ELOB directed in Eq.2 is intuitive and insightful. It also provides me theoretical support for the fact that employing two-level modeling and downscale transformation to generate a more vivid image is reasonable. 


Weakness:
1. From a technical perspective, the proposed method is just the combination of flow-based VAE and auxiliary VAE. By using 3 auxiliary variables, the authors infer one of them by a discrete and determined variational distribution q(y|x) to simplify the training objective, where the downscale image y plays an important role in this model. My question is why not regard y as observed data and then model the joint distribution p(x,y).

2. There are some mistakes in the derivation of Eq 2. In appendix A.4, during computing the entropy of q(w|x), the authors expresses it as E_{q(w|x)}[\log q(w|x)] = E_{q(z|y,x)}[\log q(z|y,x)] + ... . However, the first term in RHS is completely wrong. Actually, it should be E_{q(z|y,x)q(y|x)}[\log q(z|y,x)]. It seems that the authors use an equation in many places, that is E_{q(z|y,x)}[\log q(z|y,x)] = E_{q(z|y,x)q(u|y)q(y|x)}[\log q(z|y,x)]. But this equation is not ture, because the term q(z|y,x) insided the expectation is dependent on variable y. Besides, in the choice of distribution p(y|u) and p(x|z,y), they are set to be a mixture discrete logistic distribution. For each image x or y,  are their pixels assumed to be i.i.d ? If so, you miss \prod_{y_j \in y} outside the \sum_{i=1}^{I} in the distribution definition.

3. Bijective prior (RealNVP) is proposed in other works, and here simply employing it should not be regard as a contribution of this paper. Moreover, the authors only compare the effectiveness of different priors (i.e. Gaussian, mixture Gaussian, and RealNVP) on vanilla VAE and confirm the superiority of using an adaptive prior. However, I want to know what is the performance of self-supervised VAE if only using a standard Gaussian prior.

4. Section 3.3 is not presented well and the idea behind the sentences is hard to follow. What are the differences between these generation and reconstruction methods, and what application scenarios are corresponding to them? They are just simply listed, without providing any analysis of the logic behind them.

5. The experimental results cannot support the superiority of the proposed model in both of the quantity and quality comparisons. From the generated images, I cannot see too much difference between the SelfVAE and the vanilla VAE model, without to naming the more superior generative models, like GLOW, GANs etcs. Also, for the quantity comparison, the model is only compared with the outdated vanilla VAE in CelebA and Imagenet64, more recent generative models should be included here.


review_title: A interesting idea, but the results look not competitive

Meta-review: Reviewers appreciated the model and the ideas presented and found them very interesting.

The main reason for rejection is the extent of the empirical work.  Unfortunately, and I think what is a bad sign for the ICLR community, the authors could not do adequate empirical work due to their computational resources.  Not belonging to an organisation with extensive computational resources myself, I am in strong symparthy with the authors, though I do not see any way this can be satisfactorily accounted for in reviewing.  Several reviewers commented on the datasets, the extent of evaluations, and the comparisons made with prior work.  For instance, the small CIFAR10 images are not ideal to demonstrate the technique and comparative results with the other data sets are limited.

The reviewers had a number of concerns on the theoretical work and these were well discussed by the authors.

In summary, this is promising research but needs more empirical work.



Decision: Reject