Self-Supervised Variational Auto-Encoders
Authors: Ioannis Gatopoulos
Jakub Mikolaj Tomczak
Venue:  (2021)
Keywords: generative modeling
deep learning
deep autoencoders

Abstract: Density estimation, compression, and data generation are crucial tasks in artificial intelligence. Variational Auto-Encoders (VAEs) constitute a single framework to achieve these goals. Here, we present a novel class of generative models, called self-supervised Variational Auto-Encoder (selfVAE), that utilizes deterministic and discrete transformations of data. This class of models allows performing both conditional and unconditional sampling while simplifying the objective function. First, we use a single self-supervised transformation as a latent variable, where a transformation is either downscaling or edge detection. Next, we consider a hierarchical architecture, i.e., multiple transformations, and we show its benefits compared to the VAE. The flexibility of selfVAE in data reconstruction finds a particularly interesting use case in data compression tasks, where we can trade-off memory for better data quality, and vice-versa. We present the performance of our approach on three benchmark image data (Cifar10, Imagenette64, and CelebA).

Under review as a conference paper at ICLR 2021
SELF-SUPERVISED VARIATIONAL AUTO-ENCODERS
Anonymous authors
Paper under double-blind review
ABSTRACT
Density estimation, compression, and data generation are crucial tasks in artificial intelligence. Variational Auto-Encodert (VAEs) constitute single framework achieve these goals. Here, we present a novel class of generative models, called self-supervised Variational Auto-Encoder (selfVAE), that utilizes deterministic and discrete transformations of data. This class of models allows performing both conditional and unconditional sampling while simplifying the objective function. First, we use single self-supervised transformation a transformation is either downscaling or edge detection. Next, we consider a hierarchical architecture, i.e., multiple transformations, and we show its benefits compared to the VAE. The flexibility of selfVAE data reconstruction finds a trade-off
latent variable, where
particularly interesting use case in data compression tasks, where memory for better data quality, and vice-versa. We present the performance our approach on three benchmark image data (Cifar10, Imagenette664, and CelebA).
INTRODUCTION
The framework of variational autoencoders (VAEs) provides a principled approach for learning latent- variable models. As utilizes meaningful low-dimensional latent space with density estimation capabilities. it forms an attractive solution for generative modelling tasks. However, its performance in terms of the test log- likelioodood and quality of generated samples is often disappointing, thus, many modifications were proposed. In general, one can obtain a tighter lower bound, and, thus, more powerful and flexible model, by advancing over the following three components: the encoder (Rezende et al., 4; van den Berg et al., 201 8; Hoogeboom et al., 2020; Maaløe et al., 2016), the prior (or marginal over latents) (Chen et al., 2016; Habibian et al., 2019; Lavda et al., 2020; Lin & Clark, 2020; Tomczak & Welling, 2017) and the decoder (Gulrajani et al., 2016). Recent studies have shown that by employing deep hierarchical architectures and by carefully designing building blocks of the neural networks, VAEs successfully model high -igeesiinnii data and reach state-of-the-art test likelihoods (Zhao et al., 2017; Maalge et al., 2019; Vahdat & Kautz, 2020).
In this work, we present novel class of VAEs, called self-supervised Variational Auto-Encoders, where we introduce additional variables to VAES that result from discrete and deterministic transfor- mations of observed images. Since the transformations are deterministic, and they provide a specific aspect of images (e.g., contextual information through detecting edges or downscaling), we refer to them as self-supervised representations. The introduction of the discrete and deterministic variables allows to train deep hierarchical models efficiently by decomposing the task of learning highly complex distribution into training smaller and conditional distributions. In this way, the model allows to integrate the prior knowledge about the data, but still enables to synthesize unconditional samples. Furthermore, the discrete and deterministic variables could be used to conditionally reconstruct data, which could be of great use in data compression and super-resolution tasks.
We make the following contributions: i) porating self-supervised representations of the data. ii) We analyze the impact of modelling natural images with different data transformations as self-supervised representations. iii) This new type of generative model (self-supervised Variational Auto- Encoders)),, which is able perform both conditional and unconditional sampling, demonstrate improved quantitative performance in terms of density estimation and generative capabilities image benchmarks.
We propose an extension of the VAE framework by incor-

---

Under review as a conference paper at ICLR 2021
2 BACKGROUND
2.1 VARIATIONAL AUTO-ENCODERS
X .D be vector of observable variables, where X' C RR or X' Z, and E RM denote vector of latent variables. Since calculating px( = (x, z)dz is computationally intractable for non-linear stochastic dependencies, variational family of distributions could be used for approximate inference. Then, the following objective function could be derived, namely, the evidence lower bound (ELBO) (Jordan al., 1999):
In (x)) 2 (x(()) po 2) + px) go(x)x)]
where g (z(x) is the variational posterior (or the encoder), p=(x)z) is the conditional likelihood function (or the decoder) and (2( is the prior (or marginal), @, and À denote parameters.
The expectation is approximated by Monte Carlo sampling while exploiting the reparameterization trick in order to obtain unbiased gradient estimators. The models are parameterized by neural networks. This generative framework is known as Variational Auto-Encoder (VAB) (Kingma & Welling, 2013; Rezende et al., 2014).
2.2 VAES WITH BLECTIVE PRIORS
Even though the lower-bound suggests that the prior plays a crucial role in improving the variational bounds, usually a fixed distribution is used, e.g., a standard multivariate Gaussian. While being relatively simple and computationally cheap, the fixed prior known result in over-regularized models that tend to ignore most of the latent dimensions (Burda et al., 2015; Hoffman & Johnson, 2016; Tomezak & Welling, 2017). Moreover, even with powerful encoders, VAFs may still fail to match the variational posterior to unit Gaussian prior (Rosca al., 2018).
However, is possible to obtain a rich, multi-modal prior distribution p(z) by using a bijective (or flow-based) model (Dinh et al., 2016). Formally, given a latent code z, a base distribution pv over latent variables v E RM and f R M - M consisting of a sequence of L diffeomorphic transformations! where (v;-1) Vs, sequentially to express the distribution of z as a function as follows:
and vL z, the change of variable can be used
(
log log Σ log
(2)
~v,-1
where
@fi(v=-1) is the ov.-1
of the ith transformation.
Thus, using the bijective prior yields the following fower-bounds
0f (v.)
J.
Inp(x) 2 Bystz|x) log pa (x(z) - log qa (x)x) + log +
log
(3)
Ov;
In this work, utilize RealNVP (Dinh al., 2016) the prior, however, any other flow-based model could be used (Kingma & Dhariwal, 2018; Hoogeboom et al., 2020). For the experiments and ablation study that shows the impact of the bijective prior VAEs, we refer to the appendix A.
METHOD
3.1 MOTIVATION
The idea of sulf-super learning is about utilizing original unlabeled data to create additional context information. It could be achieved in multiple manners, e.g., by adding noise data (Vincent et al., 2008) or masking data during training (Zhang et al., 2017). Self-supervised learning could also be seen as turning an unsupervised model into a supervised by, e.g., treating predicting next pixels as classification task (Hénaff et al., 2019; Oord et al., 2018) 8). These are only a few examples of quickly growing research line (Liu et al., 2020).
That is, invertible and differentiable transformations.
2

---

Under review as a conference paper at ICLR 2021
Here, we propose to non-trainable transformations to obtain information about image data. Our main hypothesis is that since working with highly-quali images is challenging, we could alleviate this problem by additionally considering partial information about them. Fitting model to images of lower quality, and then enhancing them match the target distribu- tion seems to be overall an easier task (Chang et al., 2004; Gatopoulos et al., 2020). By incorporating compressed transformations (i.e., the self-supervised representations) that still contain global information, with the premise that it would be easier to approxi- mate, the process of modelling a nigh-dimensional complex density breaks down into simpler tasks. In this way, the expressivity of the model will grow and gradually result into richer, better generations. A positive effect of the proposed framework is that the model allows to integrate prior knowledge through the image transformations, without losing its uncon- ditional generative functionality. Overall, we end up with two-level VAE with three latent variables, where one is data transformation that can be obtained in a self-supervised fashion. In Figure schematic representation the proposed approach with downscaling is presented.
Figure 1: The proposed approach.
A number of exemplary image transformations are presented in Figure 2. We notice that with these
transformations, even though they discard a lot of information, the global structure is preserved. As result, in practice the model should have the ability to extract general concept of the data, and add local information afterwards. In this work, we focus on downscaling (Figure 2.b, e & d) and edge detection sketching (Fig. 2.1).
Oviginal Image
Inte patepolations
<) Bieabic Interpolation x3
Inteppoltion
Gaussian Kennel (Ix])
2
1o
Greyscale
Figure 2: Image Transformations. All of these transformations still preserve the global structure of the samples but they disregard the high resolution details different ways.
3.2 MODEL FORMULATION
In our model, consider representations that result from deterministic and discrete transformations of an image. Formally, we introduce transformation d x' + X that takes x and returns an image representation y, e.g., e.g.,, downscaled image. Since lose information about the original image, z could be seen as variable that compensates lost details in x. Further we propose to introduce an additional latent variable, u E N model y and z. We can define the joint distribution of and y as follows: p(x. y) ply xx), where p(y x) 8(y - d(x)) due to the deterministic transformation d(-), where 8(-) the Kronecker delta. Thus, the empirical distribution is 8(y d(x)) (x).. However, since we are interested in decomposing the problem of modeling a complex distribution
p(x), we propose to model p(x(y)n (yy) instead, and utilize the variational inference of the form Quu, 2|x, y) = qully (x(2)x) that yields:
In y) 2 EE po )) In Inp(z/u, (np(ylu) + Inp(u) In gz//2)) In (ng(())) (4)
3

---

Under review as a conference paper at ICLR 2021
Intuitively, the premise for selfVAE is that the latents will capture the global structure of the input data and the latents z will encode the missing information between y and x, guiding the model to discover the distribution of the target observations. In order to highlight the self-supervised part in our model, we refer to it as the self-supervised Variational Auto-Encoder (or selfVAE for short).
Further, propose choose the following distributions:
p(v) = N. 1v)
G (uly) = N (u) (y), diag (a (y)))
PA (u) = p(v) II det (f((v:-1)
qo. (z|x) = N He (x), diag (a (x)))
vv
po, (y/u)= Σ
(1) Dlogistic (44", ;88 (11)
p8, (zly, N (z) (y, u), diag (o (y, u))
pes (x|z,y)
(zy) (a Dlogistic (u ( s (ay))
(wx),
where Dlogistic is defined the discretized logistic distribution (Salimans al., 2017), and utilize a flow-based model for px (u). Notice that we use the discretized logistic distribution, because images are represented by values between 0 and 255. For integer-valued random variables, other distributions like Gaussian are inappropriate.
3.3 GENERATION AND RECONSTRUCTION IN SELFVAE
As generative models, VAEs can be used to synthesize novel content through the following process: p(z) + zx) but also to reconstruct a data sample x" by using the following scheme: z (((xx²) + - pxx/z).
Interestingly, our approach allows to utilize more operations regarding data generation and recon- struction. First, analogously to VAEs, the selfVAE allows to generate data by applying the following hierarchical sampling process (generation):
(u) + ~ p(ylu) u) - 2 y( + x p(x 2).
However, we can use the ground y (i.e, y d(x') )) and sample or infer Then, the generative process for the former (conditional generation) is:
q(((x")) p(xly", z).
and for the latter (conditional reconstruction):
u~ q(uly"())))))) + 2 p(zuu x ~ p(xly*,,))
If y is downscaling transformation of the input image, Sel can be used in manner similar to the super-resolution (Gatopoulos et al., 2020). Alternatively, we can sample (or generate) y instead, and choose sample infer In this way, reconstruct an image in ways, namely, reconstruction I:
y" =d=((x") -(x"))+~~> and reconstruction 2:
+ p(ylu) p(zlu,y) x p(x|z,y),
(y* u + p(x|u))( then z ~ g(z[x") +x p(x|y, 2).
The presented versions of generating and reconstructing images could be useful in the compression task. As we will see in the experiments, each option creates different ratio of the reconstruction quality against the memory that we need to allocate to send information. However, every inferred variable needs to be sent, thus, more sampling corresponds to lower memory requirements.
4

---

Under review as a conference paper at ICLR 2021
3.4 HIERARCHICAL SELE-SUPERVISED VAE
The proposed approach can be further extended and generalized by introducing multiple transformations, in the way that is illustrated in Figure 3. By incor- porating single (or multiple) self-supervised repre- sentation(s) of the data, the process of modelling a high-dimensional complex density breaks down K simpler modeling tasks. Thus, we obtain K K-leeel VAE architecture, where the overall expressivity of the model grows even further and gradually results into generations higher quality. Some transfor- mations cannot be applied multiple times (e.g., edge detection), however, others could be used sequentially, e-g., downscaling.
u
7
7
7
y
We take K self-supervised data transformations di(-) that give K representations denoted by YI:K ",....,yk]] and the following variational distribu- tions:
Generative Model
Inference Model
Figure 3: Hierarchical selfVAE.
K-1
Quu, z/x,
II a
(5)
that yields the following objective:
In (x.k) 2 (n(Inpo(x()1:)) +
(Inp(z)lys 2x+1) + Inp(yxlye+1,
+ (np(zk/u,yk) Imp(yx/u) Inp(u) - ing(u)y)) 1nn--mgg11 ln((z(())
Σ In Ing(2++1y0)) (6)
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets We evaluate the proposed model CIFAR-10, Imagenette64 and CelebA:
CIFAR-1O The CIFAR-10 dataset a well-known image benchmark data containing 60.000 training examples and 10.000 validation examples. From the training data, put aside 15% randomly selected images as the test set. We augment the training data by using random horizontal flips and random affine transformations and normalize the data uniformly in the range (0, 1).
Imagenette64 Imagenette64 is a subset 10 classes from the downscaled Imagenet dataset. We downscaled the dataset to 64px x 64px images. Similarly to CIFAR-10, put aside 15% randomly selected training images as the test set. We used the same data augmentation as in CIFAR-10
CelebA The Large-scale CelebFaces Attributes (CelebA) Dataset consists of 202.599 images of celebrities. We cropped original images on the 40 vertical and 15 horizontal component of the top left corner of the crop box, which height and width were cropped 148. Besides the uniform normalization of the image, no other augmentation was applied.
Architectures Encoders and decoders consist of building blocks composed of DenseNets (Huang et al., 2016), channel-wise attention (Zhang et al., 201 8), and ELUs (Clevert et al., 2015) as activation functions. The dimensionality of all the latent variables were kept at 8 x 8 x 16 1024 and all models were trained using AdaMax (Kingma & Ba, 2014) with data-dependent initialization (Salimans Kingma, 2016). Regarding the selíVAEs, CIFAR-10 10 we used an architecture with single downscaled transformation (selfVAE-downscale)) while the remaining two datasets (CelebA

---

Under review as a conference paper at ICLR 2021
and Imagenette64) used hierarchical 3-leveled selfVAE with downscaling, and selfVAE with sketching. All models were employed with the bijective prior (RealNVP) comparable in terms of the number of parameters (the range of the weights of all models was from 32M to 42M). For more details, please refer to the appendix section A.2.
Evaluation We approximate the negative log- oigllikelhoood using 512 IW-samples (Burda et al., 2015) and express the scores in bits per dimension (bpd). Additionally, for CIFAR-I0, we use the Fréchet Inception Distance (FID) (Heusel et al., 2017).
Table I: Quantitative comparison on test sets from CIFAR-IO, CelebA, and Imagenette64. *Measured on training set.
Dataset
Model
bpd 4
FID |
PixeICNN (van den Oord et al., 2016)
3.14
65.93
GLOW (Kingma & Dhariwal, 2018)
3.35
65.93
ResidualFlow (Chen et al., 2019)
3.28
46.37
CIFAR-10
BIVA (Maaløe al., 2019)
3.08
NVAE (Vahdat & Kautz, 2020)
2.91
DDPM (Ho al., 2020)
3.75
5.24 (3.17*)
VAE (ours)
3.51
41.36 (37.25*)
seliVAE-downscale
3.65
34.71 (29.95*)
RealNVP (Dinh et al., 2016)
3.02
VAE (ours)
3.12
CelebA
sellVAE-sketch
3.24
selfVAAE-oonnscaee-3WW
2.88
VAE (ours)
3.85
Imagenette64
sel/VAE-downnccle--llv
3.70
4.2 QUANTITATIVE RESULTS
We present the results of the experiments on the benchmark datasets in Table 1. First, we notice that on CIFAR-10 10 our implementation of the VAE is still lacking behind other generative models in terms of bpd, however, is better or comparable in terms of FID. The seloVAA-downscale achieves worse bpd than the VAE. A possible explanation may lie in the small image size (32 x 32), as the benefits of breaking down the learning process two or more steps are not obvious given the small target dimensional space. Nevertheless, the than the other generative models. This result could follow from the fact that downscaling allows to maintain context information about the original image and, result, general coherence is of higher quality.
achieves significantly better FID scores
Interestingly, the two other datasets, three-level sel/VAE-downscale achieves significantly better bpd scores than the VAE with the bijective prior. This indicates the benefit of employing a multi- leveled self- supervised framework against the VAE in figher-dimensionn fails to scale efficiently. It seems that the hierarchical structure of self-supervised random variables allows encode the missing information all information about images must be coded in z. This result is promising and indicates that the proposed approach is of great potential for generative modelling.
data, where the plain model
efficiently in 2g, in contrast the vanilla VAE, where
4.3 QUALITATIVE RESULTS
We present generations on CIFAR-10 10 and Imagenette64 in Figure 4 and on CelebA in Figure 5, and reconstructions CIFAR-I1 10 and CelebA in Figure 6.

---

Under review as a conference paper at ICLR 2021
selEVAE downscale
VAR
Imagenete64
CIFAR- 10
First level
Second level
Third level
Figure 4: Uncoditional generations on Imagenette64 and CIFAR-IO.
i) selfVAE downscale. 31vl
ii seliVAE sketch
M VAE RealNVP
Figure 5: Unconditional CelebA generations from (i) the three-level self sulfvvupe VAE with downscaling, (ii) the self-supervised VAE employed with edge detection (sketches), (iii) the VAE with RealNVP prior.
7

---

Under review as a conference paper at ICLR 2021
VAE
seliVAE-downscule-2/
Sarple
Recetobaciii
Recuninction
Recosbraction
Conditioaal
Genontion
Clnditioo Recorsmotie
SeltSepercaal
Regmmettaioon
Sell-Seperreed Recoramction
CIFAR-I0 10
VAE
seIVVAE-dowsscal--VV
scl/VAE-sketch
Duta Sample
Reomsimation
Receesraationn
Recossmaction:
Resonamatioe
Cancitonal Recestractm
Recrobrrctan
Reorsimation
CelebA
Compression 8%
16%
25%
33%
16%
ratio:
Figure Comparison on image reconstructions with different amount of sent information.
We first notice that the generations from selfVAE seem be coherent, in contrast with these from VAE that produces overall more contextless and distorted generations. This result seems to be in line with the FID scores. Especially for CelebA, we observe impressive synthesis quality, great sampling diversity and coherent generations (Figure 5). On the Imagenette64 dataset, we can also observe crisper generations for our method compared the VAE (Figure 4).
Furthermore, the hierarchical selfVAE seems to be of a great potential for compression purposes. In contrast the VAE, which is restricted to using a single way of reconstructing an image, the selfVAE allows four various options with different quality/memory ratios (Figure 6). In the selfVAEEsketch we can retrieve the image with high accuracy by using only 16% of the original data, as manages to encode all the texture of the image to (Figure 11). This shows the advantage of choosing prior knowledge into the learning process. Lastly, the latents learn to add extra information, which defines the end result, and we can alter details of an image like facial expressions (Figure 12.ii).
CONCLUSION
In this paper, we showed that taking deterministic and discrete transformations results in coherent generations of high visual quality, and allows to integrate prior knowledge without loosing its unconditional generative functionality. The experimental results seem to confirm that hierarchical architectures perform better and allow to obtain both better bpd scores and better generations and reconstructions. In the experiments, considered two classes of image transformations, namely, downscaling and edge detection (sketching). However, there a vast of possible other transformations (see Figure 2), and we leave investigating them for future work. Moreover, approach interesting for the compression task. A similar approach with multi-scale auto-encoder for image compression was proposed, e.g, by Mentzer al. (2019) Razavi al. (2019). However, we still use probabilistic framework and indicate that various non con-trainable image transformations (not only multiple scales) could be of great potential.
find the proposed
ACKNOWLEDDMENTS
Anonymized for the double-blind review.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv, 2015.
Hong Chang, Dit-Yan Yeung, and Yimin Xiong. Super-resolution through neighbor embedding. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 1, pp. I-1. IEEE, 2004.
Ricky I. Q. Chen, Jens Behrmann, David Duvenaud, and invertible generative modeling. arXiv, 2019.
Jacobsen. Residual flows for
Xi Chen, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. arXiv, 2016.
Djork-Aréé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv, 2015.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv, 2016.
Eduardo S. L. Gastal and Manuel M. Oliveira. Domain transform for edge-aware image and video processing. ACM TOG, 30(4))69: 19:))))9:166:1111 2011. Proceedings of SIGGRAPH 2011.
Ioannis Gatopoulos, Maarten Stol, and Jakub M. Tomezak. Super-resolution variational auto-encoders. 1rXiv.0006.02288, 2020.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference Learning Representations, 2019. URL https: http://ppeeeeeeeeeeeeee
http.ttttt/////oeeeeeeeeeee net/ feret/forumt forum? SeBygg9j0999
Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vazquez, and Aaron Courville. Pixelvae: A latent variable model for natural images. arXiv, 2016.
Amirhossein Habibian, Ties van Rozendaal, Jakub M Tomezak, and Taco S Cohen. Video compression with rate-distortion autoencoders. Proceedings of the IEEE International Conference on Computer Vision, pp. 7033-7042, 2019.
Olivier Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding, arXiv preprint UrXiv:1905.09277 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. arXiv, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in
Neural Information Processing Systems, 33, 2020.
Matthew D Hoffman and Matthew J Johnson. Elbo surgery: yet another way to carve the variational evidence lower bound. In Workshop Advances in Approximate Bayesian Inference, NIPS, volume 1, pp. 2, 2016.
Emiel Hoogeboom, Victor Garcia Satorras, Jakub M Tomezak, and Max Welling, The convolution exponential and generalized sylvester flows. arXiv preprint arXiv:2006.0/910, 2020.
Gao Huang. Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks, arXiv, 2016.
Michael I. Jordan, Zoubin Ghahramani, and et al. An introduction to variational methods for graphical models. In Machine Learning, pp. 183-233. MIT Press, 1999.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv, 2014.

---

Under review as a conference paper at ICLR 2021
Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. arXiv, 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv, 2013.
Frantzeska Lavda, Magda Gregorová, and Alexandros Kalousis. Data-dependent conditional priors for unsupervised learning of multimodal data. Entropy, 22(8):888, 2020.
Shuyu Lin and Ronald Clark. Ladder: Latent data distribution modelling with a generative prior. arXiv preprini arXiv:2009.00000 2020.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Zhaoyu Wang. Li Mian, Jing Zhang, and Jie Tang. Self- supervised learning: Generative or contrastive. arXiv preprint arXiv:2006.082. 2020.
Lars Maaløe, Casper Kaae Sønderby, Søren Kaae Sønderby, and Ole Winther. Auxiliary deep generative models. arXiv, 2016.
Lars Maalće, Marco Fraccaro, Valentin Liévin, and Ole Winther. Biva: A very deep hierarchy of latent variables for generative modeling. arXiv, 2019.
Fabian Mentzer, Eirikur Agustsson, Michael T'chaanen,, Radu Timofte, and Luc Van Gool. Practical full resolution learned lossless image compression. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 10629-10638, 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: /80.00//07.3378,, 2018.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In Advances in Neural Information Processing Systems, pp. 14866-14876, 2019.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv, 2014.
Mihacla Rosca, Balaji Lakshminarayanan, and Shakir Mohamed. Distribution matching in variational inference. arXiv, 2018.
Tim Salimans and Diederik P. Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. arXiv, 2016
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. Pixelcnn+++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. arXiv, 2017.
Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using efficient sub-pixel convolutional neural network. arXiv, 2016.
Jakub M Tomezak and Max Welling, Vae with vampprior. arXiv preprint arXiv: 2017.
Arash Vahdat and Jan Kautz. Nvac: A deep hierarchical variational autoencoder. arXiv, 2020.
Rianne van den Berg, Leonard Hasenclever, Jakub M. Tomezak, and Max Welling. Sylvester normalizing flows for variational inference. arXiv, 2018.
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. arXiv, 2016.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103, 2008
Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1058-1067, 2017.
Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image suser-resolution using very deep residual channel attention networks. arXiv, 2018.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Learning hierarchical features from deep genera- tive models. In International Conference on Machine Learning, pp. 4091-4099, 2017.
10


Reviews:
[{"review_id": "s2btPNPN3I_", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n## Summary\n\nThe paper presents a self-supervised variational auto-encoder called selfVAE. The work proposes the use of downscaling and edge detection as simpler representations of the input images to be reconstructed. The model should then learn to improve the low dimensional approximations to recover the higher dimensional ones in a hierarchical fashion. \n\n## Quality & Clarity\n\nThe paper is generally quite difficult to follow and the purpose, contributions and experiments are not presented clearly enough. The figures are not discussed in order, and the paper often references figures that are far away.\n\nThere are a number of grammatical errors in the paper.\n\n## Outcome\n\nThe message of the paper generally was quite unclear and it could do with restructuring to assist readers.", "extra_scores": {}, "extra_text": {}}, {"review_id": "kmPCa5FOsV", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper targets richer and higher-quality generation with VAE. Two techniques are adopted to achieve the goal: 1). bijective model to enrich data generation with flexible prior. 2). presenting compressed variants of the input data, i.e. self -supervision as additional condition $y$, for reconstruction. The two techniques interact through a hierarchical sampling process, $... y\\sim p(y|u)\\rightarrow z\\sim p(z|u,y)$, thus benefits VAE generation with data-dependent prior and condition generation. \n\nThe idea novel and reasonable. the paper is clearly presented. Here are some of my concerns.\n\n1.  The author specifically argues the transformation $x \\rightarrow y$ to be 'non-trainable', i.e. the mapping between x and y is deterministic.  BUT, will modeling $q(x|z,y)$ with discretized logistic distribution, affect the generation quality, since the likelihood is classicly assumed to be Gaussian distributed? \n\n2. The HIERARCHICAL SELF-SUPERVISED VAE is presented here to show the model can adopt multi-scaling information to benefit generation step-by-step. However, I  am afraid, in this way, the inference would be much difficult since the flow-based bijective operation is hard to train already. \n\n3. Is the conditional information, e.g. the sketches, also need in the test phase? or unconditional generational setting is adopted here.\n\n4.  It seems the experiments are not conducted on High-quality datasets. To me, the presented results can not obviously demonstrate the achievements of the model. \n\n5. Can you please explain the connection between your self-supervised VAE to the general conditional VAE model in [1].\n\n[1] Sohn, Kihyuk, Honglak Lee, and Xinchen Yan. \"Learning structured output representation using deep conditional generative models.\" Advances in neural information processing systems. 2015.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZPKmNgEG8bx", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on the task of generating high-quality data with generative models. To be specific, the authors proposed a variant of variational autoencoder (VAE) model, named self-supervised VAE. The intuition behind this model is that by breaking down the complex generation task into simpler/smaller ones, complex models can be trained steadily with the guidance from the simpler-level task. To his end, a hierarchical generative model with multiple-level latent variables is proposed, in which lower-level latent variables are governed by lower-level data features. The lower-level feature is generally obtained by a determined and discrete transformation, like down scaling. In addition, to further the modeling capability, a flow-based prior is proposed to fit the data distribution. Experiments were conducted to evaluate the performance of the proposed generative model.\n\n\nStrength:\n1. The idea of guiding the complex image generation with easer tasks is interesting, and is maybe the right way to accomplish complex tasks.\n\n2. the ELOB directed in Eq.2 is intuitive and insightful. It also provides me theoretical support for the fact that employing two-level modeling and downscale transformation to generate a more vivid image is reasonable. \n\n\nWeakness:\n1. From a technical perspective, the proposed method is just the combination of flow-based VAE and auxiliary VAE. By using 3 auxiliary variables, the authors infer one of them by a discrete and determined variational distribution q(y|x) to simplify the training objective, where the downscale image y plays an important role in this model. My question is why not regard y as observed data and then model the joint distribution p(x,y).\n\n2. There are some mistakes in the derivation of Eq 2. In appendix A.4, during computing the entropy of q(w|x), the authors expresses it as E_{q(w|x)}[\\log q(w|x)] = E_{q(z|y,x)}[\\log q(z|y,x)] + ... . However, the first term in RHS is completely wrong. Actually, it should be E_{q(z|y,x)q(y|x)}[\\log q(z|y,x)]. It seems that the authors use an equation in many places, that is E_{q(z|y,x)}[\\log q(z|y,x)] = E_{q(z|y,x)q(u|y)q(y|x)}[\\log q(z|y,x)]. But this equation is not ture, because the term q(z|y,x) insided the expectation is dependent on variable y. Besides, in the choice of distribution p(y|u) and p(x|z,y), they are set to be a mixture discrete logistic distribution. For each image x or y,  are their pixels assumed to be i.i.d ? If so, you miss \\prod_{y_j \\in y} outside the \\sum_{i=1}^{I} in the distribution definition.\n\n3. Bijective prior (RealNVP) is proposed in other works, and here simply employing it should not be regard as a contribution of this paper. Moreover, the authors only compare the effectiveness of different priors (i.e. Gaussian, mixture Gaussian, and RealNVP) on vanilla VAE and confirm the superiority of using an adaptive prior. However, I want to know what is the performance of self-supervised VAE if only using a standard Gaussian prior.\n\n4. Section 3.3 is not presented well and the idea behind the sentences is hard to follow. What are the differences between these generation and reconstruction methods, and what application scenarios are corresponding to them? They are just simply listed, without providing any analysis of the logic behind them.\n\n5. The experimental results cannot support the superiority of the proposed model in both of the quantity and quality comparisons. From the generated images, I cannot see too much difference between the SelfVAE and the vanilla VAE model, without to naming the more superior generative models, like GLOW, GANs etcs. Also, for the quantity comparison, the model is only compared with the outdated vanilla VAE in CelebA and Imagenet64, more recent generative models should be included here.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "EZPCIMRKXV_", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "###################################\nPros:\n\n$\\bullet$ VAEs can ignore some dimensions of the latent code. Enforcing the posterior distributions to consider desired factors of variations in the input can be fulfilled by either making it more structured (i.e., quantization as in VQ-VAE-2) or introducing additional constraints. This paper tackles this problem by applying the latter, two self-supervised tasks: edge maps and downscaled versions of inputs.\n\n$\\bullet$ The idea of adding self-supervised tasks to improve latent representation is very interesting. When learning a more structured latent representation, image superresolution, or sketch-to-image networks are also trained.\n\n###################################\nCons:\n\n$\\bullet$ *Hierarchical self-supervised tasks* In section 3.4, multiple transformations explained. However, none of the experiments are conducted as a consecutive set of transformations. Does 3-level downscale mean a single downscaling three times or generating from $u$ using four different networks and match each of these levels with $z$? If yes, why does not the selfVAE-sketches model apply in a similar way hierarchically?  \n\n$\\bullet$ *How to train/balance operations* In Figure 3, there are several modes of operations given. How did you balance these modes during training?\n\n$\\bullet$ *Performance of self-supervised tasks:* What is the effect of self-supervised tasks' performance on the quality of latent representations? Considering the literature in image superresolution and sketch-to-image, did you use a pretrained auxiliary generator?\n\n$\\bullet$ *What is RE and KL in Table?* Are they the summation of both reconstruction ($RE_x, RE_y$) and KL divergence ($KL_z, KL_u$) terms in the loss (Eq.2)?  The reason why previous methods' RE/KL values were omitted should be stated. Similarly, why were the FID scores on CelebA and ImageNet-64 not given?   \n\nFurthermore, the state-of-the-art FID scores on CIFAR-10 is better than the methods compared in Table 1. For instance, some examples of FID scores on CIFAR-10 are 18.9 in MoML [1], 29.3 in WP-GAN [2], 29.3 in spectrally normalized GAN [3], 26.4 in adversarial score matching [4], and so on.\n\n[1] https://arxiv.org/pdf/1806.11006.pdf\n[2] https://arxiv.org/pdf/1706.08500.pdf\n[3] https://arxiv.org/pdf/1802.05957.pdf\n[4] https://arxiv.org/pdf/2009.05475.pdf\n\nAs the results on CelebA and Imagenet-64 were not compared with previous literature, it is difficult to understand whether the contribution w.r.t. vanilla VAE is due to the self-supervised task or merely the use of an additional stochastic variable ($u$) and networks. \n\nMinor issue: \"Imagenette64\" might cause confusion, I did not see this dataset name before. I suppose that it is \"ImageNet resized to 64x64\" as in PixelCNN paper. References for all datasets should be added.\n\n###################################\nReasons for score: \nOverall, I rate towards rejection. Even though the idea of bijective priors and doing this through self-supervised tasks is a novel approach, my major concern is that it is beyond the state-of-the-art in CIFAR-10, not compared to any other method on CelebA and ImageNet-64. Hopefully, the authors address my concerns above in the rebuttal period. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers appreciated the model and the ideas presented and found them very interesting.

The main reason for rejection is the extent of the empirical work.  Unfortunately, and I think what is a bad sign for the ICLR community, the authors could not do adequate empirical work due to their computational resources.  Not belonging to an organisation with extensive computational resources myself, I am in strong symparthy with the authors, though I do not see any way this can be satisfactorily accounted for in reviewing.  Several reviewers commented on the datasets, the extent of evaluations, and the comparisons made with prior work.  For instance, the small CIFAR10 images are not ideal to demonstrate the technique and comparative results with the other data sets are limited.

The reviewers had a number of concerns on the theoretical work and these were well discussed by the authors.

In summary, this is promising research but needs more empirical work.



Decision: Reject