Consistency Regularization for Variational Auto-Encoders
Authors: Samarth Sinha
Adji Bousso Dieng
Venue: NeurIPS 2021 Poster (2021)
Keywords: Variational Auto-Encoders

Abstract: Variational Auto-Encoders (VAEs) are a powerful approach to unsupervised learning. They enable scalable approximate posterior inference in latent-variable models using variational inference. A VAE posits a variational family parameterized by a deep neural network---called an encoder---that takes data as input. This encoder is shared across all the observations, which amortizes the cost of inference. However the encoder of a VAE has the undesirable property that it maps a given observation and a semantics-preserving transformation of it to different latent representations. This "inconsistency" of the encoder lowers the quality of the learned representations, especially for downstream tasks, and also negatively affects generalization. In this paper, we propose a regularization method to enforce consistency in VAEs. The idea is to minimize the Kullback-Leibler (KL) divergence between the variational distribution when conditioning on the observation and the variational distribution when conditioning on a random semantics-preserving transformation of this observation. This regularization is applicable to any VAE. In our experiments we apply it to four different VAE variants on several benchmark datasets and found it always improves the quality of the learned representations but also leads to better generalization. In particular, when applied to the Nouveau VAE (NVAE), our regularization method yields state-of-the-art performance on MNIST, CIFAR-10, and CELEBA. We also applied our method to 3D data and found it learns representations of superior quality as measured by accuracy on a downstream classification task. Finally, we show our method can even outperform the triplet loss, an advanced and popular contrastive learning-based method for representation learning.

Consistency Regularization for Variational Auto-Encoders
Samarth Sinha
Adji B. Dieng Google Brain Princeton University
Vector Institute University of Toronto
Abstract
Variational auto-encoders (VAES) are a powerful approach to unsupervised learning. They enable scalable approximate posterior inference in latent-variable models using variational inference (v1). A VAE posits variational family parameterized by deep neural network-callee an encoder that takes data as input. This encoder shared across all the observations, which amortizes the cost of inference. However the encoder of a VAE has the undesirable property that it maps a given observation transtormation of it to different latent representations.
and semantics-presee This "inconsistency" of the encoder lowers the quality of the learned representations, especially for downstream tasks, and also negatively affects generalization. In this paper, we propose a regularization method to enforce consistency in VAES. The idea is minimize the Kullback-Leinn bution when conditioning on the observation and the variational distribution when conditioning random crmantic-preserving transformation of this observation. This regularization is applicable to any VAF. our experiments we apply it to four different VAE variants on several benehmark datasets and found it always improves the quality of the learned representations but also leads to better generalization. In particular, when applied the nouveau variational auto-encoder (NVAE), our regularization method yields -heee----heetaat performance on MNIST, CIFAR- 10, and CELEBA. We also applied our method 3D data and found it learns representations of superior quality as measured by accuracy on a downstream classification task. Finally, we show our method can even outperform the triplet loss, an advanced and popular contrastive learning-based method for representation learning.
(KI.) divergence between the variational distri-
Introduction
Variational auto-encoders (VAES) have significantly impacted research on unsupervised learning. They have been used several areas, including density estimation (Kingma & Welling, 2013; Rezende et al., 2014), image generation (Gregor et al., 2015), text generation (Bowman al., 2015; Fang et al., 2019), music generation (Roberts et al., 2018), topic modeling (Miao al., 2016; Dieng et al., 2019), and recommendation systems (Liang et al., 2018 ()). VAES have also been used for different representation learning problems such as semi-supervised learning (Kingma et al., 2014), anomaly detection (An & Cho, 2015; Zimmerer et al., 2018), language modeling Bowman et al. (2015), active learning (Sinha et al., 2019), continual lcarning (Achille et al., 2018), and motion prediction of agents (Walker et al., 20. This widespread application of VAE representations makes it critical that focus on improving them.
VAES extend deterministic auto-encoders probabilistic generative modeling. The encoder of VAE parameterizes an upproximate posterior distribution over latent variables of generative model. The encoder is shared between all observations, which amortizes the cost of posterior inference. Once fitted, the encoder of a VAE can be used to obtain low-dimensional representations of data, (e.g. for
Code for this work can be found at wttpt:::////gtt tttp///ttttttttte/e/e
35th Conference on Neural Information Processing Systems (NeurTPS 2021).

---

(a)
(b)
Figure 1: Illustration of the inconsistency problem in VAES and how CR-VAES address this problem. The red dots correspond to the representations of few images from MNIST. The blue dots correspond to the representations of the transformed images. The transformations used here are rotations, translations, and scaling; they are preeerrici...eeiiingg..... The arrows connect the representations of any two pairs of an image and its transformation. The shorter the arrow, the better. (a): The VAE maps the two sets of images to different areas in the latent space. (b): Even when trained with the original dataset augmented with the transformed images, the VAE still maps the two sets of images to different parts in the latent space. (c): The CR-VAE maps an image and its transformation to nearby areas in the latent space.
downstream tasks.) The quality of these representations is therefore very important to successful application of VAES.
Researchers have looked ways improve the quality of the latent representations of VAES, often tackling the so-called latent variable collapse problem-in which the approximate posterior distribu- tion induced by the encoder collapses the prior over the latent variables (Bowman et al., 2015; Kim et al., 2018; Dieng et al., 2018; He et al., 2019; Fu et al., 2019).
In this paper, focus on a different problem pertaining to the latent representations of VAES for image data. Indeed, the encoder of fitted VAE tends to map an image and a semantics smmantics-preserving transformation of that image different parts in the latent space. This "inconsistency" of the encoder affects the quality of the learned representations and generalization. We propose a method to enforce consistency in VAES. The idea is simple and consists in maximizing the likelihood of the images while minimizing the Kullback-Leible (KL) divergence between the approximate posterior distribution induced by the encoder when conditioning on the image, on one hand, and its transformation, on the other hand. This regularization technique can be applied to any VAE variant to improve the quality of the learned representations and boost generalization performance. We call VAE with this form of regularization, a mansistencc-reguariizee variational auto-encoder (CR-VAB).
Figure illustrates the inconsistency problem of VAES and how CR-VAES address this problem on MNIST. The red dots are representations of few images and the blue dots are the representations of their transformations. We applied semantics- semantics-preserving transformations: rotation, translation, and scaling. The VAE maps each image and its transformation to different parts in the latent space as evidenced by the long arrows connecting each pair (a). Even when we include the transformed images to the data and fit the VAE the inconsistency problem still occurs (b). The CR-VAE does not suffer from the inconsistency problem; it maps each image and its transformation to nearby areas in the latent space, evidenced by the short arrows connecting each pair (c).
In our experiments (see Section 4), we apply the proposed technique to four VAE variants, the original VAE (Kingma& Welling, 2013), the importance-weigghe weighted auto-encoder (IWAE) (Burda et al., 2015), the 3-VAE (Higgins et al., 2017), and the nouveau variational auto-encoder (NVAE) (Vahdat & Kautz, 2020). We found, on four different benchmark datasets, that CR-VAES always yield better representations and generalize better than their base VAES. In particular, consisenccyreegulaizee nouveau variational auto-encoders (CR-NVAES) yield ffat-oof-theaa performance on MNIST and CIFAR-10. We also applied CR-VAES to 3D data where these conclusions still hold.
Method
We consider a fatent-variablee model po(x, z) = po
po(x(2 x2] p((),, where x denotes an observation and is its associated latent variable. The marginal (z) is prior over the latent variable and pe (x|z)

---

is an exponential family distribution whose natural parameter is a function of parameterized by 6, e.g. through neural network. Our goal is learn the parameters @ and a posterior distribution over the latent variables. The approach of VAES is to maximize the evidence lower bound (FIBO), a lower bound on the log marginal likelihood of the data,
po z) 96 (z|xx)
VAE = ELBO Egg (x/x))
log
(I)
where qo (z(x)) is an approximate posterior distribution over the latent variables. The idea of a VAE to let the parameters of the distribution (z(z)) be given by the output of a neural network, with parameters 6. that takes x as input. The parameters 0 and & are then jointly optimized by maximizing a Monte Carlo approximation of the ELBO using the reparameterization trick (Kingma Welling, 2013).
Consider a prrnttrcig- transformation t(xxx) of data x (e.g. rotation or translation for images.) A good representation learning algorithm should provide similar latent representations for x and x. This not the case for the VAE that maximizes Equation I and its variants. Once fit data, the encoder of a VAE is unable to yield similar latent representations for a data x and its tranformation (see Figure 1). This because there is nothing Equation that forces this desideratum.
We now propose regularization method that ensures consistency of the encoder of a VAE. We call a VAE with such regularization a CR-VAE. The regularization proposed is applicable to many variants of the VAE such as the IWAE (Burda et al., 2015), the 3-VAE (Higgins et al., 2017), and the NVAE (Vahdat & Kautz, 2020). In what follows, we use the standard VAE, the one that maximizes Equation 1, as the base VAB regularize to illustrate the method.
Consider an image x. Denote by f(x(x) the random process by which we generate x. semantics- preserving transformation of x. We draw x from xx as follows:
x ~ xx
= c~ p(c) and x = g(x,
(2)
Here g(x, <) is a semantics-preserving transformation of the image x, e.g. translation with random length e drawn from p(e) U[[68,]] for some threshold d. A CR-VAE then maximizes
CERVAE (x) frve (x) + (x(()))) (fvvE()))
R(x, Φ)
(3)
where the regularization term R(x, o) is
R(x, 6) (((x))) [KL (2x(21
(4)
Maximizing the objective in Equation 3 maximizes the likelihood of the data and their augmentations while enforcing consistency through R(x, Φ). Minimizing R(x, Φ), which only affects the encoder
(with parameters Φ), forces each observation and the corresponding augmentations to lie close to each other in the latent space. The hyperparameter Z controls the strength of this constraint.
The objective in Equation 3 is intractable but we can easily approximate it using Monte Carlo with the reparameterizztioon trick. In particular, we approximate the regularization term with one sample from t x( and make the dependence to this sample explicit using the notation R(x, X, Φ). Algorithm 1 illustrates this in greater detail. Although we show the application of consistency regularization using the VAE that maximizes the ELBO, (-va(-) in Equation 3 can be replaced with any VAE objective.
3 Related Work
Applying consistency regularization to VAES, as we do in this paper, has not been previously explored. Consistency regularization is widely used technique for semi-supervised learning (Bachman et al., 2014; Sajjadi et al., 2016; Laine & Aila, 2016; Miyato et al., 2018; Xie et al., 2019). The core idea behind consistency regularization for semi- cumi-supervised learning to force classifiers learn representations that are insensitive to presservnn changes to images, so as to improve classification of unlabeled images. Examples of semantics-preserving changes used in the literature include rotation, zoom, translation, crop, or adversarial attacks. Consistency is often enforced by minimizing the L22 distance between a classifier's 's logit output for an image and the logit output for its semantics preserics-preserving transformation (Sajjadi et al., 2016; Laine & Aila, 2016), by minimizing
3

---

Algorithm 1: Consistency Regularization for Variational Autoencoders
input Data consistency regularization strength A, latent space dimensionality K Initialize parameters 0,6 for iteration 1, 2, ... do
Draw minibatch of observations } for n 1, .......... B do
Transform the data: en p(En) and Xn T(x,; En)
Get variational mean and variance for the data:
nn = W NN(x,;;) Φ) + a and TT softplus NN(x,, Φ) + b) Get S samples from the variational distribution when conditioning on x,:
(7(*) ~ I). and Zn (x)
f,, -7(()) on for
Get variational mean and variance for the transformed data:
fn W NN(*,;;) Φ) + a and o, softplus(Q NN( Φ) + +b) Get S samples from the variational distribution when conditioning on Xn
x)(+) ~ I), and (.(²) in, =n(() on for 1....,
end
Compute LuVE(X): (x):
(x) B ==== S -==1
D
log po 21 )- log gol (z!:") [xx)
Compute (R):
( B Ex=1 5 E==1
log (o z((1) log qal (2( (xn)
Compute KL consistency regularizer:
R(x, &, Φ) = E
Hunh 1 1 log aak Frik
Compute final loss:
LeR-vA(()) xx== L (x) + Lx) R(x,x.0)
Backpropagate through L(x, -CeRwA((X) and take gradient step for and d
end
the KL divergence between the classifier's label distribution induced by the image and that of its tranformation (Miyato et al., 2018; Xie et al., 2019).
More recently, consistency regularization has been applied to generative adversarial networks (GANS) (Goodfellow et al., 2014). Indeed Wei et al. (2018) and Zhang et al. (2020) show that applying consistency regularization on the discriminator of GAN-also a classifier-can substantially improve its performance.
The idea we develop this paper differs from the works above two ways. First, applies consistency regularization to VAES for image data. Second, it leverages consistency regularization, not in the label or logit space, as done in the works mentioned above, but in the latent space.
Although different, consistency regularization for VAES relates works that study ways to constrain the sensitivity of encoders to various perturbations. For example, denoising auto-encoders (DAES) and their variants (Vincent et al., 2008, 2010) corrupt image into x', typically using Gaussian noise, and then minimize the distance between the reconstruction x' and the un-corrupted image x. The motivation is learn representations that insensitive the added noise. Our work differs in that we do not constrain the decoder to recover the original image from the corrupted image but, rather, to constrain the encoder recover the latent representation of the original image from the corrupted image via a KL divergence minimization constraint.
Contractive auto-encoders (CAES) (Rifai al., 2011) share similar goal with CR-VAES. A CAE an auto-encoder whose encoder is constrained by minimizing the norm of the Jacobian of the output of the encoder with respect to the input image. This norm constraint on the Jacobian forces the representations learned by the encoder to be insensitive to changes in the input. Our work differs in several main ways. First, CR-VAES are not deterministic auto-encoders, contrary to CAES. We can casily sample from CR-VAE, as for any VAE, which is not the case for CAE. Second, CAB does

---

Table 1: CR-VAES learn better representations than their base VAES on all three benchmark datasets. Although fitting the base VAE with augmentations does improve the representations, adding the consistency regularization further improves the quality of these learned representations. The value of
B for the 8-VAE is inside the parentheses.
MNIST
OMNIGLOT
CELEBA
Method
MI
AU
MI
AU
MI
AU
VAE
124.5 + 1.1
36 0.8 0.8
105.4
50 0.0
33.8
+ 0.2
32 = 0.9
VAE Aug
125.9 + 0.2
42 0.5
0.7
34.1 5.0
33
1 0.9
CR-VAE
126.3 + 0.9
47 4 0.5
107.8
5.0
0.5 34.9
33 $ 1.2
IWAE
127.1
39 0.5
110.3
5.0
36.9 0.5
36 1.6
IWAE+Aug
129.0
45 0.8
112.9
+.0
37.0
0.2
36 1.2
CR-IWAR
129.7 + 1.0
50 + 0.0
115.3 0.8
+.
38.4 + 0.5
36 + 1.9
B-VAE (0.5)
284.3 1.1
50 + 0.0
1.0
5.0
0.5
49 0.5
8-VAE (0.5) Aug
289.3 1.0
5.0
159.6
+.0
7
49 0.0
B-CR-VAE (0.5)
$91.9 0.7
0.0
169.5 0.5
50
77.1 0.1
50 0.0
3-VAE (10)
6.3 1
8.7
1.2
4
3.6
7 0.8
8-VAE (10) + Aug
6.5 0.5
9111
1.6
0.5
3.7
7 0.0
3-CR-VAE (10)
6.9 - 0.6
10
1.6
0.5
3.7 6.7004 0.4
9 0.9
not apply transformations the input image, which limits the sensitivities it can learn to limit to those exhibited in the training set. Finally, CAES use the Jacobian to impose a consistency constraint, which not easy to compute the KL divergence on the variational distribution induced by the encoder.
4 Empirical Study
In this section we show that a CR-VAE improves the learned representations of its base VAE and positively affects generalization performance We also show that the proposed regularization method amenable different VAE variants by applying it not only to the original VAE but also to the IWAE, the 3-VAF, and the NVAE, We showcase the importance of the KI. regularization term by conducting ablation study. We found that only regularizing with data augmentation improves performance but that accounting for the KL term (A 0) further improves the quality of the learned representations and generalization.
We will conduct three sets of experiments. In the first experiment, we will apply the regularization method proposed in this paper to standard VAES such as the original VAE, the IWAE, and the f-VAE. use MNIST, OMNIGLOT, and CELEBA as datasets
this experiment. For CELEBA, we choose the 32x32 resolution for this experiment. Our results show that adding consistency regularization always improves upon the base VAE, both in terms of the quality of the learned representations and generalization. We conduct an ablation study and also report performance of the different VAE variants above when they are fitted with the original data and their augmentations. The results from this ablation highlight the importance of setting À 0.
In the second set experiments we apply our method to a large-scale VAE, the latest NVAE (Vahdat Kautz, 2020). We use MNIST, 10, and CELEBA datasets for this experiment. We increased resolution for CELEBA dataset for this experiment to 64x64. We reach the same conclusions for the first sets of experiments; CR-VAES improve the learned representations and generalization of their base VAES. In this particular setting, the CR-NVAE achieves -fate-of-the-at generalization performance both MNIST and CIFAR- 10. This state-of-the-an performance couldn't 't be reach simply by training the NVAE with augmentations, as our results show.
Finally, in third set of experiments, apply regularization technique to 3D point-cloud dataset called ShapeNet (Chang et al., 2015). We adapt a nigh-performing auto-encoding method called FoldingNet (Yang et al., 2018) to its VAE counterpart and apply the method we described in this paper to that VAE variant on the ShapeNet dataset. We found that adding consistency regularization yields better learned representations.
We next describe great detail the set up for each of these experiments and the results showcasing the usefulness of the regularization method propose in this paper.
5

---

Table 2: CR-VAES learn representations that yield higher accuracy on downstream classification than their base VAES. These results correspond to the accuracy from a linear classifier that was fitted on the training. We fed this classifier with the representations learned by each method. both MNIST and CIFAR- 10, CR-VAES yield higher accuracy.
Method
MNIST
CIFAR-I8
VAE
98.5
32.6
VAE+Aug
98.9
40.1
CR-VAE
99.4
44.7
IWAE
98.6
35.8
IWAE+Aug
99.9
37.1
CR-IWAR
99.9
44.8
B- VAE (0.5)
97.6
27.0
B- VAE (0.5)+Aug
98.7
27.6
B- CR-VAE (0.5)
98.9
30.0
3- VAE (10)
99.4
36.5
VAE (10)+Aug
99.6
42.1
B- CR-VAE (10)
99.6
46.1
Table 3: CR-VAES generalize better than their base VAES on almost all cases; they achieve lower negative log- se-likkeiioooss. Although training the base VAES with the augmented data improves general- ization, adding the consistency regularization term further improves generalization performance.
Method
MNIST
OMNIGLOT
CELEBA
83.7 0.3
128.2 + 0.8
66.1 + 0.2
VAE + Aug
82.8 = 0.4
125.7 + 0.2
66.0 0.2
CR-VAE
81.2 = 0.2
124.1 0.1
6.2
IWAE
81.7 = 0.3
127.5 0.5
6.1
IWAE+Aug
80.4 = 0.2
125.0=0..
6.1
CR-IWAE
79.7 = 0.3
123.6 + 0.5
6.20
8-VAE (0.5)
92.6 = 0.3
137.1 0.2
6..
8-VAE (0.5) Aug
90.0 = 0.5
134.6 0.5
6.2
B-CR-VAE (0.5)
85.7 = 0.6
132.5 + 0.3
0.1
3-VAE (10)
126.1 + 1.8
157.5 - 1.1
9..7
3-VAE (10) Aug
127.1 + 1.0
157.3 0.5
0.3
B-CR-VAE (10)
126.2 0.5
157.6 = 0.6
92.6
4.1 Application to standard VAES on benchmark datasets
We apply consistency regularization, described in this paper, the original VAEE, the IWAE, and the S-VAE. We now describe the set up and results for this experiment.
Datasets. We study three benchmark datasets that briefly describe below. We first consider MNIST. MNIST is a handwritten digit recognition dataset with 60, 000 images in the training set and 10, 000 images in the test set (LeCun, 1998). We form validation of 10, 000 images randomly sampled from the training set.
We also consider OMNIGLOT, handwritten alphabet recognition dataset (Lake et al., 2011). This dataset is composed of 19, 280 images. We use 16, 280 randomly sampled images for training and 1, 000 for validation and the remaining 2, 000 samples for testing.
Finally we consider CELEBA. It a dataset of faces, consisting of 162, 770 images for training, 867 images for validation, and 19, 962 images for testing (Liu al., 2018). We set the resolution 32x32 for this experiment.
Transformations 1(x x). We consider three transformations variants for image data ()) The first randomly translates image 2- 2] pixels in any direction. The second transformation randomly

---

Table 4: The regularization term À affects both generalization performance and the quality of the learned representations. Many values of À perform better than the base VAE. However large enough value of X, e.g. 1, can lead to worse performance than the base VAE because for large values of the regularization term takes over the data-term in the objective function.
MI
AU
NLL
VAE
124.5
36
83.7
CR-VAE
0.001
125.0
38
83.5
CR-VAE
0.01
125.9
41
82.4
CR-VAE
0.1
126.3
47
81.2
CR-VAE
124.3
47
83.9
Table 5: The choice of augmentation affects both generalization performance and the quality of the learned representations. Jointly using all augmentations works best.
Augmentation
MI
AU
NLL
Rotations only
125.8
45
82.1
Translations only
126.1
45
81.9
Scaling only
125.1
42
82.7
All
126.3
47
81.2
rotates an image uniformly in [-15, 15] degrees clockwise, Finally the third transformation randomly scales an image by a factor uniformly sampled from [0.9, 1.1].
Evaluation metrics. The regularization method we propose in this paper mainly aimed improving the learned representations of VAES. To assess these representations we use three metries: mutual information, number of active latent units, and accuracy on downstream classification task. We also evaluate the effect of the proposed method generalization to unseen data. For that we also report negative log- liglliiel.oood. We define each of these metrics next.
Muiual information (MI). The first quality metric is the mutual information [(z; x) between the observations and the latents under the joint distribution induced by the encoder,
[(; Bp)()) (gs [[(()) (z)
(5)
where pa px(x) is the empirical data distribution and (z) is the aggregated posterior, the marginal over z induced by the joint distribution defined by (x() and go (z)(2)x2 x). The mutual information is intractable but
approximate it with Monte Carlo. Higher mutual information corresponds to more interpretable latent variables.
Number of active latent units (AU). The second quality metrics consider is the number of active latent units (AU). It is defined in Burda et al. (2015) and measures the "activity" of a dimension of the latent variables A latent dimension is "active" if
Coax (Burgs > 8
(
where 5 is a threshold defined by the user. For our experiments we set = 0.01. The higher the number of latent active units, the better the learned representations.
Accuracy on downstream classification. This metric is calculated by fitting a given VAE, taking the learned representations for each data in the test set and computing the accuracy from the prediction of the labels of the images in that same test set by a classifier fitted on the training set. This metric is only applicable to labelled datasets.
Negative log- likelihood. We use negative held-out log- likelihood to assess generalization. Consider an unseen data x", its negative held-out log- legllielloood under the fitted model is
[xoxx 22,
1)
log (x*) = -log (Buo(eix))
(x(2)x*)
7

---

Table 6: The CR-VAE outperforms a popular and advanced contrastive learning technique called triplet loss on both generalization performance and quality of learned representations.
Method
MI
AU
NLL
VAE
124.5
36
83.7
VAE + augmentations
125.9
42
82.8
VAE + triplet loss
124.9
39
83.1
CR-VAE
126.3
47
81.2
This is intractable and we approximate it using Monte Carlo,
5 po fnxxx2"0 22()))
lxg 2- - 5 go (2(0) x*)
(8)
where z (5)) (z)x)))
Settings. The VAES built the same architecture as Tolstikhin et al. (2017). The networks trained with the Adam optimizer with learning rate of 10 (Kingma & Ba, 2014) and trained for 100 epochs with batch size of 64. We set the dimensionality of the latent variables to 50, therefore the maximum numher of active latent units in the latent space is 50. We found X = 0.1 to be best according pross-validation using held-out log-likelihood and exploring the range [1e-1, 1.0] datasets. In an ablation study we explore À = O. For the ß-VAR we set À = 0.1 . / and study both , = 0.1 and 10, two regimes under which the 3-VAE performs qualitatively very differently (Higgins et al., 2017). All experiments were done on a GPU cluster consisting of Nvidia P100 and RTX. The training took approximately I day for most experiments.
Results. Table shows that on all the three benchmark datasets all the different VAE variants we studied, consistency regularization developed in this paper always improves the quality of the learned representations as measured by mutual information and the number of active latent units. These results confirmed by the numbers shown in Table 2 where CR-VAES always lead to better accuracy on downstream classification.
We proposed consistency regularization way to improve the quality of the learned representations. Incidentally, Table 3 also shows that it can improve generalization as measured by negative log- likelihood.
Ablation Study. We now look the impact of each factor that goes into the regularization method we introduced in this paper using MNIST. We test the impact of the regularization term X and the impact of the choice of augmentation on all metrics. Table and Table 5 show the results.
Table 4 shows that even small consistency regularization (a small À value) results in improvement over the base VAE but that large enough value can hurt performance.
Table 5 shows that rotations and translations are more important than scaling, but the combination of all three augmentations works best for CR-VAES.
Comparison to Contrastive Learning. We look how CR-VAES compare against a popular and technique, the triplet loss (Schroff et al., 2015) using MNIST. Table shows that the CR-VAE outperforms the triplet loss on both generalization performance and quality of learned representations. Table 6 also confirms existing literature showing simply applying augmentations can outperform complex contrastive learning-bast based methods such as the triplet loss (Kostrikov et al., 2020; Sinha & Garg, 2021).
advanced -eerrastive-larninng--
4.2 Application to the large-scale NVAE on benchmark datasets
Along with standard VAE variants, we also experiment with a large scale state-of-the-an VAE, the NvAe(Vahdat & Kautz, 2020). Similar to before, simply add consistency regularization using the image-based augmentations techniques to the NVAE model and experiment on benchmark datasets: MNIST (LeCun, 1998), CIFAR-10 (Krizhevsky al., 2009) and CELEBA (Liu al., 2018).
The results for large scale generative modeling are tabulated Table and Table 7, where we see that using CR-NVAE we are able to learn representations that yield better accuracy on downstream
8

---

The CR-NVAES learns better representations than the base NVAE measured by accuracy on a downstream classification both MNIST and CIFAR- 10, We get to this same conclusion when looking at the number of active units as an indicator for the quality the learned latent representations; CR-NVAE recovers 226 units whereas NVAE recovers 211 units.
Table 7:
Method
MNIST
CIFAR- 10
NVAE
99.9
57.9
NVaE+Aug
99.9
66.4
CR-NVAE
99.9
71.4
Table Large-scale experiments with NVAES with and without resisttencyrrgullrizatiioo on benchmark datasets: dynamically binarized MNIST, CIFAR- 10 and CELEBA. We report generalization using negative likellili on MNIST and bits per dim on CIFAR-10 10 and CELEBA. On all datasets
consistency regularization improves generalization performance. In particular CR-NVAE achieves state-of-the-ard performance on MNIST and CIFAR-10.
MNIST (28 28)
CIFAR-10 (32 32)
CELEBA (64 64)
NVAE NvgE+Aug
78.19
2.91
2.03
77.53
2.70
1.96
CR-NVAE
76.93
2.51
1.86
Figure 2: Interpolation between two samples of lamp, airplane and table using trained CR- FoldingNet trained on the ShapeNet dataset. The CR-FoldingNet able to learn an interpretable latent space.
classification and set new state-of-the-ee -rr values on each of the datasets, improving upon the baseline log-likelihin likelikelhood values. This shows the ability of consistency regularization to work at scale on challenging generative modeling tasks.
4.3 Application to the FoldingNet on 3D point-cloud data
Along with working with image data, we additionally experiment with point cloud using a FoldingNet Yang et al. (2018) and the ShapeNet dataset Chang et al. (2015) which consists of 55 distinct object classes. FoldingNet learns deep AutoEncoder to learn unsupervised representations from the point cloud data. To add consistency regularization, first substitute the AutoEncoder to

---

Table 9: The FoldingNet yields higher accuracy when paired with consisteney regularization on the ShapeNet dataset. The results shown here correspond to FoldingNet that was trained with augmented data, the same used to apply consistency regularization. As can be seen from these results, enforcing consistency through KI. as we do in this paper leads to representations that perform well on downstream classification. Here the classifier used a linear SVM. We also report mean reconstruction error through Chamfer distance where the same conclusion holds.
Method
Accuracy
Reconstruction Loss
Folding Net (Aug) 82.5%
0.0355
CR-Folding Net
84.6%
0.0327
VAE by adding the KL term from the ELBO to the baseline FoldingNet. We then add the additional consistency regularization KL term to the latent space of FoldingNet.
For the ShapeNet point cloud data, we perform data augmentation using a similar scheme to what we did for the previous experiments, randomly translate, rotate and add jitter to the (I, y, coordinates the point cloud data. We follow the same scheme detailed in FoldingNet (Yang et al., 2018).
We train both the FoldingNet turned in a VAE and the CR-FoldingNet with these augmentations. To train CR-FoldingNet, we additionally apply the consistency regularization term as proposed in Equation 3. The results on the validation set for reconstruction (as measured by Chamfer distance) and accuracy are shown in Table 9.
We also visualize the point clouds reconstructions and interpolations between different object classes using a CR-FoldingNet Figure We perform 4 interpolation steps for each of the objects, to highlight the interpretable learned latent space. Additionally, we perform the same interpolation on the baseline FoldingNet model. We show these interpolations the appendix.
5 Conclusion
We proposed simple regularization technique to constrain encoders of VAES to learn similar latent representations for an image and a comantics-preservve consists in maximizing the likelihood of the pair of images while minimizing the KI. divergence between the variational distribution induced by the encoder when conditioning on the image on one hand, and its transformation, on the other hand. We applied this technique to several VAE variants on several datasets, including 3D dataset. We found it always leads to better learned representations and also better generalization to unseen data. In particular, when applied to the NVAE, the regularization technique we developed in this paper yields state-of-the-an results on MNIST and CIFAR-IO.
transformation of the image. The idea
Broader Impact
In this paper, we propose a simple method that performs a KL-based consistency regularization scheme using data augmentation for VAES. The broader impact of the study includes practical applications such as graphics and computer vision applications. The method we propose improves the learned representations of VAES, and as an artifact, also improves their generalization this regard, any implications of VAES also apply to this work. For example, the generative model fit by a VAE may be used to generate artificial data such as images, text, and 3D objects. Biases may arise as a result of poor data selection. Furthermore, text generated from generative systems may amplify harmful speech contained in the data. However, the method we propose can also improve the performance of VAES when used in certain practical domains we discussed in the introduction of the paper.
unseen data. In
6 Acknowledgements
We thank Kevin Murphy, Ben Poole, and Augustus Odena for their comments this work.
10

---

References
Achille. A., Eccles, T.. Matthey, L., Burgess, C. P. Watters, N., Lerchner, A., and Higgins. 1. Life-long disentangled representation learning with cross-domain latent homologies. arXiv preprint arXiv: (a:::18088000 06508. 2018.
An, J. and Cho, S. Variational autoencoder based anomaly detection using reconstruction probability. Special Lecture IE, 2(1), 2015.
Bachman, P. Alsharif, O., and Precup. D. Learning with pseudo-ensembles. In Advances in neural information processing systems, pp. 3365 3373, 2014.
Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A. M., Jozefowicz, R., and Bengio, S. Generating sentences from continuous space, arXiv preprint arXiv: 1.1.v15511003444 2015.
Burda, Y., Grosse, R., and Salakhutdinov, R. arXiv:1509.00577 2015.
Importance weighted autoencoders.
arXiv preprint
Chang. A. X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q. Li, Z., Savarese, S., Savva, M., Song. S., Su, H., et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv: 1512.03012 20155.
Dieng, A. B., Kim, Y., Rush, M., Blei, Avoiding latent variable collapse with generative skip models. arXiv preprint 1exxiv:807..4450 2018.
Dieng, A. B., Ruiz, J.J., and Blei, D. M. Topic modeling embedding spaces. arXiv preprint arXiv: 1907.04977 2019.
Fang, L.,, Li, C., Gao, J., Dong, W., and Chen, C. Implicit deep latent variable models for generation. arXiv preprint arXiv: /908. 1527, 2019.
Fu, H., Li, C., Liu, X., Gao, Celikyilmaz, A., and Carin, L. Cyclical annealing schedule: A simple approach to mitigating vanishing. arXiv preprint (t5.::1903.11111 2019.
Goodfellow, I., Poudet-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y.
Generative adversarial nets. In Advances in neural information processing systems, pp. 2672-2680, 2014.
Gregor, K., Danihelka, I. Graves, A.. Rezende, D. J., and Wierstra, D. Draw: A recurrent neural network for image generation. arXiv preprint arXiv:1502.04400 2015.
Hadjeres, G., Nielsen, F., and Pachet, F. Glsr-vae: Geodesic latent space regularization for variational autoencoder architectures. In 2017 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 1-7. IEEE, 2017.
He. J.. Spokoyny, D., Neubig, G., and Kong-Kiikpatrrick T. Lagging inference networks and posterior collapse in variational autoencoders. arXiv preprint arXiv: 1001:1900105500 1.5534 2019.
Higgins, I., Matthey, I.,, Pal, A., Burgess, C., Glorat, X., Botvinick, M., Mohamed, S., and I Lerchner, A, beta-vae: Learning basic visual concepts with a constrained variational framework. Icbr, 2(5)6, 2017.
Jun, H., Child, R., Chen, M., Schulman, Ramesh, A., Radford, A., and Sutskever, Distribution augmentation for generative modeling. In International Conference on Machine Learning, pp. 5006 5019. PMLR, 2020,
Kim, Y., Wiseman, Miller, A. C., Sontag, D., and Rush, M. Semi-amortized variational autocncoders. arXiv preprint ar 1802.200..22 0805002550 2018.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXhv:1412.6980 2014. 2014.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013.
Kingma, D. P., Rezende, D. J., Mohamed, S., and Welling, M. Semi-supervised learning with deep generative models. arXiv preprint arXiv: 1406.5298, 2014.
Kostrikov, L. Yarats, D., and Fergus. R. Image augmentation all you need: Regularizing deep reinforcement learning from pixels. arXiy preprint arXiv:200411649, 2020.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Laine, S. and Aila, T. Temporal ensembling for semi-supervised learning. arXiv preprint 1rX.v.161 2016.
Lake, B., Salakhutdinov, R., Gross, J., and Tenenbaum, J. One shot learning of simple visual concepts. In Proceedings of the annual meeting of the cognitive science society, volume 33, 2011.
11

---

LeCun, Y. The mnist database of handwritten digits. http://yann. lecun. com//xxlb/mnist// 1998.
Liang. D., Krishnan, R. G., Hoffman, M. D., and Jebara, Variational autoencoders for collaborative filtering. In Proceedings of the 2018 World Wide Web Conference, pp. 689 698, 2018.
Liu, Z., Luo, P., Wang, X., and Tang, Large-scale celebfaces attributes (ccleba) datasct. Retrieved August, 15: 2018, 2018.
Miao, Y., Yu, L.,, and Blunsom, Neural variational inference for text processing. In International conference on machine learning, pp. 1727-1736, 2016.
Miyato, T., Maeda, S.-i., Ishii, S., and Koyama, M. Virtual adversarial training: a regularization method for supervised and semi semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 2018.
Osada, G., Ahsan, B., Bora, R. P., and Nishide, T. Regularization with latent space virtual adversarial training. In European Conference on Computer Vision, pp. 161-581. Springer, 2020.
Rezende, D. J., Mohamed, S., and Wierstra, Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.100 100,,402, 2014.
Rifai, S., Vincent, P., Muller, X., Glorot, X., and Bengio, Contractive auto-encoders: Explicit invariance during feature extraction. 2011.
Roberts, A., Engel, J., Raffel, C., Hawthorne, C., and Eck, D. hierarchical latent vector model for learning long-term structure in music. arXiv preprint 1803vv1800005420 2018.
Sajjadi, M., Javanmardi, M., and Tasdizen, Regularization with stochastic transformations and perturbations for deep cupersuped learning. In NeurlPS, 2016.
Schroff, F., Kalenichenko, D., and Philbin, J. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 815-823, 2015.
Sinha, S. and Garg, A. S4rl: Surprisingly simple self-supervision for offline reinforcement learning. arXiv preprint 103.020033 2021.
Sinha, S., Ebrahimi, S., and Darrell, T. Variational adversarial active learning. In Proceedings of the IEEE International Conference on Computer Vision, pp- 5972-5981, 2019.
Tolstikhin, I., Bousquet, O., Gelly, S., and Schoelkopf, B. Wasserstein auto-encoders. arXiv preprint arXiv: 171.00711.05558 2017.
Vahdat, A. and Kautz, J. Nvae: A deep hierarchical variational autoencoder. arXiv preprint arXi::0007.3389. 2020.
Vincent, P., Larochelle. H., Bengio, Y., and Manzagol. P.-A. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference Machine learning, pp. 1096-1103, 2008.
Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., and Manzagol, P.-A. useful representations in deep network with local denoising criterion. Journal of machine learning research, 11(Dcc:33311-3408 2010.
Stacked denoising autoencoders: Learning
Walker, J., Doersch, C., Gupta, A., and Hebert, M. An uncertain future: Forecasting from static images using variational autoencoders. In European Conference on Computer Vision, pp. 835-851. Springer, 2016.
Wei, X., Gong, B., Liu, Z., Lu, W.. and Wang, L. Improving the improved training of wasserstein gans: consistency term and its dual effect. arXiv preprint arXiv: 1803.01541, 2018.
Xie, Q. Dai, Z., Hovy, E.., Luong, M.-T., and Tee, Q V. Unsupervised data augmentation for consistency training. arXiv preprint arXiv: 1904. 12848, 2019.
Yang, Y., Feng, C., Shen, Y., and Tian, D. Foldingnet: Point cloud auto-encoder via deep grid deformation. In
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 206-215, 2018.
Zhang, H. Zhang, Z., Odena, A., and Lee, H. Consistency regularization for generative adversarial networks. 2020.
Zimmerer, D., Kohl, S. A., Petersen, J., Isensee, F., and Maier-Hein, K. H. Context-encoding variational autoencoder for unsupervised anomaly detection. arXiv preprint arX.v.1812.05941, 2018.
12


Reviews:
[{"review_id": "c4HGATW8ySw", "reviewer": "Reviewer_jSSg", "summary": "The paper  investigates a problematic behavior of current VAE models that lies in the inconsistency of current amortized encoder networks in the sense that the latent representations of semantically similar transformations (translation, rotation etc) of image data are dissimilar. Moreover, it proposes a tractable variant of the ELBO objective that is based on data augmentation and adds a KL-regularization term that penalizes the discrepancy in the posterior of the original and the transformed image data. The VAEs trained with this objective exhibit significant benefits (in terms of the utilization of the latent space and  marginal likelihood ) when compared to naive data augmentation demonstrating sota results in image generation and classification tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very well written and easy to follow. The new ELBO variant is intuitively justified and applicable to all exisiting VAEs. I really enjoyed reading this paper. I only have a few questions on the experiments that I feel they need to be clarified and some suggestions for further applicability of the consistency-driven ELBO variant.\n\n(1) Could the authors make the computational demands, especially on NVAE more clear? How many GPUSs are being used? It is stated that training takes 1 day approximately for all the experiments. However, this is inconsistent with the training time reported in the NVAE paper: (Table 6 in the appendix https://arxiv.org/abs/2007.03898) the training takes 55h so, given the augmentation I would consider that the consistent vae requires ~ 100 h?? Also, how many GPU-s are you using?\n\n(2) I also believe that especially for the deep VAEs which make use of very large latent spaces, including AU as you do in Table 1 as well as the KL divergence (how much higher is it compared to NVAE without regularization) would help clarify that the benefits come from the better use of the latent space.\n\n(3) How many samples (S in Algorithm 1) are you using when approximating the expected value?\n\n(4) I believe it would be really interesting (albeit not required for acceptance) if the authors demonstrate consistency on more sophisticated encoders, beyond Gaussians, such as those based on normalizing flows.\n\n(5)In evaluation metrics, it is mentioned \"we also evaluate the effect of the proposed generalization to unseen data\". I think it would be very informative if the authors also demonstrate another aspect of inconsistency in VAEs-- how well it would perform on data generated by its own generative model. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "X4maM9UDD9O", "reviewer": "Reviewer_aPrg", "summary": "The paper proposes a novel method of using data augmentation when training Variational Autoencoders. Given a family of semantics-preserving transformations, the idea is to enforce the invariance of latent code w.r.t. these transformations by adding a KL penalty term. This penalty improves the quality of learned latent representation and of the generative model significantly.\n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a novel method of using data augmentation when training Variational Autoencoders. Given a family of semantics-preserving transformations, the idea is to enforce the invariance of latent code w.r.t. these transformations by adding a penalty term. The penalty is KL divergence between encoder distribution at the augmented data point and encoder distribution at the original point.\n\nAuthors do extensive tests of the impact of this penalty on the quality of learned latent representations. Most experiments are performed on image datasets with augmentations being shifts, rotations and scaling. A variety of VAE architectures show improved quality of latents, compared to the standard way of applying augmentations. Quality of representations is accessed using mutual infromation estimates, active latent units and linear classifier accuracy (trained on latent features)\nAuthors also find that proposed method leads to better generative models with higher log-likelihood.\n\nThe proposed idea is original in the VAE context. The paper is well-written and experiments convincingly demonstrate the value of the proposed penalty.\n\nDue to it's simplicity and effectiveness the proposed method will likely be widely used in practice. \n\n*****************************\nPost-rebuttal\n\nI'd like to thank the authors for their response and will keep my already high score the same.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper could be further improved by considering different families of augmentations and analysing the impact of the augmentation choice on latent representation and log-likelihood.\n\nAnother interesting extension to the proposed method could be to assume equivariance of latent code rather then invariance, by parameterizing learnable(e.g. affine) transforms acting on z variable and enforcing the equivalence of  x -> z -> z' and x -> x' -> z'.\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "VNuhLFzRvl", "reviewer": "Reviewer_svud", "summary": "The paper improves VAEs by ensuring that the learned encoder maps augmented inputs $\\tilde{x}$ and unaugmented inputs $x$ to the same representations. To do so, they minimize the KL divergence between the encoder conditioned on $\\tilde{x}$ and $x$. They empirically show that such consistency regularised VAE (CR-VAE) achieves very strong density modeling results.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Overall**: This is a clear paper that provides a very simple and useful method to improve VAE’s. My main issue is that the authors motivate the work as a way to learn useful representations but they (1) do not appropriately discuss related work in representation learning; (2) do not provide empirical evidence that supports the usefulness of the CR-VAE's representations compared to standard baselines in representation learning. Overall, I still think that the proposed method will be useful and interesting to people that work with VAEs.\n\n**Strengths**\n- **Very simple method**: the proposed method is very simple, which means that it can easily be applied to any VAE in a few lines of code.\n- **Very clear paper**: the paper is very clear and well written.\n- **Strong results for density modeling**: the log-likelihood results on MNIST / CIFAR / Celeba are strong, and the gains are significant compared to only using data augmentations.  \n\n**Weaknesses**\n- **Missing related work**  this paper is essentially trying to learn representations that are invariant to standard augmentations. As a result, this makes the proposed method very similar to self-supervised learning (SSL). For example, the same regulariser already appeared in [1].  This decreases somewhat the novelty of CR-VAE as these ideas have been well investigated for representation learning. From my perspective, this paper uses ideas from SSL to improve VAE’s. This is not inherently an issue as long as SSL is correctly discussed (note that lines 119-121 won’t be true anymore). Specifically:\n    - [1]is very related to your method and uses the same regulariser. The key difference is that instead of reconstructing the image (learning p(x|z)) they use contrastive learning to maximize the mutual information I[Z, X]. This exemplifies well the relation between your method and SSL.\n    - Most SSL methods aim to make representations of an augmented and unaugmented example very similar. For example [2] does that by minimizing the variance instead of KL divergence (see their Appendix A for relation to KL divergence).  Similar ideas of learning approximately invariant representations go way back, e.g., [3] from 2006.\n    - [4] seems to be another very related work (although online since the beginning of March so contemporary). In particular, the Variational Invariant Compressor (sec. 4.1.) is essentially a VAE that takes as input augmented inputs but reconstructs unaugmented inputs to ensure “consistency” of the encoder. \n- **Lack of results showing the usefulness of the learned representations**: an important motivation of the work seems to be about learning useful representations, yet empirical results do not really support this claim:\n    - the experimental results of downstream linear classifications are extremely weak compared to typical SSL methods, which suggests that the learned representations of CR-VAE are not useful for downstream tasks. Indeed, the best CIFAR-10 accuracy in this paper is 71.4% while standard SSL with linear evaluation achieves more than 94% (e.g. SimCLR Appendix B.9.).\n    - The authors evaluate representation learning using mutual information, but (1) they do not compare to typical representation learning baselines; (2) it is well known that higher mutual information does not mean better representations [5].\n- **Somewhat incomplete experimental results**: Given that this is a completely empirical paper (based on sensible intuition) I find the experimental section to be somewhat incomplete. Here are results that I would have liked to see:\n    - (As previously said) a result that shows the usefulness of representations\n    - Lineplot showing for one dataset (e.g. CIFAR10) the effect of $\\lambda$ on the log-likelihood of CR-VAE, that would help to understand the robustness of the model to the additional hyperparameter.\n    - Experiment analyzing the impact of the choice of augmentations on CR-VAE, to understand the importance of the choice of augmentations.\n    - Standard errors for table 2/4/5/6. Especially table 5 as you claim SOTA.\n    - Qualitative samples from a CR-VAE for 2D images to understand the effect of the regulariser.\n\n**Suggestions / specific issues**\n- I’m not fond of your use of “consistency”, which means something very specific in (frequentist) statistics and the title could thus be understood as a method to ensure that the variational posterior converges to the true posterior (similar to [6]). I would at least consider using another word. I understand if the authors ultimately keep it given that it’s (unfortunately) the term used in semi-supervised learning.\n- The paper [7] is in the references but never actually cited in the text. Given that this paper shows comparable results on CIFAR10 and really shows the importance of data augmentation for density modeling, I think that it should be correctly discussed in your related work section.\n- Line 202 please explain in the text why more active units should be better\n\n**Questions**\n- Why did you choose the KL divergence in this direction? Please explain in the text, and ideally add an experimental comparison.\n- Another way of forcing “consistency” of the representations would be to augment the input but reconstruct the unaugmented examples (as in [4]), Have you considered this instead of your regulariser? I’d be curious to hear what you think, although I wouldn’t be surprised if “unaugmenting” the inputs is harder to do in practice.\n\n**Minor Suggestions**\n- Line 18: state of the art performance -> state of the art log-likelihoods          (I initially thought that SOTA performance was on downstream classification)\n- figure 1: I would add the sub captions for those that skim the paper: (a) VAE   (b) VAE with augmentations   (c) CR-VAE (ours)\n- Line 71 / 91 : e.g. -> e.g.,\n- Line 82: is unable -> may be unable\n- Line 148: will conduct -> conduct \n- Line 148 will apply -> apply\n- Line 185: Finally -> Finally,\n- Line 191: likelihood -> likelihoods\n- Equation 5 : KLs use different fonts\n- Line 198: metrics -> metric\n- Line 199/200: using different quotes on left and right of “active”\n\n**References**\n- [1] Federici, Marco, et al. \"Learning robust representations via multi-view information bottleneck.\" arXiv preprint arXiv:2002.07017 (2020).\n- [2] Zbontar, Jure, et al. \"Barlow twins: self-supervised learning via redundancy reduction.\" arXiv preprint arXiv:2103.03230 (2021).\n- [3] Hadsell, Raia, Sumit Chopra, and Yann LeCun. \"Dimensionality reduction by learning an invariant mapping.\" 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06). Vol. 2. IEEE, 2006.\n- [4] Dubois, Yann, et al. \"Lossy Compression for Lossless Prediction.\" arXiv preprint arXiv:2106.10800 (2021).\n- [5] Tschannen, Michael, et al. \"On mutual information maximization for representation learning.\" arXiv preprint arXiv:1907.13625 (2019).\n- [6] Wang, Yixin, and David M. Blei. \"Frequentist consistency of variational Bayes.\" Journal of the American Statistical Association 114.527 (2019): 1147-1161.\n- [7] Jun, Heewoo, et al. \"Distribution augmentation for generative modeling.\" International Conference on Machine Learning. PMLR, 2020.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As previously stated, the authors need to discuss the limitation of CR-VAE's representation compared to standard representation learning methods. \n\nPotential societal impact is adequately discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "Tw9mjGgHdvE", "reviewer": "Reviewer_69Xg", "summary": "This paper proposes a study on enforcing semantic consistency on VAEs. The authors show that the current VAEs are sensitive to semantical-preserving transformations and it will lower the quality of representations and generalization. To solve this issue, the authors propose a new algorithm, which first transforms a training sample into multiple semantically similar samples and then minimize the KL-divergence between the encoded distributions of original and new samples. Experiments on both 2D and 3D datasets show that the new algorithm is effective in increasing representation quality of VAEs.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. This paper discusses a very interesting problem, how to make representations semantically invariant.\n2. The method is simple and can be applied to all variants of VAEs.\n3. Experiments show the proposed algorithm is useful in a wide range of datasets.\n\nCons:\n1. This paper looks like a direct application of contrastive learning on VAEs. Though the loss function is new,  the idea, algorithm, and even transformations are very similar to works on contrastive learning. The authors mention CAEs in line 130. However, more advanced contrastive methods are not compared as baselines.\n2. Some experiment results are not significant. \n  (a) In Table2, the accuracy increase of using CR on MNIST is marginal. \n  (b) In Table 3, the differences of estimated log-likelihoods are also small in most settings.\n3. The choice of some parameters is not verified. \n  (a) The most important parameter \\lambda. I cannot figure out how to choose it. The authors say that ablation studies are performed, but I still cannot find the results.\n  (b) The transformations (including translation, rotation, and scaling) and their parameters fully depend on handcrafting. Though it is a common problem for contrative learning, the authors should think about how to do it more automatically.\n\nOther comments:\n1. A large portion of this paper is devoted to introducing variational autoencoders and other background knowledge. For example, the first several paragraphs in the introduction and method part. The authors should talk more about their own staff with a very brief introduction of necessary backgrounds.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors mention that this paper might be used to generate artificial data or introduce biases. However, I think these are common problems for all deep generative models. So I don't think the authors need to improve this specific paper to solve the potential negative impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach to improving the representations of generalization of VAEs using semantic-preserving transformations and a regularizer that enforces a congruence between representations of the original and transformed inputs.  This is shown to provide notable performance improvements for a variety of VAE models, with these gains, critically, larger than those achieved using simple data augmentation.

Overall, the reviewers were quite positive about the work, praising its simplicity, clarity, and empirical results.  Some issues were raised about the fact that most aspects of the paper are relatively straightforward applications of previous techniques, with the novelty predominantly originating only from their use in a VAE context, and insufficient discussion and comparison to other representation learning approaches that utilize semantic-preserving transformations, such as contrastive learning approaches.  However, all reviewers felt that the paper still deserved to be accepted in spite of these (albeit in some cases only marginally).

I believe that this assessment by the reviewers is quite fair and that, though not the most spectacular of papers, it is good solid work that will be of interest to the community and likely see some practical usage.  I thus recommend accepting it to the conference.

That said, I do think it is important that the authors make sure to make updates and improvements to the work, as per the reviewer's suggestions, for the camera-ready version.  In particular, I think the suggested additional discussion of related work is very important to add.  I would also strongly encourage the authors to include some non-VAE self-supervised baselines: though I mostly buy the argument that VAEs are important in their own right and do not think beating these baselines should be a condition for the work to be acceptable, I think it is still important to quantify how the approach compares to natural alternatives practitioners might consider (e.g. to see if there are aspects that this VAE-based approach is relatively good and bad at and how far off they are to SOTA approaches in the area).  Specifically, given much of the utility of the work is empirical, I feel that it is important that it properly investigates how relatively well VAE based approaches can perform in the target setting where semantically equivalent inputs can be generated.
Decision: Accept (Poster)