AAVAE: Augmentation-Augmented Variational Autoencoders
Authors: William Alejandro Falcon
Ananya Harsh Jha
Teddy Koker
Kyunghyun Cho
Venue: ICLR 2022 Submitted (2022)
Keywords: Self-Supervised Learning
Autoencoders
Variational Autoencoders
Data Augmentation

Abstract: Recent methods for self-supervised learning can be grouped into two paradigms: contrastive and non-contrastive approaches. Their success can largely be attributed to data augmentation pipelines which generate multiple views of a single input that preserve the underlying semantics. In this work, we introduce augmentation-augmented variational autoencoders (AAVAE), yet another alternative to self-supervised learning, based on autoencoding. We derive AAVAE starting from the conventional variational autoencoder (VAE), by replacing the KL divergence regularization, which is agnostic to the input domain, with data augmentations that explicitly encourage the internal representations to encode domain-specific invariances and equivariances. We empirically evaluate the proposed AAVAE on image classification, similar to how recent contrastive and non-contrastive learning algorithms have been evaluated. Our experiments confirm the effectiveness of data augmentation as a replacement for KL divergence regularization. The AAVAE outperforms the VAE by 30% on CIFAR-10, 40% on STL-10 and 45% on Imagenet. On CIFAR-10 and STL-10, the results for AAVAE are largely comparable to the state-of-the-art algorithms for self-supervised learning.

Under review as a conference paper at ICLR 2022
AAVAE: AUGMENTATION--AUGEENTNR VARIATIONAL AUTOENCODERS
Anonymous authors
Paper under double-blind review
ABSTRACT
Recent methods for self-supervised learning be grouped into two paradigms: contrastive and non-contrastive approaches. Their success can largely be at- tributed data augmentation pipelines which generate multiple views of sin- gle imput that preserve the underlying semantics. In this work, we introduce augmentation -ugmeented variational autoencoders (AAVAE), yet another alter- native to self-supervised learning, based on autoencoding. We derive AAVAE starting from the conventional variational autoencoder (VAE), by replacing the KI. divergence regularization, which is agnostic to the input domain, with data augmentations that explicitly encourage the internal representations to encode domain-specific invariances and equivariances. We empirically evaluate the pro- posed AAVAE on image classification, similar to how recent contrastive and non- contrastive learning algorithms have been evaluated. Our experiments confirm the effectiveness of data augmentation as a replacement for KL divergence regulariza- tion. The AAVAE outperforms the VAE by 30% on CIFAR-10, 10, 40% STL-10 and 45% on Imagenet. On CIFAR-10 and STI,-10, the results for AAVAE are largely comparable to the state-of-the-an algorithms for self-supervised learning.
INTRODUCTION
The goal of self-supervised learning (SSL) (DeSa 1993) is to learn good representations from un- labeled examples. good representation often defined as the one that reflects underlying class structures well. The quality of representation obtained from SSL is evaluated by measuring down- stream classification accuracy on a labelled dataset. recent years, two families of approaches have emerged the state-of-the-ee art for SSL: contrastive and non-contrastive learning.
At its core, a contrastive learning algorithm stochastically creates two views from each training example, called positive and anchor examples, and selects one of the other training examples negative (Bachman et al. 2019 Tian et al.) 2019; Chen et al. 2020b a). The positive and anchor examples are brought closer in the representation space, while the negative example is pushed away from the anchor. This definition of contrastive loss brings in two interconnected issues. First, there no principled way to choose negative examples, and hence these negatives are chosen somewhat arbitrarily each time. Second, the contrastive loss function is not decomposed over training examples because negative examples come from other training examples.
Partly to address these limitations, recent studies have proposed non-contrastive approaches that have removed the need for negative examples (Grill et al. 2020 Caron et al., 2020 Zbontar al. 2021]. These approaches avoid the necessity of explicit negatives by constraining dataset-level statisties of internal representation (Zbontar al.) 2021 Caron et al. 2021 Genevay et al. 2018). Dataset-level statistics, which are intractable to compute, are instead approximated using a minibatch of training examples. This often results the need of large minibatches. Also, the use of batch-level statistics means that non-contrastive losses not decomposable well.
regularizing
Despite the apparent differences between these two families of algorithms, they all recognize the im- portance of and rely heavily on data augmentation as a way of incorporating domain knowledge. For instance, Chen et al. (2000a) have highlighted that the downstream accuracy after finetuning varied between 2.6% and 69.3% on ImageNet (Deng et al. 2009), depending the choice of data augmen- tation. This is perhaps unsurprising since the importance domain knowledge has been reported in various domains beyond computer vision. In reinforcement learning, Kostrikov al. (2020) and

---

Under review as a conference paper at ICLR 2022
Raileanu et al. (2020) have shown the benefit of adding domain information via pixel-level data augmentation in continuous control. In natural language processing, Ng et al. (2020) demonstrate the effectiveness of domain-specific augmentation by using a pretrained denoising autoencoder to build a robust classifier.
A variational autoencoder (VAE) implements latent variable model using a composition of two neural networks. A neural net decoder maps a latent variable configuration to an observation, and neural net encoder approximately infers the latent variable configuration given the observation Kingma & Welling 2013) It is often trained to maximize the variational lowerbound or its vari- ant (Kingma & Welling 2013 Higgins et al. 2016). Careful inspection of this learning objective shows two parts: autoencoding and latent-space regularization. Autoencoding ensures that there an approximately one-to-one mapping between individual inputs and internal representations. This prevents the collapse of internal representations onto a single point, similar to what negative examples contrastive learning and regularization of batch-level statistics in non-contrastive learn- ing do. Latent-space regularization, arranged semantically a compact subset of the space. It is often done by minimizing the KL divergence (Kullback & Leibler 195I) from the approximate posterior, returned by the encoder, to the prior distribution and adding noise to the representation during training (i.e., sampling from the approximate posterior). This performs role similar to that of data augmentation in contrastive and non-contrastive approaches but is different in a way that it is agnostic to the input domain.
the other hand, ensures that the internal representation is
Based on these observations: (1) the importance of data augmentations and (2) variational autoen- coders for representation learning, we propose a third family of self-supervised learning algorithms in which we augment variational autoencoders with data augmentation. We refer this family of models as cetteeettationAuumenttee Variational Autoencoders (AAVAE). In AAVAEs, we replace the usual KL-divergence (Kullback Leibler 195T) term in ELBO (Kingma & Welling, 2013) with a denoising criterion (Vincent et [ll]] 2010) based on domain-spection We hypothesize that this new approach allows the representations learned by AAVAFs to encode domain-specific data invariances and equivariances. The resulting model offers few advantages over the existing contrastive and non-contrastive methods. First, the loss function is not dependent on the batch-level statistics, which suspect enables to use smaller minibatches. Second, the AAVAE does not necessitate an arbitrary choice of negative sampling strategy.
data augmentation.
We pretrain AAVAEs image datasets: CIFAR-10 (Krizhevsky & Hinton 2009), STL-10 (Coates jet al., 2011) and Imagenet (Deng et al.] 2009), and as is the norm with other recently proposed approaches (Goyal et al. |019 fication tasks corresponding to the dataset using a single linear layer without propagating gradients back to the encoder. We find that our autoencoding- based method gives downstream classifica- tion performance comparable to the current state-of-the-ar SSL methods, with 87.14% accuracy on CIFAR-10 and 84.72% on STL-10. On Imagenet, the AAVAE outperforms the carefully crafted pretext tasks for SSL, such as Colorization (Zhang et al. 2016), Jigsaw (Noroozi & Favaro 2016) and Rotation (Gidaris al., 2018). demonstrating that designing such complex pretext tasks is un- necessary. As anticipated from our formulation, representation learned by the AAVAE is robust to the choice of hyperparameters, including minibatch size, latent space dimension, and the network architecture of the decoder. Our observations strongly suggest that autoencoding is viable third family of self-supervised learning approach in addition to contrastive and non-contrastive learning.
al. 2020a Caron et al. 2020), we evaluate them classi-
SELF-SUPERVISED LEARNING
Self-supervised learning (SSL) aims to derive training signal from the implicit structure present within data (DeSa }993). This enables SSL methods leverage large unlabeled datasets to learn representations (Goyal et al. 2021) which can then be used to solve downstream tasks, such as clas- sification and segmentation, for which it is often expensive to collect a large number of annotations. Here, we summarize quite few variations of this approach proposed over the last few years.
2.1 PRETEXT TASKS
Pretext tasks are designed to train a neural network to predict a non-trivial but easily applicable transformation applied to the input. For example, Gidaris al.) (018) randomly rotate an input
2

---

Under review as a conference paper at ICLR 2022
image by 09, -9000 180". or 270 and train network to predict the angle of rotation. The colorization pretext task (Zhang et al. 2016) creates a training signal by converting RGB images to grayseale and training network to restore the removed color channels. Image inpainting (Pathak et al. 2016) learns representations by training an encoder-decoder network to fill in artificially-occludee parts of an image. Both jigsaw (Noroozi & Favaro 2016) and relative patch prediction (Doersch al.) 2015) tasks divide an imput image into patches. The jigsaw task (Noroozi & Favaro 2016) shuffies the spatial ordering of these patches and trains a network predict the correct order. In contrast, relative patch prediction (Doersch et al. 2013) selects two patches of an image and asks the network to predict their relative spatial positions. More recently. Doersch & Zisserman (2017) combined various pretext tasks into a single method. [ooyal et al. (2019) have, however, shown that training neural network backbones using pretext tasks often does not capture representations invariant to pixel-space perturbations. Consequently, these representations perform poorly on downstream tasks while they solve the original pretext task well.
2.2 CONTRASTIVE LEARNING
Between the two major families of state-of-the-art methods for self-supervised learning, we discuss the one based on the so-called contrastive loss function (Hadsell et al.) 2006). The contrastive loss is defined such that when minimized, the representations of similar input points are pulled towards each other, while those of dissimilar input points are pushed away from each other. The contrastive loss has its roots in linear discriminant analysis (Fisher 1936) and is closely related to the triplet loss (Weinberger et al., 2006). Recent approaches in contrastive learning are characterized by the InfoNCE loss proposed by pord et al. (2018). CPC uses InfoNCE as a lower bound of mutual infor- mation (MI) and maximizes this lowerbound, by using negative examples. Deep InfoMax (Hjelm) et al. 2018) similarly proposes to use the idea of maximizing MI while considering global and lo- cal representations of an image. [Hjelm et al. (2018) tested three bounds on MI: Donsker- Varadhan (Donsker & Varadhan 1983.). Trnsen-Shannon (Nowozin et al.) 2016). and InfoNCE (Oord et al.) 2018). and found that the InfoNCE objective resulted in the best downstream classification accu- racies. Since then, several more advances in contrastive self-supervised learning have happened, such as AMDIM (Bachman et al. 2019) and CMC (Tian et al.] 2019). both of which focus on using multiple views of each image. Hénaff al. (2019) extend CPC with an image patch prediction task, and YADIM (Falcon & Cho. (220)) combines these ideas of augmentation and InfoNCE loss from both CPCv2 (Hénaff et 2019) and AMDIM (Bachman et al. 2019) under a single framework.
The success of contrastive learning comes from using a large number of negative examples. Misra & van der Maaten (2020) empirically demonstrate with PIRL the benefits of using large number of negative examples for downstream task performance. PIRL uses momentum-updated memory bank (Wu et al.] 2018) to provide this large cache of negatives. Memory bank models (Wu et al., 2018 Misra & van der Maaten 2020) need to store and update representations for each data point and hence cannot be scaled up efficiently. To remove the dependence on memory bank, MoCo (He) eet al. 22000: Chen et al. 2020b) instead introduces momentum momentum-updatte encoder and compara- tively smaller queue of representations to set up positive and negative pairs for contrastive learning. SimCLR (Chen et al. (020a) removes memory banks and momentum-updaatt up the batch size to provide a large number of negatives from within each mini-batch. The necessity of large quantity of negatives for the contrastive loss function work well proves to be major challenge in scaling up these methods.
encoders and scales
2.3 NON-CONTRASTIVE APPROACHES
The second family consists of non-contrastive learning algorithms that aim to learn good represen-
tations without negative samples by relying on data-level or batch-level statistics. These algorithms
can be classified into two groups: clustering-basee
(Caron et al.) 2018 Asano et al.
2019
Caron
ctt al. 2020) and distillation-
(Grill et al.) [2020] Chen & He 2020)
Gidaris et al.
2020
Caron 2021
et al.
2021) approaches. A more recently proposed method Barlow Twins (Zbontar al.
does not fall under either group.
Clustering based methods, such as DeepCluster (Caron al.) 2018), generate pseudo-labels for training examples by grouping them in the latent space of a neural network. The lseudo-labels then used to train the neural network. These two steps are repeated several times. Like any
3

---

Under review as a conference paper at ICLR 2022
(=/===
posterior
Encoder
Decoder
x² A(a)
Ey(x*)
Dy(#)
# r (al22
Figure 1: AAVAE: The input to the model is an augmented view of x+ ~ A(z), the target is the original input æ. The loss the reconstruction term of the ELBO (Eq. without the KL--divergence
classical clustering algorithm, such k-means, this approach exhibits degenerate solutions and requires additional regularization to avoid these solutions. One such degenerate solution is to put all examples into single cluster. SeLA (Asano al. 2019) regularizes the clustering process with the Sinkhorn-Knopp algorithm (Cuturi) 2013), encouraging training examples to be equally distributed across the clusters. Caron al. (2020) extend this approach use data augmentations and online soft assignments of training examples.
Instead of clustering examples, distillation-basee approaches (Tarvainen & Valpola, 2017) rely on having separate neural network called teacher network to provide a student network with target class for each training example. Similar clustering- caaseriin-bbssd approaches above, this strategy also exhibits trivial solutions, such as the teacher and student networks being constant functions without proper regularization. BYO (Grill et al. 2020 Richemond et al. 2020 Tian et al. 2020b), and its simpler variant called SimSIAM (Chen & He (020), rely asymmetry in the network architecture between the teacher and student to avoid such degeneracy. To simplify things, SimSIAM (Chen) & He 2020) goes one step further than BYOL (Grill et al. 2020) and removes the momentum- based updates for the teacher network. On the other hand, DINO (Caron et al. 2021) retains the nomentum-based updates for the teacher network, replaces the architectural asymmetry with tering of representations of examples within each minibatch, and demonstrates that these techniques combined with tempered softmax are sufficient regularizers avoid degeneracy.
Barlow Twins (bbontar et al. 2021 stands out as an alternative to these two families of approaches. mixes three principles; (1) batch-level statistics, (2) data augmentation, and (3) whitening (redun- dancy reduction). At each update, Barlow Twins (Zbontar et al., (021) normalizes the representa- tions of the training examples within each minibatch to have zero-mean and unit-variance along each dimension. It then tries to maximize the cosine similarity between the representation vectors coming of a pair of samples drawn from stochastic data augmentation pipeline applied over a single training example. Finally, Barlow Twins (Zbontar et al. 2021 minimizes the cross-correlation be- tween different coordinates of these vector representations, which amounts to reducing redundancy at the second-order moment. A similar approach has also been proposed by Bardes et al. (2021).
AUGMENTATION-
VARIATIONAL AUTOENCODERS
Here we revive the idea of autoencoding as a third paradigm for self-supervised learning, in addition contrastive and non-contrastive learning, which are described in the previous section. In partieu- lar, we start from variational autoencoders (VAEs) (Kingma & Welling 2013) to build new self- supervised learning algorithm for representation learning. There are three mechanisms by which VAE captures good representations of data; (1) autoencoding, (2) sampling at the intermediate layer, and (3) minimizing KL divergence (Kullback & Leibler 1951) from the approximate posterior to the prior distribution, all of which are largely domain-agnotter We thus introduce domain-speciff knowledge by replacing the first mechanism (autoencoding) with denoising (Vincent et al.) 2008 2010) via data augmentation. Furthermore, remove the third mechanism expect KI. di-
4

---

Under review as a conference paper at ICLR 2022
vergence minimization be redundant in representation learning. In this section, explain the original VAE and then carefully describe our proposal of Sugmentation-auggmenn augmented VAE.
3.1 TRAINING A VAE WITH THE EVIDENCE LOWERBOUND (ELBO)
We describe algorithms in this section with the assumption that are working with images, as has been often done with recent work in self-supervised learning (Oord et al. 2018 Hjelm et al. 2018 Chen al. 2020a). Hence, let the input be an image, where Rex with c color channels of height & and width w. The VAE then uses a continuous latent variable 2 E R to map the high dimensional input distribution, as p(x) J. p(=)=)p(=)dz.
It is however intractable to marginalize in general, and instead we use tractable lowerbound to the average pr--babbbiltty of the training examples. Let qa (z|z) be an approximate posterior distribu- tion to the intractable distribution p(z[zr, parametrized by the output of the encoder (z))). (xl2))
a probability distribution over the imput x, parametrized by the output of the decoder (z). The variational lowerbound (ELBO) (Kingma & Welling 2013) to the log-marginal probability logp(x)
logp(x) 2 E(x)= (e [logpw(r]) + 3 (log p(=) log go (z(x)))
(I)
The VAE is then trained by minimizing
Jv
Σ ((x"),
where x is the 12-th training example.
The first term in Eq. serves two purposes. First, it minimizes the reconstruction error, which en- courages the intermediate representation of the VAE to be more or less unique for each observation. In other words, ensures that the internal representations of the inputs do not collapse onto each other. The second purpose, expressed as the expectation over the approximate posterior, is to make the representation space smooth by ensuring small perturbation the representation does not alter the decoded observation dramatically.
The second term, the KL divergence (Kullback Leibler 195I) from the approximate posterior to the prior, serves single purpose. It ensures that the representation of any observation under the data distribution is highly likely under the prior distribution. The prior distribution is often constructed to be standard Normal, implying that the probability mass is highly concentrated near the origin (though not necessarily on the origin). This ensures that the representations from observations are tightly arranged according their semantics, without relying any domain knowledge.
3.2 AUGMENTATION-AUGEENTT
VARIATIONAL AUTOENCODER
The AAVAE removes the KL divergence Kullback & Leibler 1951) from the formulation because it does not embed domain-specifie information and replaces in favor of an augmented view of the original example. Mathematically. this proposed replacement results in the following loss function:
1 JAAVAE v)) NN
Σ
E xt A(xa)
[ Besggoen at) (x
where = (a1, 02. a. is stochastic process that applies sequence of stochastic input trans- formations an A transforms any input r to generate a view x ~ A(x), while preserving the major semantic characteristics of
The proposed replacement effectively works by forcing the encoder of the AAVAE to put repre- sentations of different views of each example close to each other since the original example must
5

---

Under review as a conference paper at ICLR 2022
be reconstructed from all of them. This is unlike the original KL divergence term, which packs the representations globally into the prior. other words, we replace this global packing with the local packing, where the domain-specifi transformations define the local neighborhood. Further- more, domain-aware transformations have the effect of filling in the gaps hetween training examples, which indirectly achieves the goal of global packing.
Comparison existing approaches Compared to the existing approaches, both contrastive and non-contrastive ones, the AAVAE has unique advantage. AAVAE's loss function is decomposed over the examples, which avoids the need of approximating data-level statistics and computing its gradient for learning. This is advantageous, because we know precisely what are computing when we use a small minibatch approximate the gradient of the whole loss function. Generally, this is not the case with algorithms where need to approximate the gradient of data-level statistics using a small mini-batch. Based on this observation, we expect our approach to be robust to minibatch size, which we later confirm experimentally in the paper.
relatively minor but related advantage the proposed approach over constrastive learning is that there is need to design strategy for selecting negatives for each training example. Considering a flurry of recent work reporting on the importance of mining better negative examples (Tian et al., 2020a Chuang al., 2020 Robinson et al. (021), our approach based autoencoding greatly simplifies self-supervised learning by entirely eliminating negative examples.
EXPERIMENTS
In this section, we talk about the experiments done for training and evaluating AAVAEs. The design and setup of these experiments are discussed in detail in Appendix A.1
4.1 QUALITY REPRESENTATION DOWNSTREAM CLASSIFICATION ACCURACIES
First, look at the accuracies from variants of autoencoders, the family to which the proposed AAVAE belongs, presented the bottom half of Table [ (left). We consider the vanilla autoen- coder (AE), augmention-auggennte autoencoder (AAAE), and the variational autoencoder (VAE) as baselines. Our first observation is that there is a significant gap between the proposed AAVAE and all the baselines, with up to 30% points on CIFAR-10, 40% points on STL-10, and 45% points on Imagenet. This demonstrates the importance of data augmentation and noise in the intermediate representation space in making autoencoding competitive alternative for self-supervised learning- When we add only one of these components, augmentation in the case of AAAES or sampling in the case of VAEs, we see a big performance degradation from AAVAE. The gap between VAE and AAVAE exposes the inadequacy of KL-divergence regularizer for the latent space.
Table 1: Classification performance of Resnet-50 (He et al., 2016 backbone CIFAR-10
Krizhevsky & Hinton| 2009), STL-10 (Coates et al. 2011) and Imagenet (Deng et al. 2009 across different methods. All models were pretrained on the corresponding dataset without labels and finetuned using the protocol described in SimCLR (Chen et al.) 2020a). The autoencoder trained with denoising criterion (AAVAE) outperforms the baseline VAE by 30% CIFAR-10, 40%
on STL-10 and 45% on Imagenet. Methods marked with & either use different backbone than
Resnet-50 different (non-linear) evaluation strategy.
Method
CIFAR-10
STL-10
Imagenet
Method
Imagenet
+ CPC (large)
48.7
Colorization
39.6
CPCv2
84.52
78.36
63.8
Rotation
48.9
AMDIM (small)
92.10
91.50
63.5
Jigsaw
45.7
YADIM
91.30
92.15
59.19
BigBiGAN
56.6
SIMCLR
94.00
92.36
69.3
NPID
54.0
AE
56.34
42.26
0.89
MoCo
60.6
AAAE
50.62
41.94
1.29
SWAV
75.3
VAE
57.16
44.15
4.58
BYOL
74.3
AAVAE
87.14
84.72
51.0
Barlow Twins
73.2

---

Under review as a conference paper at ICLR 2022
42
0.52
376e3
08
076
074
30m0
1204
300.
6006
30H4
400
2009
3001
100%
000
602
402
60Ck
val. log pó41))
online finetuning
log podi))
online fineturing
(a) learned distribution width (logscale)
(b) fixed scalar distribution width (logscale)
Figure 2: The AAVAR uses a Gaussian likelihood on pixels for the reconstruction loss with a speci- fied width of the distribution (logscale). In (a), we let the decoder learn the logscale and observe the illusion of overfitting as mentioned in Mattei & Frellsen 2018). J. In (b), we fix the logscale parameter to an arbitrary scalar by sampling uniformly between [-5, 2]. In both cases, we fail to observe any correlation between the quality of density estimation and learned representation. Plots shown for CIFAR-10 Krizhevsky & Hinton, 2009) dataset.
We then put the performance of the proposed AAVAE in the context of existing self- sulf-super learning algorithms presented in the top half of Table [] (left), and Table [ (right). We confirm once again what others have observed All three autoencoder baselines (AF, AAAF, and VAE) severely lag behind the other state-of-the- art self-supervised learning approaches. However, the proposed modification that led AAVAE significantly narrows this gap on CIFAR-I0 and STL-10. On Imagenet, the AAVAE lags behind the current crop of state-of-the-arr methods; however, it performs better than any existing pretext task designed for SSL. These results suggest that autoencoding is viable alternative to contrastive and non-contrastive learning algorithms when designed and equipped appropriately and developed further on from here.
why autoencoding fell out of interest in recent years.
4.2 REPRESENTATIONAL QUALITY DOES NOT DETERIORATE
major downside of the proposed strategy of replacing the KL divergence term in the original loss with data augmentation is that we lose the interpretation of the negative loss as the lowerbound to the log probability of an observation. However, we find less concerning as the quality of representation is not necessarily equivalent to the quality of density estimation. Furthermore, we make a strong conjecture that the representation quality, which largely depends on the encoder, does not suffer from overfitting (in terms of downstream classification accuracy), even when the quality of density estimation does. Our conjecture comes from the observations that the representation output of the encoder must cope with multiple copies of the same input and noise added in the process of sampling. On the other hand, the decoder can arbitrarily shrink the width of the output distribution per latent configuration, resulting in overfitting to training examples. This conjecture is important since it implies that we should train the AAVAE as long as the computational budget allows, rather than introducing sophisticated early stopping criterion. More importantly, this would also imply that we do not need to assume the availability of downstream tasks at the time of pretraining.
We test two setups. First, we let the decoder determine the width (in terms of the diagonal covariance of Gaussian) on its own. In this case, we expect the model to overfit the training examples severely, as was observed and argued by Mattei & Frellsen (2018). while the representation quality never deteriorates. In the second setup, we fix the width to an arbitrary but reasonable scalar, which would prevent overfitting in the context density estimation as long as it is chosen to be reasonably large.
As presented in Fig- in both cases, we observe that the quality of representation, measured in terms of the downstream accuracy, does not deteriorate. Furthermore, as anticipated, we observe that the quality of density estimation quickly overfits in learning the width of output distribution (Figure 2 (a)). Fixing the width to a scalar did not necessarily help avoid the issue of overlitting (Figure 2 (b)). Still, more importantly, we fail to observe any clear relationship between the qualities of density estimation and learned representation.
This finding suggests the need for further study to define and measure the quality of representation distinct from both density estimation quality and downstream accuracy. The former will not only help us measure the learning progress in pretraining time, but will also shed light what we mean
7

---

Under review as a conference paper at ICLR 2022
67.9
0.001
0.1
3 couffieent
Figure 3: Downstream classification accuracy on CIFAR-I0 (Krizhevsky Hinton 2009 when we add back KL divergence based regularization with 8-coefficient (Higgins et al. 2016 to the loss function of AAVAE defined in Eq. We observe a negligible change in the quality of rep- resentations, as measured by the classification task, when the KL-term is weighted with a B < 1. For values of 2 1, the quality representation starts deteriorating, as seen by the decrease in classification accuracy.
by representation and representation learning. The latter will be needed for future downstream tasks, as the main promise of pretraining is that results representations that are useful in the unknown.
4.3 COMBINING VAE AND AAVAE
Although we designed AAVAE by replacing the KL divergence based regularization with data aug- mentation based denoising. these two may well be used together. Earlier, Im Im et al. (2017) studied this combination with a simple corruption distribution that is agnostic to the input domain in the context of density estimation. Here, we investigate this combination, with -ppeii- transfor- mations, in the context of representation quality.
While keeping the data augmentation based perturbation scheme intact, vary the coefficient of the KL divergence term. When 3 = 0, it is equivalent to the proposed AAVAE. We present the downstream classification accuracies on CIFAR-10 in Figure
We first observe that the KL divergence term has negligible impact when the coefficient is small, i.e., B & However, as B grows, we notice a significant drop the downstream classification accuracy, which we view as proxy to the representation quality. We attribute this behavior to the tension, or balance, between domain-aware and domain-agnostic regularization the representation space. As 3 + the domain-agnosti regularization overtakes and results in the arrangement of the representations that does not reflect the domain -speciffc structures, leading to worse downstream classification accuracy.
From this experiment, we conclude that for self-supervised pretraining, the proposed approach of data augmentation is better way to shape the representation space than the donain-agnostic KL divergence based regularization.
4.4 HYPERPARAMETES SENSITIVITY
The proposed AAVAE, or even the original VAE, sets itself apart from the recently proposed self- supervised learning methods in that its loss function is decomposed over the training examples (within each minibatch.) Thus, we believe that training the AAVAE is less sensitive to minibatch size, as even with a single-example minibatch, our estimate the gradient unbiased. This is often not guaranteed for a loss function that is not decomposed over the training examples. We test this hypothesis by rumning experiments with varying sizes of minibatches.
As shown in Fig. (a), observe almost no difference across different minibatch sizes, spanning
from 128 to 1024. This is true for both the downstream accuracy (representation quality) and the speed of learning. This is contrary recent findings from self- self-supervised learning algorithms, where large minibatches have been identified as an important ingredient (Chen et al. 2020a Tian et al. 2020b).
This insensitivity to the minibatch size raises a question about other hyperparameters, such as the dimensionality of latent space (Fig. (b)), the decoder architecture (Fig. (c)) and the logscale or
8

---

Under review as a conference paper at ICLR 2022
(4) batch size
(b) Iatent dim
[c] decoder arch
(6) logacall
20G
1224
120
29G
dec54
decS0
+42214
-0.0es -1.332
1.3103
46.5
Epoch
Epoch
Epoch
Epoch
Figure 4: On CIFAR-I0 10 Krizhevsky & Hinton 2009). we demonstrate AAVAES insensitivity to hy- perparameters: (a) batch size, (b) latent space dimension, (c) decoder architecture, and (d) logscale parameter (width of the Gaussian likelihood). We vary one specific hyperparameter while keep- ing the rest fixed for these insensitivity ablations. We select the minibatch size between 1 128-1024, the dimensionality of the latent space between 64-512, the decoder architecture from decoders that mirror (resnet18, 18, resnet34 resnet50) encoders, and sample the logscale values from uniform distribution between [-5, 2].
width of the output distribution (Fig. (d)). We test the sensitivity of the proposed AAVAE each of these hyperparameters. We find that the quality of representation, measured by the downstream classification accuracy, is largely constant to the change these hyperparameters. Together with the insensitivity to the minibatch size, this finding further supports our claim that autoencoding- basse approaches form valuable addition to supe-vupe learning.
CONCLUSION
In this paper, we attempt to revive the idea of autoencoding for self-supervised learning of repre- sentations. We start by observing that data augmentation is at the core of all recently successful self-supervised learning algorithms, including both contrastive and non-contrastive approaches. We then identify the KL divergence in variational autoencoders (VAE) shaping the representation space and hypothesize that this makes it inadequate for representation learning. Based on these two observations: the importance of data augmentations and KL diver- gence's inadequacy, we propose replacing the KI divergence regularizer with denoising criterion and domain-specific data augmentations in the VAE and call this variant augmentation-augmeentd VAE (AAVAF).
domain-agnostic way of
Our experiments reveal that the AAVAE learns substantially better data representation than the orig- inal VAE or any other conventional variant, including the vanilla autoencoder and the augmentation- augmented denoising autoencoder. We use downstream classification accuracy from finetuning linear layer as the metric to measure representation quality and observe more than a 30% improve- ment on all datasets over the VAE. This result is better than any pretext task for SSL and one of the earlier versions of contrastive learning, CPC. Although the AAVAE still lags behind the more recent methods for SSL., this gap is significantly narrower with the AAVAE than with any other autoencoding variant.
One consequence of autoencoding is that the loss function of AAVAE is decomposed over the ex- amples within each minibatch, unlike contrastive learning (with negative examples from the same minibatch) and non-contrastive learning (which often relies on minibatch statistics). We anticipated that this makes AAVAE learning less sensitive to various hyperparameters, especially the minibatch size. Our experiments reveal that the AAVAE is indeed insensitive to the minibatch size, latent space dimension, and decoder architecture.
Although the proposed AAVAE has failed to outperform or perform comparably to the existing fam- ilies of self self-super learning algorithms, our experiments indicate the potential for the third cat- egory of self-supervised learning algorithm based on autoencoding. The quality of representations can be significantly pushed beyond that of the vanilla autoencoder and variational autoencoder by making them encode domain specific invariances. Furthermore. aattoencoding-bssee methods, rep- resented by the AAVAE, are robust to the choice of hyperparameters. Based on these observations, we ad advocate for further research in the direction of autocncoding- based superviee learning.

---

Under review as a conference paper at ICLR 2022
6 REPRODUCIBILITY STATEMENT
As mentioned in Section 4.4 of the main paper, our proposed model AAVAE insensitive to batch size, dimensionality of the latent space, decoder architecture and width of the output distribution (logscale parameter). This in itself makes replicating results for AAVAES lot easier across datasets than models that have highly fine-tuned hyperparameters. Then, Appendix [A.] we list out the encoder/decoder architecture, datasets, augmentation pipeline, optimizer hyperparameters, evalua- tion protocols etc. that have been used for training AAVAEs and other baseline autoencoder models. Finally, the code used to replicate results from our paper is anonymized and uploaded to the link: http://ggitab..comaaaaaaccc/2222
REFERENCES
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultancous clustering and representation learning. CoRR, abs/1911.05371, 2019. URL http:://rxxxi
org/abs/191 911 05371
Philip Bachman, R. Devon Hjelm, and William Buchwalter. Learning representations by maximiz- ing mutual information across views. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Informa- fion Processing Systems 32: Annual Conference Neural Information P Processing Systems 2019, NeurIPS 2019, December 8- 841, 2019, Vancouver, BC, Canada, pp. 15509-15519,20 2019.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: artaaaeeiinainnneeoovaaareeerrrr for self-supervised learning. CoRR, abs/2105.04906 2021. URL https::/azxiv.crry
abs 2bb5. 64906
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for un- supervised learning of visual features. CoRR, sss/1807.05520, 2018. URL http://arxiv
Brg/abs/1800 05520
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.09882, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Ar- mand Joulin. Emerging properties in self-supervised vision transformers. CoRR, abs/2104.14294, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey F. Hinton. A simple framework for contrastive learning of visual representations. Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 2020a. URL mlr. http:/v111/cccee21111
chen2 html
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. CoRR,
abs/2011. 105566, 2020. URL httts::////s/ttiii wttp:/iwwwwrg//b////0//0 10566
Xinlei Chen, Haoqi Fan, Ross B. Girshick. and Kaiming He. Improved baselines with momen- tum contrastive learning. CoRR, abs/2000.004297, 2020b. URL. htt.:/ttsss//xxi..oro//////
Ching- Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. De- biased contrastive learning. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurlPS 2020, De- cember 6-12, 2020, virtual, 2020. URL
Mao
http:/ocweediinn......iipppeeeee nwwrips.......... wttppp.cc//aae cc/paper/
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, PMLR 15:215-223, 2011.
10

---

Under review as a conference paper at ICLR 2022
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 26. Curran Asso- ciates, Inc., 2013. URL https:::p//.......... httt::ttt:///tttttttt http::::s///prooeeenggggg www.innnn......ueeeeeeeeee wwwries.............. httpep//pps.ccpppppp//////111111 a aF725b05776022277777577777533333
pdf
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-24882 245, 2009. doi: 10. 1...109/VVP...O
Virginia R. DeSa. Learning classification with unlabeled data. In NIPS 1993, 1993.
Adji B. Dieng, Yoon Kim, Alexander M. Rush, and David M. Blei. Avoiding latent variable collapse with generative skip models. In Kamalika Chaudhuri and Masashi Sugiyama (eds.). The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019, 16-18 April 2019, Naha. Okinawa, Japan, volume 89 of Proceedings of Machine Learning Research, pp. 2397-2405. PMLR, 2019. URL html
Carl Doersch and Andrew Zisserman. Multi-task self-supervised visual learning. abs/1708.078s 07860, 2017.
CoRR,
Carl Doersch, Abhinav Gupta, and Alexei A. Efros. Unsupervised visual representation learning by context prediction. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 1422-1430. IEEE Computer Society, 2015. doi: 10.1 10.1109/1CCV2205..100 167. URL https::: doi org. www.119I/.....110
M.D Donsker and S.R.S Varadhan. Asymptotic evaluation of certain markov process expectations for large time, iv. Communications on Pure and Applied Mathematics, 36-2):183-212, 1983.
William Falcon and Kyunghyun Cho. A framework for contrastive self-supervised learning and designing new approach. arXiv preprint arXiv:2009.00104, 2020.
WA Falcon al. Pytorch lightning. GitHlu. Note: http://gghhhaacoom//////ooinninh lightming, 3, 2019.
R. A. Fisher. The use of multiple measures in taxonomic problems. Annals of Eugenics, 7:179-188. 1936.
Aude Genevay, Gabriel Peyré, and Marco Cuturi. Learning generative models with sinkhorn di- vergences. In Amos J. Storkey and Fernando Pérez-Crus (eds.). International Conference Artificial Intelligence and Statistics, AISTATS 2018, 9-11 April 2018, Playa Blanca, Lanzarote, Canary Islands, Spain, volume 84 of Proceedings of Machine Learning Research, pp. 1608-1617. PMLR, 2018. URL. httt:tt/t////occeee
ngs m.
/ewevvyy//////////////////////// Ba ht.m1
Spyros Gidaris, Praveer Singh. and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. CoRR,
2018. URL /OLL://////xxxx..r////333
1803. 07728
Spyros Gidaris, Andrei Bursuc, Gilles Puy, Nikos Komodakis, Matthieu Cord, and Patrick Pérez. Online viguoo---suullworddd generation for unsupervised representation learning. CoRR, abs/2012.11552. 2020. URL http:tttts/sxi...coom
Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: training im- agenet 1 hour. CoRR, abs/1 706.02677, 2017. URL attp:///axxl..og//bbb//// 02677]
Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra. Scaling and benchmarking self- supervised visual representation learning. In 2019 IEEE/CVF International Conference on Com- puter Vision, ICCV 2019. Seoul, Korea (South). October 27 November 2, 2019, pp. 6390-6399 URL https: htttt::://ddi.ooooe www.:oooooo..ia.o wtt.../..000/....1//////...
IEEE, 2019. doi: 10. 10.1109//CC
201 9. 00649
11

---

Under review as a conference paper at ICLR 2022
Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang. Vivek Pai, Man- nat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, and Piotr Bojanowski. Self- supervised pretraining of visual features in the wild. CoRR, abs/2103.01988, 2021. URL
https:://rra fttt:s://axxiv.rr//bbboo. abs /2t../2133..0 .01988
Jean-Basticn Grill, Florian Strub, Florent Altché, Corentin Tallec. Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprini arXiv:2006.07733, 2020.
R. Hadsell, S. Chopra, and Y. LeCun. Dimensionality reduction by learning invariant map- ping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06) 06), volume 2. pp. 1735-1742, 2006. doi: 10. 109/1////P 100.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 9726-9735. IEEE, 2020. doi: 10. 10.1100 111.///VV22200022200000000
URL https: : /tt:://doo doi /ttp:pss//aci.or/111....
CVPR42 2000.02200002 00975
Olivier J. Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, S. M. Ali Eslami, and Aäron van den Oord. Data-efficient image recognition with contrastive predictive coding. CoRR, Abs/1905.09272 2019.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. 2016.
R Devon Hjelm, Alex Fedorov, Samuel Marchilar,,,,,on, Karan Grewal, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximiza- tion. arXiv preprint arXiv: 1808, 066000 2018.
Daniel Im Im, Sungjin Ahn, Roland Memisevic, and Yoshua Bengio. Denoising criterion for vari- ational auto-encoding framework. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 31, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
Iyya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regulariz- ing deep reinforcement learning from pixels. CoRR, abs/2004. 13649, 2020. URL https: 7/ //azziv.erg//as/2244.111
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, Citeseer, 1, 2009.
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathe- matical statisties, 22(1):79-886 1951.
Fierre-Alexane Mattei and Jes Frellsen. Leveraging the exact likelihood deep latent variable models. arXiv preprint arXiv: /00.:0080024 2018.
12

---

Under review as a conference paper at ICLR 2022
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representa- tions. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 2020.00674. Utttttsss:/ / doi. org/16........ w0000/0/422000022222222222 2020. 00674
IEEE, 2020. doi: 10. 10.1109/C2
Nathan Ng, Kyunghyun Cho, and Marzyeh Ghassemi. SSMBA: self-supervised manifold based data augmentation for improving of-mmin robustness. CoRR, abs/2009.10195, 2020. URL
htttps://arxiv.org//aa wttp:////pabbb/..ooa//////// 10195
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw
puzzles. CoRR, abs/1603.09224. 2016. httttttp///xxii// www....w....aa//ooee.. 09246
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprini arXiv: 887.03778 2018.
Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Context encoders: Feature learning by inpainting. CoRR, abss/664.00739, 2016. URLAtttp://arxi
org/abs/16t /044 07379
Roberta Raileanu, Max Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in deep reinforcement learning. CoRR, abs/2006.12862, 2020.
URL /RRL:/ttpp///axx..rr//////
12862
Pierre H. Richemond, lean-Bastien Grill, Florent Altché, Corentin Tallec, Florian Strub, Andrew Brock, Samuel Smith, Soham De, Razvan Pascanu, Bilal Piot, and Michal Valko. Byol works even without batch statistics, 2020.
Joshua David Robinson, Ching- Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learn- ing with hard negative samples. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https: //open
Antti Tarvainen and Harri Valpola. Weight- consistency targets improve semi supe-supe deep learning results. CoRR, abs/1703.01780, 2017. httpttttp///xi..otttt
01780
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. CoRR, abs/1906.05844, 2019. /Rttp:
arg/abs//999 1584/
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neu- ral Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, vir- tual, 2020a. URL https::/proceedingg/ neurs http://ss...pppppe//////eeeeeeea .htaaattttmmmm
Yuandong Tian, Lantao Yu, Xinlei Chen, and Surya Ganguli. Understanding ser-euerreeeereeeeeeeeeeeeee leaniing with dual deep networks. CoRR, abs/2010.00578. 2020b. htRpsttss//aa /bs/
20100.055
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Picrre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In ICML, pp. 1096-1103, 2008. URL
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoing Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in deep network with a local denoising criterion. Journal of machine learning research, 11(12), 2010.
K.Q. Weinberger. J. Blitzer, and L.K. Saul. Distance metric learning for large margin nearest neigh- bor classification. In Advances in neural information processing systems, pp. 1473-1480, 2006.
13

---

Under review as a conference paper at ICLR 2022
Yuxin Wu, Alexander Kirillov, Francisco Massa, Yen Lo, and Ross Girshick. Detectron2. 2019.
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non-
parametric instance-level discrimination. CoRR, abs/1805.01978, 2018. Uttttt/://arxiv
brg/abs/1805 01978
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. CoRR, abs/2103.03230, 2021. URL https://arxiv.
org/aba/ 2103. 03230
Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful image colorization. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016 14th Euro- pean Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III, vol- ume 9907 of Lecture Notes in Computer Science, pp. 649-666. Springer. 2016. doi: 10. 1001 978-3-319-46487-91 URL https: /ooi /ttt::://o1..rrg111100001//
319-
40
14

---

Under review as a conference paper at ICLR 2022
A
APPENDIX
A.1 EXPERIMENTAL SETUP
The encoder & in our experiments is composed of a residual network backbone (He et al. 2016) followed by projection layer similar to the one described in (Chen et al. 2020a). The decoder to is an inverted version of residual backbone with its batch normalization (loffe & Szegedy) 2015) layers removed. We use Resner-50 default option for both the encoder and decoder, but later experiment with varying the decoder architecture.
Architecture
Datasets We test the proposed AAVAE and other more conventional autoencoder models by pre- training them three datasets: CIFAR-IA 10 (Krizhevsky & Hinton 2009), STL-10 (Coates et al. 2011] and Imagenet (Deng et al., 2009). CIFAR-10 consists of 50,000 32x32 images in the training set and 10,000 images in the test These images are equally divided across 10 labeled classes. For pretraining we use 45,000 image from the training set while 5,000 images are kept for validation. The STL-10 dataset consists of 100,000 unlabelled images resized to 96x96 which are split into 95,000 images for self-supervise pretraining and 5,000 for validation. It further consists of 5,000 training images and 8,000 test images that are labelled across classes. We split the 5,000 training images into 4,500 images for training the downstream classification task and the remaining 500 are kept for validation. Imagenet consists of 1.2 million images the training split and 50, 000 images in the validation split, spread across 1000 classes. We separate 5000 images from the training set to create our own validation set for finetuning the hyperparameters. The official validation set of Imagenet is what we report the final results on.
Augmentation pipeline As mentioned in the paragraph above, we choose image datasets for our experiments with AAVAEs, and hence setup the denoising criterion with an appropriate domain- specific data augmentation pipeline. We define sequence of common image augmentations A = {a1, 0g, .... an such as random flip, random channel drop. We also define a, as a special transform that applies a random resize and crop an input æ. Formally, a₂ maps x Rexxxx
Rexxxu where g I h and u S w. For every input x to a AAVAE we define x+ ~ A(a,(x) (x)) as a view of a. The augmentation pipeline defined here is kept the same that of SimCLR (Chen al. 2020a), for a fair comparison with other self-supervised learning approaches.
Optimization and Hyperparameters We use Adam optimizer (Kingma & Ba 2014) during pre- training. We use linear warmup schedule for the learning rate, which is held fixed after the initial warmup. For all our ablation experiments, we keep the weight decay coefficient fixed at O. When studying the effect of minibatch size, we follow (Goyal et al. (017) and linearly scale the learning rate and the warmup epoch count with minibatch size. For the hyperparameter sensitivity ablations on CIFAR-10, we vary a particular hyperparameter while keeping the others fixed their default values. By default, we use a learning rate of 2.5 x 10 warmup the learning rate until 10 epochs, and keep the batch size at 256. For STL-10 experiments, we set the learning rate x 10-1, warmup epochs count at 10, and keep the batch size at 512. For Imagenet pretraining, we set the total batch size at 512 across 4 GPUs, the learning rate at 5 x 10-4, -4, warmup epochs count at 10 and run the pretraining for all autoencoder models until 5 million training iterations.
Finetuning Downstream classification accuracy via finetuning has become widely-used proxy for measuring representation quality. We follow the finetuning protocol put forward by Chen et al.) (2020a). After pretraining without any labels, we add and train a linear layer the pretrained encoder (representation), without updating the encoder. We train the linear layer for 90 epochs with learning rate defined by: 0.1 BatchSize /256, using SGD with Nesterov momentum.
Semi-supervised learning evaluation We run semi-supervised classification task on our models that have been pretrained on the Imagenet dataset. We follow the evaluation process mentioned in previous works (Caron et al. 2020 Zbontar et al. 2022), and train the model on 1% and 10% labeled splits of Imagenet. The training is carried out for 20 epochs with a batch size of 256, using an SGD optimizer with momentum of 0.9 and weight decay. Since this is semi-supervised learning setup with a certain percentage of labels available from the dataset, the backbone is unfrozen during the training process and is trained at learning rate of 0.01 for the 10% labeled split and at 0.02 for
15

---

Under review as a conference paper at ICLR 2022
the 1% labeled split. The linear layer appended top of the backbone is trained at learning rate of 0.2 for the 10% labeled split and at rate of 0.5 for the 1% labeled split.
Transfer learning tasks For the linear classification transfer learning task dataset with the commonly used evaluation protocol (Zbontar et al.] 2021; Caron et al. 2020). We train a single linear layer on top of our model for 14 epochs with SGD optimizer with learning rate of 0.01, momentum of 0.9 and a weight decay of 5e-4. The learning rate is multiplied by factor of 0.1 at equally spaced intervals during the training.
Places205
For the object detection transfer learning task, we use the VOCO7+12 trainval set for training and VOCO7 test set for eval as previously done by Zbontar et al. (02T). ). Faster R-CNN with a C4 backbone is used for this downstream task. We train with batch size of 16 across 8 GPUs for 24000 iterations with base learning rate of 0.01. We use detectron2 (Wu et al. 2019) library to perform this evaluation.
Pretraining duration As we demonstrate in the paper, the proposed AAVAE benefits from being trained long as it is feasible. We report the downstream accuracies measured different points of pretraining. More specifically, we run linear evaluation on our encoder after 400, 800, 1600, and 3200 epochs for the CIFAR-10 experiments. For STL-10, we pretrain models till 3200 epochs. For Imagenet, we train upto 5 million training steps, which is approximately 2100 epochs.
Compute and Framework All CIFAR-10 Krizhevsky & Hinton 2009) experiments are done on single GPU with a memory size of at least 16GB. All STL-I0 experiments are done using two GPUs in the same category. We select GPUs from mix of NVIDIA RTX 3090s and V100s for CIFAR-10 10 and STL-10 experiments. Imagenet experiments and downstream evaluations are carried out on 4 A 100s. Our codebase uses PyTorch Lightning (Falcon al. 2019).
A.2 SEMI-SUPERVISED LEARNING
We finetune the Resnet-50 (He et al. 2016 backbone pretrained by AAVAEs on specified labeled subsets of Imagenet. The two subsets used contain 1% and 10% labeled images of the total num- ber present in the dataset. Table | shows the results for the baseline autoencoder models and our proposed AAVAE. The baseline autoencoders are pretty poor in their performance for this semi- supervised evaluation task. In some instances, their performance 0.1% accuracy on Imagenet, which is equivalent to chance. The AAVAE outperforms the remaining autoencoders considerably on this task with 21.37% accuracy on the 1% labeled subset and a 39.85% accuracy on the 10% labeled subset. However, this is still quite behind when compared against the supervised results or results from other current SSL methods.
A.3 TRANSFER LEARNING TO OTHER TASKS
For transfer learning to classification tasks, we finetune linear layer on top of the frozen Resnet-50 backbone pretrained by VAE and AAVAE on Places205 dataset for scene classification.. The finetun- ing protocol is kept the same as the previous works of Zbontar al. (2021) Misra & van der Maaten
Table 2: Semi-supervised evaluation of Resnet-50 encoder with 1% and 10% labels on Imagenet. Entries with next to them performed equivalent to chance result on for Imagenet.
Imagenet
Method
1%
10%
Supervised
25.4
56.4
SimCLR
48.3
65.6
Barlow Twins
55.0
69.7
B BYOL
53.2
68.8
AE
0.1*
0.1*
AAAE
0.1*
0.31
VAE
0.1*
0.98
AAVAE
21.37
39.85
16

---

Under review as a conference paper at ICLR 2022
Table 3: Transfer performance of Imagenet pretrained Resnet-50 backbones on classification and object detection tasks. Places205 dataset is used for classification transfer task with the table report- ing classification accuracy. For object detection, we use VOCO7+ 12 dataset with Faster R-CNN algorithm and C4 bakcbone.
Places205
VOC07+12
Method
Acc.
APau
AP50
AP75
Supervised
51.1
53.5
81.3
58.8
Jigsaw
41.2
48.9
75.1
52.9
Rotation
41.4
46.3
72.5
49.3
Barlow Twins
54.1
56.8
82.6
63.4
VAE
6.78
2.45
6.49
1.67
AAVAE
41.45
15.22
35.69
10.09
(020). Table shows the results for this downstream evaluation. For comparison, also include results on Places205 from pretext tasks of Jigsaw (Noroozi & Favaro 2016) and Rotation (Gidaris) al. 2018), while the same time including results from one of the current high performers this evaluation, namely. Barlow Twins Zbontar et al. 2021).
The finetuning process of object detection transfer task is done on VOC07+12 trainval dataset and the task is evaluated on VOCO7 test set. The results are shown in Table 3 Even though the AAVAE performed comparable to the Jigsaw and Rotation pretext tasks on Places205 classification, its performance is greatly affected on the VOCO7 detection task. It is far behind the results of these pretext tasks. This result asks whether reconstruction based SSL techniques are good for transferring representations for object detection tasks. This is something that can be explored in future work.
A.4 DIRECT INSPECTION OF REPRESENTATION
major motivation behind our proposal was domain-specifer data augmentation encourage representations to encode domain-specific invariances. AAVAES indeed reflect such invariances, expect vector representations coming of domain-specifie perturbations of an individual ex- ample to be highly aligned with each other. We test whether this property holds with the AAVAE more strongly than the original VAE by inspecting cosine similarities between pairs of perturbed inputs produced by the same example and between pairs of perturbed inputs produced by differ- ent examples. When the former higher than the latter, we can say the representation encodes domain-specific invariances induced by data augmentation.
In Fig. 5 (a)(i. we make two observations. First, the representation vectors are all extremely aligned for the original VAE. We can interpret this from two perspectives. The first perspective is the so-
Original
Views
38
(6) cosine sim. between vows of same example and different examples
(b) images and their views used to generate cos sim. matrix
Figure 5: Part (a) shows cosine similarity matrices between pairs of vectors produced by views the same example and between pairs of vectors produced by views of different examples. We observe a posterior collapse in the case of VAFs in (a)(). For AAVAES in (a)(ii) we see strong alignment between views of the same example while the views of different examples other in the representation space. In (b), we show images from the STL-10 dataset (Coates et al. 2011) and their corresponding perturbed versions that generate the cosine similarity matrices in (a).
far apart from each
17

---

Under review as a conference paper at ICLR 2022
called posterior collapse (Higgins et al. 2016 Dieng et al. 2019), in which all the approximate posterior distributions, ie., the representation vectors, are detached from the input and collapse onto each other. The second perspective is the lack of domain-specint invariance, which is evident from the lack of any clusters. Fither way, is obvious that the representations extracted by the original VAE do not reflect the underlying structure of the data well.
On the other hand, with the proposed AAVAE, we see clear patterns of clustering in Fig. 5 (i) The vectors produced from one example are highly aligned with each other, while the vectors pro- duced from two different examples are less aligned. In other words, the representations capture domain-specific invariances, induced by data augmentation, and the AAVAE does not suffer from posterior collapse. Both these things were well anticipated from the design of our algorithm.
18


Reviews:
[{"review_id": "oh6LZyZVbF0", "reviewer": "Reviewer_Dk62", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Here are some comments:\n\n- It is not cot clear what authors mean by \"batch-level statistics mean that non-contrastive losses are not decomposable as well\". Why decomposition is of greater importance?\n- Non-contrastive approaches can be trained with relatively smaller batch-sizes, compared to contrastive approaches. See Fig. 3 of BYOL.\n- In introduction section, authors motivate the use of VAE as way of getting rid of large batch size and being agnostic to the input domain. However, one paragraph after, they suggest removing the KL and replacing it with a domain-specific loss term for better performance. The design does not follow the motivation.\n- While Eq. 1 constitutes a valid ELBO, I'm not sure whether Eq. 3 does. Authors haven't discussed why that results in a valid ELBO.\n- I don't agree with the discussion around approximating data-level stats and the requirement of large batch-sizes and how the proposed method handles that.\n- While AAVAE shown to be better than other AE methods developed by the authors, there's a huge gap to existing non-contrastive methods, such as BYOL, MoCo, SwAV, and DINO (missing in experiments). I'd argue that I rather prefer 24% better accuracy on ImageNet than a decomposable loss. I don't agree with the argument in Section 4.1; autoencoding is not a viable alternative to non-contrastive methods, while it can be another approach with similar performance to some of pretext tasks designed for SSL.\n- Fig. 4(a) suggests that larger batch size does not help significantly. This experiment should be done on ImageNet, where a comparison to other recent SSL method with larger batch-size is possible. In that case, one can conclude with equal batch-size, decomposable loss in AAVAE is beneficial. Also, I believe one can train BYOL with smaller batch-size. For instance, see Fig. 3 of BYOL: Even with batch of 256 it achieves more than 73% accuracy on ImageNet, which is still 22% higher than AAVAE.\n- I also encourage authors to go beyond one downstream task of classification. What are the benefit of AAVAE in other tasks such as detection and segmentation?", "summary_of_the_paper": "This paper proposes a VAE-based approach for the task of self-supervised learning. Authors argue that the need for large batch size can be alleviated by using a augmentation-augmented VAE, which also has the benefit of decomposable loss. In a number of experiments, authors show that AAVAE is comparable to or better than some of the pretext-task based approaches such as Jigsaw and Rotation.", "summary_of_the_review": "I don't agree that AAVE can be considered as a third family of SSL methods. There is not enough justification around why one would accept 22%-24% lower performance (even with small batch-size) and use AAVE.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "luUwKH8Apev", "reviewer": "Reviewer_qnqK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The proposed method is simple to follow.\n2. The experiments results show that the proposed method greatly surpasses the existing alternatives. \n\nWeaknesses:\n1. By removing the KL in traditional VAE, I think the proposed method can not be called AAVAE since the `V` in VAE has special meanings. It might cause some misunderstanding to the people who are familiar with VAE.\n2. Also, when removing KL, I doubt whether AAVAE could sample like VAE dose since I do not find any sampling results in the paper.\n3. At last, the features for classification naturally become better when removing KL without considering. Thus, the paper lacks novelty for me.", "summary_of_the_paper": "This paper introduces a self-supervised learning method augmentation-augmented variational autoencoders (AAVAE) by removing the KL divergence in the traditional VAE. And the experiments on image classification show that the learned features by AAVAE have better properties than the existing alternatives.", "summary_of_the_review": "See the main review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "W7vAAxZlwFh", "reviewer": "Reviewer_ebYf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Postives:\n- The method is very simple and practical for using\n- The method can be implemented easily\n- The performance is very promising\n- The hyperparameter study done is very promising\n\nNegatives: \n- The paper claims that \"In reinforcement learning, Kostrikov et al. (2020) and Raileanu et al. (2020) have shown the benefit of adding domain information via pixel-level data augmentation in continuous control.\" which is not true. Both those papers simply add common image-level augmentations to the RL pipeline and are not specific to the domain of DM Control suite environment. Similar papers have also used similar methodology [1,2,3] to a variety of different environments (such as procgen) which share minimal domain specificity as DM Control Suite. In fact its probably pretty easy to claim that self-supervised learning works as a domain in-specific algorithm since different works have performed similar methods across a variety of tasks by just changing how the augmentations are added using generic and well-known techniques, where for ex [4] adds simple gaussian noise to state values.\n- The paper critiques the self-supervised learning techniques by stating: \"First, there is no principled way to choose negative examples, and hence these negatives are chosen somewhat arbitrarily each time.\" which is misleading. In contrastive learning / triplet learning, there exist many methods that help with mining both the positive and negative samples. See [9] for just a few (of many) different possibilities that may be helpful.\n- The paper motivates VAE and the related works for a significant portion of the paper and does not talk about the proposed idea itself. \n- By removing the KL loss, is the existent method still a VAE? \n- The paper spends alot of time motivating VAEs but ends up removing the KL loss, which is somewhat confusing. Also in Figure 1, I am not sure how the model is updating the posterior if it has gotten rid of the KL loss (and thereby does not have a prior distribution over the latents).\n- It would also be useful if the authors can state the differences between a denoising autoencoder (where the noise function is augmentations or A(.)) and AAVAE. \n- Further explaining what the paper means by \"domain specific transformations\" would also be useful. To the reviewers knowledge, augmentations do not need to resemble real life physical transformations, so I am not convinced if augmentations are indeed \"domain specific\" in any context\n- Do the authors have any intuition as to why the VAE outperforms the AAAE since the authors suggest that the reason to drop the KL loss was to make things domain invariant, but its unclear if the KL loss would have helped or not, given that VAE is empirically stronger than AAAE. \n- It would be interesting to see Fig 3 with CIFAR-100 instead of CIFAR-10 since CIFAR-10 is possibly to easy of a dataset to require the KL loss. I am not convinced why the KL term + the denoising autoencoder criterion is not helpful since the VAE outperforms the AAAE in the \n- The paper misses a few important citations [5-8]\n\n\n\n[1] Laskin, Michael, et al. \"Reinforcement learning with augmented data.\" arXiv preprint arXiv:2004.14990 (2020).\n\n[2] https://arxiv.org/abs/2102.11271\n\n[3] http://proceedings.mlr.press/v139/stooke21a.html\n\n[4] https://arxiv.org/abs/2103.06326\n\n[5] http://proceedings.mlr.press/v119/jun20a.html\n\n[6] https://arxiv.org/abs/2105.14859\n\n[7] https://arxiv.org/abs/2010.02014\n\n[8] https://openaccess.thecvf.com/content_CVPR_2020/papers/Zhu_S3VAE_Self-Supervised_Sequential_VAE_for_Representation_Disentanglement_and_Data_Generation_CVPR_2020_paper.pdf\n\n[9] https://github.com/littleredxh/EasyPositiveHardNegative", "summary_of_the_paper": "The paper presents a new approach to self-supervised learning using VAEs which uses augmentations and a non-contrastive self-supervised objective to perform unsupervised representation learning on the data. The paper compares to state-of-the-art self-supervised methods and underperforms in comparison, but the state-of-the-art methods require significantly more horse power. ", "summary_of_the_review": "The paper currently suffers clarity in explanations. See the main review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "O2kxyd46iL", "reviewer": "Reviewer_61ZD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- How does your proposed method differ from a denoising autoencoder? Is it fair to say that this method corresponds to a denoising autoencoder where the noise is a random image augmentation and, additionally, noise is added to the latents before reconstructing the image? I find the current name and presentation somehow misleading since by removing the KL divergence, your method is not really optimizing a variational objective anymore, is it?\n- By comparing with the AAAE baseline, you tested the importance of your augmentations on the usefulness of the representations. Did you also test what happens if you do not use augmentations only add noise to the latent representations?\n- Table 1: What is the difference between the left and right parts of the table? Is the right part an extension of the Imagenet column of the left table?\n- 4.2: This section is a bit confusing for me: First I thought you wanted to argue that even though your model suffers from overfitting this is not problematic; then I thought you meant that your model is robust to overfitting, and, finally, it sounded you do the exact opposite with your experiments. If it's the former: I do not agree with your conjecture regarding overfitting. I do not see why the ability of the model to learn the mapping augmented image -> clean image in a robust fashion prevents the model from overfitting. In general, the fact that your model solves some task (and might learn some kind of invariance) to some data points does not allow you to make conclusions about its general ability to solve it. If you mean that your model does not suffer from overfitting I recommend rewriting this paragraph since that's not clear. Regarding your experiments: these results would be more convincing if you trained longer such that the accuracy does not change anymore. Based on this figure one cannot conclude whether the model is prone or robust to overfitting.\n- Figure 3: Are these results for a single run or averages over multiple runs? In general, I recommend re-running the experiments - at least for the computationally cheap datasets - for multiple random seeds such that you can report the mean and error of the downstream performance (for all tables and figures, not just Figure 3).\n- 4.4. Recent methods like SimSiam also do not need large batch sizes to give reasonably good representations.", "summary_of_the_paper": "This paper presents a modification to the VAE training scheme. Here, the training objective is modified with the aim of encouraging the model to correctly learn invariances and equivariances of the data. Finally, the authors empirically evaluate the proposed model on standard datasets such as CIFAR-10, STL-10, and ImageNet.", "summary_of_the_review": "Even though there are some unclarities in the text and experimental results can be improved, the paper contains valuable information for the community. Furthermore, these changes should be feasible to implement during the revision period.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7MW1au4CCrD", "reviewer": "Reviewer_cQeC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n- The proposed self-supervised method mitigates the requirement for choosing/mining negative examples in a contrastive learning setting and the need for large batch sizes (which is not typically feasible for high dimensional data spaces, e.g., 3D images in medical applications) to derive batch-level statistics for non-contrastive learning methods.\n- Data augmentation is a principled approach to incorporate application/task-specific knowledge.\n- Comprehsive evaluation against SOTA and ablation experiments within image classification tasks.\n- Experiments empircally showcase robustness to hyperparameters including batch size.\n\nWeaknesses\n\n- The significance of decomposing a self-supervised loss over training examples is not well articulated. It is also not clear how the lack of such decomposition negatively impacts contrastive learning methods.\n- The paper ignores the large body of work that mines hard negative samples for contrastive learning methods.\n- By removing the KL term, the ELBO is not a valid lower bound. Also, the method is not variational anymore. \n- The paper lacks novelty as using data augmentation to train an auto-encoder is similar to the denoising criterion for AEs and VAEs.\n- The use of data augmentation for self-supervision requires hand-crafted/pre-defined data transformations that are specific to the application at hand, while advantageous to inject domain knowledge, this might lead to learned representations that are not generic enough for multiple downstream tasks. \n- Though presented in a different setting, self-supervise learning and injecting domain-specific prior knowledge, the core idea lacks novelty as it is very similar to training auto-encoding based models with a denoising criterion. \n- No details provided on the class of transformations used (i.e., A?).\n- Convergence and timing for the proposed model is not analyzed in comparison to SOTA, specifically for experiments where it lags or is hardly on par with SOTA.\n- Experiments lack evaluation in settings beyond classification, e.g. segmentation, regression.\n- While there is a significant improvement compared to AE-based methods, there is a huge performance gap between the proposed method and most of the SOTA considered in the experiments.\n", "summary_of_the_paper": "The paper presents a third family for self-supervised learning that relies on generative modeling, in particular variational autoencoders (VAE). The proposed model uses data augmentation in the data space as a way to promote learning richer latent representations, where the augmentation strategy reflects data/domain-specific invariances and equivariances. This augmentation strategy regularizes the VAE  learning process and is used to replace the domain-agnostic KL divergence regularization. Learned representations are evaluated on image classification downstream tasks and compared with recent contrastive and non-contrastive self-supervised methods. ", "summary_of_the_review": "The use of data augmentation and generative model as another family for self-supervise learning has merit.  The proposed methods share the idea of training VAEs with a denoising criterion, albeit removing the KL term in the proposed method. Experiments lack evaluation in settings beyond classification and performance lags behind self-supervised SOTA.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an augmentation-based training of autoencoders with stochastic latent space. The proposed method is examined on the representation learning task on several image datasets. While the reviewers found the submission interesting, simple, and easy to implement, they also raised serious concerns around the novelty of the proposed method and the impact of removing the KL term (which removes the generative interpretability of the model). Unfortunately, the experiments do not provide a convincing utility of the model compared to more popular representation learning methods (i.e., contrastive and non-contrastive methods). Given these concerns, the paper is not ready for presentation at ICLR.
Decision: Reject