Identity-Disentangled Adversarial Augmentation for Self-supervised Learning
Authors: Kaiwen Yang
Tianyi Zhou
Xinmei Tian
Dacheng Tao
Venue: ICLR 2022 Submitted (2022)
Keywords: identity disentanglement
contrastive learning
data augmentation
self-supervised learning

Abstract: Data augmentation is critical to contrastive self-supervised learning, whose goal is to distinguish a sample's augmentations (positives) from other samples (negatives). However, strong augmentations may change the sample-identity of the positives, while weak augmentation produces easy positives/negatives leading to nearly-zero loss and ineffective learning. In this paper, we study a simple adversarial augmentation method that can modify training data to be hard positives/negatives without distorting the key information about their original identities. In particular, we decompose a sample $x$ to be its variational auto-encoder (VAE) reconstruction $G(x)$ plus the residual $R(x)=x-G(x)$, where $R(x)$ retains most identity-distinctive information due to an information-theoretic interpretation of the VAE objective. We then adversarially perturb $G(x)$ in the VAE's bottleneck space and adds it back to the original $R(x)$ as an augmentation, which is therefore sufficiently challenging for contrastive learning and meanwhile preserves the sample identity intact. We apply this ``identity-disentangled adversarial augmentation (IDAA)'' to different self-supervised learning methods. On multiple benchmark datasets, IDAA consistently improves both their efficiency and generalization performance. We further show that IDAA learned on a dataset can be transferred to other datasets.  

Under review as a conference paper at ICLR 2022
IDENTITY-DISENTANGLEED ADVERSARIAL AUGMEN- TATION FOR SELF- LEARNING
ABSTRACT
Data augmentation is critical to contrastive self-supervised learning, whose goal is to distinguish a sample's sspl's augmentations (positives) from other samples (negatives). However, strong augmentations may change the sample-identity of the positives, while weak augmentation produces easy positives/negatives leading to nearly- zero loss and ineffective learning. In this paper, we study a simple adversarial augmentation method that can modify training data to be hard positives/negatives without distorting the key information about their original identities. In particular, we decompose sample to be its variational auto-encoder (VAE) reconstruction G(x) plus the residual R(x) = x G(x), where R(x)) retains most identity- distinctive information due an information-theoreti interpretation of the VAE objective. We then adversarially perturb G(x)) in the VAE's bottleneck space and adds back to the original R(xx) as an augmentation, which is therefore sufficiently challenging for contrastive learning and meanwhile preserves the sample identity intact. We apply this "identity "identity-disentaai disentangled adversarial augmentation (IDAA)" different self-supervised learning methods. On multiple benchmark datasets, IDAA consistently improves both their efficiency and generalization performance. We further show that IDAA learned on dataset can be transferred to other datasets.
INTRODUCTION
Empowered by deep neural networks and the computational capability of recent hard- ware/infrastructure, machine learning (ML) has achieved great breakthroughs on some challenging problems when sufficient labeled data available. However, deep learning known be data-hungry and annotating data in many domains, e.g., medical care or predictions of protein structures, are either difficult or expensive. To overcome this limitation, self-superrised learning (SSL) methods train a model on unlabeled data in a supervised learning manner using self-generater labels by manipulating the data, e-g., rotation degrees (Gidaris et al., 2018), Jigzaw puzzle solutions (Noroozi & Favaro, 2016), clustering (Caron et al., 2018), back-translation (Zhu et al., 2017), etc. These methods recently start to perform on par with supervised learning and exhibit potential to even surpass it Chen et al. (2020); Chen & He (2021). Moreover, their learned representations can be generally applied to different downstream tasks.
Many widely- used SSL methods are built upon sample-identiyy preservation tasks, e.g., contrastive learning (Oord et al., 2018; Tian et al., 2020a; Chen et al., 2020) and consistency regularization (Chen & He, 2021; Grill et al., 2020; Caron al., 2020), which aim learning representations that can preserve the identity of the original sample after applying data augmentations and thus distinguish the augmentations of diffferent samples. For example, contrastive learning targets on an representation space in which a sample (anchor) is closer to its own augmentations (positives) than other samples or their augmentations (negatives). The effectiveness of contrastive learning therefore heavily depends on the quality of data augmentations.
Most SSL methods (Ye al., 2019; Chen et al., 2020; He al., 2020: Chen & He, 2021) utilize pre-defined data augmentations to generate positives and negatives. However, as shown in the example (green point) of Fig. 1, they are not adaptive to the data manifold in the embedding space during training and the generated augmentations can be too easy for the sample identification task. In practice, these SSL methods need tens to hundreds times of epochs required by supervised learning to reach comparable performance on the downstream classification tasks (Chen et al., 2020; He et al., 2020; Chen & He, 2021). For the same reason, large batch-size is common and necessary for contrastive learning in order to involve sufficient hard negatives. Therefore, how to modify the positives/negatives for more informative contrastive loss is a critical yet open challenge towards more efficient SSL. Learnable and adaptive data augmentations have been explored for supervised

---

Under review as a conference paper at ICLR 2022
Dleenity-changge
hard augmentqtion
founsly Bandom Augroriaigs IDAA
lornly provving boundary Rander Augesstation C-A
Negative2
CLAM
Negotive
Identity-- frreelyee
easy ougmentation
Anchor
ldjenitt--rrseervn
Random Adjefiitation
hard ougmentation
Disience alandorronn to
Dilance dlacceeteppoooo
12
(a)
(b)
(C)
Figure| 1: Data augmentations for contrastive Learning: IDAA, CLAE and random augmentation. (a) Identity- preserving and hardness of data augmentations: random augmentation generates identily-preserved but easy samples, CLAE adversarially generates hard but ddentity-distorte samples, while IDAA (ours) can generate hard and identity-preserved samples. (b)
and hardness of augmentations by different methods: Points below the boundary line have the identity changed, while points close to the boundary are hard samples. (c) Kernel density plot for (b): IDAA generates hard samples without changing the identities.
learning (Cubuk et al., 2018; 2020; Liu et al., 2021) but have not been widely studied in SSL. Probably the most related augmentation method for SSL is CLAE (Ho & Vasconcelos, 2020), which generates hard positives/negaaiivs by adversarially attacking the input. However, as shown in Fig. 1, adversarial augmentations may change the original sample identity and it is infeasible to tune the attack strength for every sample to preserve the identity.
Hence, another challenge for data augmentation in SSL is how to preserve sample identity. Although hard positives/negattves might be achievable by stronger augmentations or adversarial augmentations listed above, they can also distort the true identities of the original samples the red point in Fig. 1 so the model may erroneously identify other different samples or their augmentations as the anchor sample. Training with those ddentity-changer augmentations might lead trivial solutions for SSL and poor representations.
To address the aforementioned two primary challenges of contrastive learning, we have consider how they interfere with each other. For better efficiency, the data augmentation needs to generate positives and negatives as challenging as possible for the model to distinguish the sample identity, but it should not remove or distort the minimum necessary information retaining the true identity. Thereby, the sample identification task is neither too trivial nor infeasible to learn. In this paper, we study how to automatically generate data augmentations that fulfill the above conditions and improve both the efficiency and effectiveness of current self-supervised learning. We relate the objective of variational antoencoder (VAE) with the sample identification task in contrastive learning perspective, which inspires us to disentangle the
from an information-theoree information of an input x as the residual of VAE reconstruction G(z), i.e., R(x) = x G(x)). As illustrated in Fig. 2, in order to modify x to be more challenging in terms of sample identification, we propose to apply adversarial perturbations to the bottleneck features of VAE, which maximize the contrastive loss in an e-ball and results in a modified Gz) We then utilize x' = G² + R(x) as a data augmentation of so the identity information captured by R(x) remains intact in x'.
Our method, called trained VAE model and does not require any labeled data. In the experiments on multiple benchmarks, when applied to different SSL methods, this simple yet principal data augmentation approach consistently brings improvements on both the efficiency and the performance on downstream tasks. In addition, we present thorough ablation study to analyze the influence of hyperparameters (e.g.. VAE hyperparameters, bottleneck dimensions, e controlling the attack strength) and experimental settings (e.g., batch size, training epochs, model architecture) on the contrastive learning's 's performance.
adversarial augmentation (IDAA)", only needs a pre-
2 BACKGROUND
Contrastive learning (CL) (Wu et al., 2018; Zhuang et al., 2019; Tian et 2020a; Ye et al., 2019; Kalantidis et al., 2020; Hu et al., 2021; Robinson et al., 2020) aims at learning representations that can distinguish different samples and their augmentations. Specifically, it formulates this sample- identification task as a classification problem on each sample (anchor), where the positives are its own augmentations and the negatives are other samples and their augmentations. A widely- loss for contrastive learning is "InfoNCE" (Tian et al., 2020a) built upon the positives and negatives created by data augmentations. For each mini-batch x (21,,22"" ... ,NN of size N. InfoNCE loss
2

---

Under review as a conference paper at ICLR 2022
Original image
dentity-dssentangee
Identity-relevant Adversarially attacked G(X) Augmented
Adversarial attack:
I - 2 + 8
8 esign(V sLACE)
SIAR
E(-)
D
+
G(-)
G(X)
x G(x)
G'(o)
G(x) + 630
Figure 2: Architecture and pipeline of Deiettt-Disenttagleed Adversarial Augmentation (IDAA).
is computed as:
N log (NCE(ile = xi), (NCR(1 xx)) exp (FP(((()))
LNCE(X) = N
exp (f(A()))) ((B((,))
where (:-) denotes inner product, A(-) and B-) denote two different transformations for data augmentation, f(-) and fs(-) are the embedding networks that can be either identical (Chen et al., 2020; Ye et 2019) different (Misra & Maaten, 2020; He et al, 2020), and aNCE(E]a x..) is the softmax probability of identifying the augmentation (xx;) is transformed from the original sample critical to the performance and efficiency of CL: weak
r₁- The choices of data augmentation augmentations might already result in fully distinguishable representations over different samples and nearly zero CL loss, while strong augmentations may overly distort the identity of sample and make CL too challenging, infeasible, or inefficient. In previous works such as SimCLR (Chen et al., 2020), different (compositions of) augmentations can result in large gaps on CL's performance but finding the best one needs brute-force search of all possible transformations and their compositions, which is usually computationally forbidden in practice.
Another challenge in CL is to keep large memory bank or mini-batch to cover sufficient amount of "hard negatives" because a sample might distant from most other samples (and their augmentations) and hard negatives close to it are sparse in the training set. Wu et al. (2018) trains a non-parametric classifier to maximally scatter the representations of samples over a unit sphere. However, needs to build memory bank to store all these representations, which is infeasible for large-scale datasets. He et al. (2020) instead maintain a fixed-sized dynamic dictionary based on a FIFO queue of representations of incoming critical to the success of CL. The lack of hard negatives forms a bottleneck to the sample efficiency of CL. Therefore, how select modify the data increase the chance of including more hard negatives is a significant open problem for CL.
during training. It finds that a relatively large dictionary
Consistency regularization consistency regularization (CR) has been studied in several recent works (Chen& He, 2021; Grill et al., 2020; Caron et al., 2020) for self-supervised or semi-supervised learning, which achieves comparable or even better performance than CL in certain scenarios. Compared to CL, CR removes the comparison to negatives and only focus on maximizing the similarity between model outputs for two augmentations of the same sample, e.g..
Los(x)= - N
(B(A)))) [A((z))) (Ih((x)))
Similar to CL. CR also aims at preserving the sample identity on the learned representations and it heavily relies the choice of data augmentations. For example, FixMatch (Sohn et al., 2020) chooses to apply multiple weak augmentation for generating the pseudo-labels (e.g.. f(A())) and strong augmentations to the other branch h( (((()).
3

---

Under review as a conference paper at ICLR 2022
3 IDENTITY-DIENTANCE
ADVERSARIAL AUGMENTATION
In this section, we will firstly relate the sample- ideetffiinntiiito task broadly used in designs of self- supervised learning with the training objective of variational auto-encoder (VAE). Specifically, we will show that VAE's training tries to remove the sample-identity related information from its bottleneck features. Hence, the residual of VAE reconstruction may retain the sample-identity information that we wish to preserve in the data augmentation for self-supervised learning. We then propose a data generation and augmentation model based on VAE and analyze the lower bound for identity preservation in its augmented data. In the end of this section, we apply adversarial attack methods to this data augmentation model, which modify samples to be hard positives and negatives for more efficient contrastive learning without changing their sample-identitics
3.1
DEENIIY-DISENTANGIIE
IN CONTRASTIVE LEARNING
The goal of sample-identifienn identification can be formulated as maximizing q0y = 1|2 2z). i.e., the likelihood estimation of identifying I; or its augmentations as sample-i, where x and are two random variables for a sample and an identity label, respectively. The following proposition shows that q i|z = æ;) provides lower bound for the mutual information [(x; y) between and y.
Proposition 1. Sammpl-identiffcatti likelihood as a lower bound of I [x::y)) If a random mini- batch of size N and the sample-identificatin likelihood estimation of x, on its correct identification label y= = to Qyy i|s 2i), the mutual information I [x:; y) can be lower bounded by
N
N
log ply = 1|1 = 2.)) 2 log N Ex
[x: y) = log N + Ex
Σ
log ql = i|2 = xz)
N
N
where p(-) is the true probability and g(-) denotes an estimation of p(-). A detailed proof is given in the Appendix. In contrastive learning, ply = i = 2;) can modeled and estimated by (NCR(i)r x;) defined in Eq. (1) using neural network embedding f(-) and h(-) of data augmentations A(-) and B(.). Hence, InfoNCE loss can provides a lower-bound of [(x; y). (, i.c.,
N
1
[(x;y) 2 log N + Ex
log (Nl = x..) = log N - Ex [[NCE(x)]
(4)
N
1=1
The above lower bound relates the mutual information I ((x;y) and contrastive learning: increasing the batch size N and/or minimizing the InfoNCE loss can improve the tightness of the lower bound and results in representations with better capability on sample identification. Due to the natural sparsity of hard positives and negatives in the original data x, data augmentations and perturbations of x are necessary to improve the sample efficiency when minimizing Ex [x)] However, without any constraints, strong data augmentations may remove the identity identity-related related information of some data and change their identities, which results in @NCE(ilæ r;) << p(y i i| 2z), a loose lower bound Eq, (4), and poor representations via contrastive learning.
3.2 IDENTITY- WWw---DIEENNLEEEE
VIA VARIATIONAL. AUTO-ENCODER
Next, we will show that the training objective of variational anto-encoder (VAE) (Kingma & Welling, 2013) is also related to [xx y) and sample identification. In particular, VAE aims at removing identity-specific information from the bottleneck features i.e., minimizing I(2, y). With an output G(x) decoded from z, VAE naturally disentangles the most identity-relevant information from æ. If we can remain such information, i.e., x G(x), intact in data augmentations and only perturb the rest part G(x)), the above problem of contrastive learning can be resolved.
Lemma 1. (VAE objective and I [(2; y) from Eq. (29) in Appendix B of Alemi et al. (2016)). Assume that the bouleneck features of VAE are denoted by z, the encoder is E(-) and produces distribution
(z(x)) the decoder is D(.) and produces distribution {π(x)z), the prior for z is p(z), and the KL-divergence regularization in the VAE objective LVAE has weight B, we have:
[[;;x) + 31 (2; u) S LVAR,
(5)
N
LVAE A --aap(a)
dxp((a) / dappe())))
N DEL (pe (z/z xx) l=)())
Σ
(6)
4

---

Under review as a conference paper at ICLR 2022
In LVAE= 3 (Higgins et al., 2016) controls how close the distribution of bottleneck features p(2|æ zx) is to the prior p(2), e.g. a standard Gaussian distribution independent of the sample identity y i. So 3 controls the strength of identity disentanglement on 2. Lemma 1 shows that VAE is trained to minimize an upper bound of - _1,, x) + (7 v). i.e., preserving most information of input in the bottleneck feature 2 but removing from the critical information about sample identity y = i. Hence, the most identity-relevan information is disentangled from the VAE output G(z) (=(2) = D()) and preserved in the residual R(x) _ x - G(z).
3.3
DATA GENERATION AND AUGMENTATION
We can study data generative model based on identity- disentanglement of VAE, which generates (2) and R(z) from 2 and y respectively and combine them to generate R(x). as shown in Fig. 3. The following lemma compares [(R(z):)) y) with [x:; y) and analyzes how much identity infor- mation can be preserved in R(z).
R(x)
G(x) +
D(z)
Figure Data generation model.
Lemma 2. (Identiy-diisetanglle data generation). For data generative model described above, [((()):)) 2 [(x; v) (2;
The detailed proof of Lemma 2 is given in the appendix. Theorem 1.
data augmentation). If we use VAE in the identity-disentangled data generative model for Lemma 2, and if we define an augmentation x' R(x) + G' (z)x) with G' (0(x222) and 2' = + 8 (a 5 5-perturbed 2). we can lower bound I [xx
T(x':y) 2 [(:;1) B (TvAE + T(2;))).
(8)
Theorem can be proven by combing Lemma and Lemma 2, which is illustrated in the appendix.
Therefore, the augmentation x' preserves most of the identity information in x if LVAe + (2; is small. VAE training aims at minimizing LVAE the first term can be kept small. There is trade-off between the second term and identity preservation (():g) since a sufficiently large I(2; x) is necessary produce augmentation x' approximately drawn from the true data distribution. A key observation of the above theorem is that we can perturb 2 to generate without hurting the lower-bound of I (1); This implies that can adversarially perturb and positives for more efficient contrastive learning without heavily distorting the original identity information. In contrast, most data augmentation techniques used in supervuee learning have not taken this into account so they may change the sample-identig and result in poor representations.
produce hard negatives
3.4
ADVERSARIAL AUGMENTATION (IDAA)
As discussed in Sec. 2 and Sec. 3. keeping sufficient amount of hard positives and negatives in each batch x is critical to the effectiveness and sample efficiency of contrastive learning. Although any perturbation on z can produce an identity-pree ppreseredd data augmentation x' according to Theorem adversarially perturbing the VAR-botrlenee feature of the original samples can produce hard positives/negatives with sample-identiiie preserved. Meanwhile, we need to make sure that x' is still close to the original data manifold, e.g., an natural image, so x' needs to reside in the vicinity of xr Fortunately, we can use off-the-shelf adversarial attack algorithms for this purpose. The main difference here is: (1) we apply them to perturb the bottleneck features instead of x;; and (2) the objective is to maximize the InfoNCE loss instead of a classification loss such as cross entropy. We call this method
disentangled adversarial augmentation (IDAA)". as illustrated in Fig. 2.
In particular, the goal of IDAA is to generate an augmentation of x in the form of x' in Theorem I such that the InfoNCE loss is maximized using an e-bounded perturbation ö to F((x) when generating x'_ For a batch of data x, this problem can be formulated as optimizing { by
max LNCE(x'), x R(x,)) + D(E(a,)) + 6z).
Sr, Vee[N]
To only perturb the positives and negatives be harder for contrastive learning while keeping the anchors intact, we slightly modify LNCE(x') to be
exp h({))
LNCE(X')= = N
Σ log 1=1
(10)
Σ exp (f(x;)
5

---

Under review as a conference paper at ICLR 2022
Original Image:
Random Augmentation: A(x)
Output of VAE: G(A()))
rdeenit Part: R(A(x))
IDAA Perturbation:
CLAE Perturbation: (x) - A(x)
Figure 4: Comparing IDAA and CLAE on their adversarial augmentations for ImageNet samples.
Since the augmentation z' can be positive for its own identification (i.e., when x is the anchor) and a negative for other samples' identification, the objective in Eq. (9) modifies sample be both hard positive and hard negative. In other words, IDAA perturbs the bottleneck features to move r' away from x to other difference samples in the embedding space. Besides InfoNCE, loss, IDAA can be applied to other self-supervised losses, e.g., replacing (-CE(.) in Eq. (9) with the consistency regularization Lcs() (-) in Eq (2).
IDAA is complementary to and can be applied to existing data augmentation techniques by replacing J; in Eq. (9) with a pre-defined data augmentation A(x,). ). can also be applied together with other data augmentations, c.g., by combining multiple InfoNCE loss terms computed different augmentations. A primary advantage of IDAA is that the generated augmentations are adaptive to the training models f()) and h-)). It aims finding the weaknesses of the self-supervised models on the sample-identifictioo task and improve them without heavily distorting the original sample-identities, which is underexplored in previous literature.
There exists various off-the-shelf adversarial attack methods that can be directly applied to solve the problem Eq. (9). In this paper, we adopt fast gradient sign method (FGSM) (Goodfellow et al., 2014) for its computational efficiency. FGSM perturbs = = B(x) for one step by adding noises along the gradient sign't ' direction of the loss w.r.t. d and the augmentation z' is generated based on the perturbed bottleneck features + 6, i.e.,
x' = R(x)) + D (0)x))+))) 5" ==sign(V//xcc((x)
(11)
4 EXPERIMENTS
In this section, we evaluate the training efficiency and the test performance of several state-of-the-art methods in (1) self-supervised learning and (2) supervise learning when using (1) their default random augmentations; (2) default augmentations + CLAE (Ho & Vasconcelos, 2020); and (3) default augmentations IDAA. IDAA consistently improves both the training efficiency and the test accuracy of all the methods and outperforms the two baselines by large margins across several standard benchmark datasets. We also observe that IDAA trained on one dataset can be transferred to other datasets and improve downstream tasks them, implying promising generalization capability of the learnable augmentation model. Moreover, we conduct a thorough sensitivity study of IDAA when changing (1) batch sizes; (2) network architectures; (3) training epochs; (4) regularization weight 3 in the VAE objective; (5) dimensions of the VAE bottleneck features; and (6) adversarial attack strength bounded by 6.

---

Under review as a conference paper at ICLR 2022
4.1 CASE STUDY OF AUGMENTATION GENERATED BY IDAA AND CLAE
We firstly present a case study of augmentations generated by IDAA, CLAE, and widely used random augmentations. We compare their identity preserving. hardness to contrastive learning, and the patterns of perturbations created by them on natural images. For random augmentations, we choose RandomFlip, ColorJitter and GreyScale for their popularity in recent SSL. methods (Ye et al., 2019; Chen et al., 2020; Chen & He, 2021). In Fig. 1 (b)-(c), we compare the distance of an anchor to positive with its distance to the nearest negative in the embedding space we apply the contrastive learning. If the latter is smaller than the former, i.e., the point is located below the "identiy--reserving boundary" in the plots, its original identity is changed and cannot be preserved in the augmentation. On the other hand, if the latter is much larger than the former, ie., the point is located on the left region of the plots, the sample identification task is too trivial and the contrastive learning is not efficient. From the plots, we can see that random augmentations are much harder but cannot preserve the original identity for some samples. In contrast, IDAA (ours) produces sufficiently hard augmentations within the boundary of identity changing, which is ideal for contrastive learning.
too easy while the CLAE augmentations
We visualize the augmentations produced by the three methods in Fig. 4 for natural images from ImageNet dataset. Both IDAA and CLAE are applied to random augmentations A(x) and introduce further perturbations A'(x)= A(z). IDAA generates more semantic perturbations to important regions of the images because its adversarial attack is conducted in the VAE bottleneck space, which supposed to capture semantic attributes of the images. On the contrary, CLAE generates the perturbations on the input pixels and produces unnatural artifacts, which may distort the original sample-identic in the embedding space. In addition, we show the identity-relevant part R(A(z)) and the ddentity-disentang important patterns to identity the original sample, e.g., the edge of the steamship (column 2), spots of the frog (column 5) and pattern of the leopard (column 7). Since IDAA keep it intact in its augmentation model, the identity is well preserved in the augmentations.
part G(A(z)) produced by VAE. The former well preserves the most
4.2 SELF-SUPERVISED LEARNING
We evaluate data augmentations on four self-supervised learning methods: Plain (InfoNCE in Eq. (1), UEL (Ye al., 2019), SimSiam (Chen & He, 2021) and SimCLR (Chen al., 2020) on three datasets, i.c., CIFARIO (Krizhevsky et al., 2009), CIFARIO0 (Krizhevsky et al., 2009), and minilmagenet (Vinyals et al., 2016). The experimental details are given in Sec, A.3 in the Appendix.
Performance on Downstream Classification Tasks We firstly evaluate the representations of different SSL methods using different data augmentations on downstream classification tasks. In Table 1, we report the test accuracy of classifiers produced by two commonly used evaluation protocols, i.e., k-nearest neighbor (KNN) with k = 200 and linear evaluation that trains a linear classifier on top of the learned representations. IDAA consistently improves the original SSL meth- ods with their default random augmentations and the improvement usually much larger than that brought by CLAE. In contrast, though CLAE achieves better or comparable accuracy than the default aug- mentations on most contrastive learning methods, it results in sig- nificant degradation when applied to SimSiam, indicating that consistency regularization without comparison to negatives is more prone to potential identity distortion caused by adversarial augmen- tations in CLAE, which may strengthen the collapse of the training objective. Instead, IDAA exhibits better generalization to different types of SSL methods and evaluation protocols.
40
70-
40
FisMotch FiAM
10
AUR0 1858 70m 18000 Lubto LTURRaU
Figure 5: Convergence curve under 2500 labels on CIFAR100 for semi-supervised learning.
Transfer Learning Performance Different from random augmentations defined by human pri- ors, IDAA and CLAE depend on models trained on specific dataset. Hence, it is crucial to evaluate whether they can be transferred to other unseen datasets to improve downstream tasks. Following the linear evaluation protocol of Chen et al. (2020), we train encoders by IDAA and CLAE on ImageNet100 (i.e.. loo classes subset of Imagenet) and then train linear classifiers on 8 datasets (Krizhevsky et al., 2009; Berg et al., 2014; Maji et al., 2013; Cimpoi et al., 2014; Parkhi et al., 2012; Nilsback & Zisserman, 2006; Welinder et al., 2010) using the encoders' output representations. The test accuracy of these classifiers are reported in Table 2. On Imagenet100, "S
7

---

Under review as a conference paper at ICLR 2022
KNN
Linear Evaluation
Method
CIFARIO
CIFARI00
minilmageNet
CIFARIO
CIFARI00
minilmageNet
Plain
82.78=0.20
54.73+0.20
46.96:10.2
79.65=0.43
51.82+0.46
44.90=0.29
Plain+CLAB
83.09=0.19
55.28=0.12
47.01=0.228
79.94=0.28
02.14+0.21
45.43=+0.15
Plain+IDAA
86.00=0.166
58.6440.15
47.83:1:0299
8..83=+0.10
56.12::0.16
46.81=0016
UEL
83.63:10.14
55.23:10.28
40.71:10.73
80.63:10.18
$2.99110.25
43.08:10.35
UEL+CLAE
84.00=00.15
55.96=#0.06
41.75:1:0.39
80.94=1..13
54.27:10.00
44.32:+0.24
UEL+IDAA
86.69=00.13
59.0410.18
03.24:10.22
83.65:10.17
57.25+0.19
45.74:10.30
SimSiam
88.22-+0.10
57.13+0.20
31.68+00.2
89.84-00.15
62.76+00.13
40.62++0448
SimSiam+CL.An CLAA
85.59-0..21
53.88+00.08
27.77+33.47
87.77+00.08
60.89++022
37.32++0.47
SimAiam-IDAA
8..0
58.19+0.199
32.14+00.58
90.99++0188
65.21+
41.24++0.51
SimCLR
80.79=0.10
41.11=0.28
30.13:10.28
86,40110.18
57.81+0.100
46.13:10.23
SimCLR+CLAE
80.27=0.18
43.57=0.17
32.23+00.08
85.25+00077
57.69
16.76+00.16
SimCLR+IDAA
83.41=00222
46.78+0.22
33.66++0.16
88.07++.222
60.90++0.08
08.23-+0.23
Table 1: Test accuracy on downstream classification: two evaluations of representations learned by four self- supervised learning methods using different data augmentations, i.e., their default ones, CLAE and IDAA (ours).
CIFAR10
CIFAR100
Birdsnap
Aircraft
DTD
Pets
Flower
CUB-200
SimCLR SimCLR+CLAE
61.83
36.55
12.68
24.19
54.35
46.46
75.00
16.73
61.59
37.13
13.61
25.87
52.12
43.55
76.82
17.58
SImCLR+IDAA
64.49
38.82
13.89
26.02
54.97
46.76
77.99
18.15
Table 2: Transfer lcarning performance (test accuracy) on other datasets (trained on ImageNer100),
achieves 61. 1.9 KNN accuracy, which outperforms 56.38% of SimCLR by large margin. This im- provement can be transferred to the datasets. As shown the Table, IDAA consistently outperforms CLAE and improves the original SimCLR. while CLAE brings degeneration one dataset, i.e., Pets. We also compare their transfer learning results when applied to SimSiam in Sec. A.4 of the Appendix.
4.3 SEMI-SUPERVISED LEARNING
Data augmentation is also critical state-of-the- art semi-supervised learning algorithms such as FixMatch (Sohn et al., 2020), which relies on accurate pseudo labeling and confidence-based data selection and their quality heavily depends on data augmentations. We apply IDAA to FixMatch to train a a WettResNee-28-95 model
CIFAR100
Method
400 labels 2500 labels 10000 labels
Fixmatch
47.76
66.30
74.13
Fixmatch+CLAE
50.34
68.58
74.54
Fixmatch+IDAA
52.88
68.96
75.28
on CIFAR 100 with only {400, 2500, 10000} la- beled samples and the rest unlabeled. We train it for x 10a steps with batch size of 512. The test accuracy of the trained models reported in Table 3, where IDAA consistently improves FixMatch's 's accuracy and the improvement is more sig- nificant with fewer labeled data available. Moreover, shown in Fig. 5, IDAA significantly improves the efficiency and convergence of FixMatch and can save a great amount of computation to reach reasonable accuracy. IDAA keeps outperforming CLAE given different amounts of labeled data.
Table 3: Semi-supervised learning performance on CI- FAR100 with different amounts of data labeled.
4.4 SENSITIVITY ANALYSTS OF HYPERPARAMETERS
Batch size We evaluate how SimCLR and SimCLR+IDAA perform using four different batch sizes and the results are shown in Fig. 6(a). As verified by previous works (Chen al., 2020; He et al, 2020), increasing the batch size can improve SimCLR and other contrastive learning methods' performance, which also reflected in our results. However, IDAA can significantly improve SimCLR's performance under small batch size, e.g., 64, because it effectively modifies various samples to be hard positives/negaatves without distorting their original identities. Due to the same reason, SimCLR+IDAA is less sensitive to the change of batch size. It is also worth noting that SimCLR+IDAA using a very small batch size of 64 can surpass SimCLR with its best batch size of 512. This demonstrates the advantage of IDAA on improving the data/memory efficiency: with IDAA, contrastive learning is no longer limited to the usage of large batch size.
8

---

Under review as a conference paper at ICLR 2022
64
70.0
68
SimQLR+1DAA
67.5
SinCLR SAMCUR+1DAA
SimG.R
SIMCLR+HDM
Accurecy
38:41
62
00.81
60-
57.63
58
$5.0
56
52.5
54-
50
64
128
255
512
1024
2048
50.0
reshett8
resnetto ArchEectura
resnet101
52
200
400
600
800
1000
Batch size
Epoch
(a)
(b)
(c)
Figure 6: SSL performance under different (a) batch sizes, (b) ResNet architectures, and (c) training epochs.
63
SImCLR
Sintul SIMCUN
SamCL
SimnLLt-IDBA
62
Accuracy
61
Accural
50
60
60
58
57
5
0.1
3.0
10
57 12a
25g
512
1024
2048
57
0.35
0.10
0.15
0.0
a
VAE bottleneck dimersion
Attack strength
(a)
(b)
(c)
Figure 7: SSL performance using different (a) 8, (b) VAF bottleneck dimensions, and (c) Attack strength e.
Model Architecture We evaluate the linear evaluation performance of SSL when training different ResNet architectures in Fig. 6(b). Increasing the model size improves the performance of both methods but SimCLR+IDAA always outperforms SimCLR by a large margin. Hence, IDAA can improve SSL of different models and it can train smaller model with less costs to match the performance of training larger model by random augmentations.
Training epochs As shown in Fig. 6(c), the performance of both methods improves when investing more training epochs but the SimCLR+IDAA saturates much earlier and only spends 300 epochs to achieve comparable performance SimCLR trained using 1000 epochs. Hence, IDAA can greatly improve the training efficiency of SSL.
Regularization weight ,3 in VAE As revealed by Theorem 1, B in the VAE objective controls the lower bound of the identity information preserved in IDAA augmentations: larger 3 enforces more identity preservation in x' and stronger deentit--disentanlleenee
However, it also controls (2;xx) which reflects the proximity of z's distribution to the true data manifold of æ: large 3 leads to small [(2; 2) and less information of preserved in 2 that can be leveraged to produce stronger adversarial attacks for hard positives/negatives. In Fig. 7(a), we observe that the trade-off reaches a sweet spot at B = 0.1 among all the four 33 values between 0 and 10.
VAE bottleneck dimension As shown in Fig. 7(b), SSL performance with IDAA is not sensitive to the change of bottleneck dimension of VAE, though 51 512-dimension performs slightly better than other choices the experiment.
Attack strength Stronger attacks may produce more hard positives/negatives but also increases the risk of identity distortion and unrealistic augmentations biased from the true data distribu- tion/manifold. Results in Fig. 7(c) shows this trade-off and its effects on the SSL performance. Nevertheless, the performance of SimCLR+IDAA still quite stable and only varies a small range when changing since the identity-relevan information in R(xx) stays intact.
CONCLUSION
We propose a simple automatic aata-auumentation method IDAA, which can generate more informative but edentity-preserved augmentations to improve the efficiency and generalization of self- supervised learning. Motivated by an information theoretical analysis of VAE and identity preserving. IDAA adds adversarial noise to an perturbed VAE outputs with an intact identity-relevant part to produce augmentations. IDAA merely relies on pretrained VAE without requiring any labeled data but consistently improves diverse set of popular self suee-uuueediieeeeenn supeeiiie learning methods across multiple benchmarks. also enhances the transfer learning performance and significantly improves the learning efficiency.
space learned by VAE and combines the
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Alexander A Alemi, lan Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint 1612: 2016.
Thomas Berg, Jiongxin Liu, Seung Woo Lee, Michelle L. Alexander, David W. Jacobs, and Peter N. Belhumeur. Birdsnap: Large-scale fine- aen-ggrined visual categorization of birds. In Proc. Conf. Computer Vision and Pattern Recognition (CVPR), June 2014.
Emmanuel J Candès, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis? Journal of the ACM (JACM), 58(3):1-37, 2011.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 132-149, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint 00XY::2006.09882, 2020.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp.
1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Ching-Yo Yao Chuang, Joshua Robinson, Lin Yen-Chen, Antonio Torralba, and Stefanie Jegelka. Debi- ased contrastive learning. arXiv preprint arXiv:2007.00224, 2020.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
Ekin D Cubuk, Barret Zoph. Dandelion Mane, Vijay Vasudevan, and Quoe V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint 00X55, 2018
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Lee. Randaugment: Practical automated data augmentation with reduced search space. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Bin Dai, Yu Wang, John Aston, Gang Hua, and David Wipf. Connections with robust pea and the role of emergent sparsity in variational autoencoder models. The Journal of Machine Learning Research, 19(1):1573-1611 1614, 2018.
Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach. Adver- sarial continual learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pp. 386-402. Springer, 2020.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 1803.07725. 2018
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint orXiv:1412.651 2014.
Jean-Bastian Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint NxXiv:2006.07733 2020.
Kaiming He, Haoqi Fan, Yuxin Wu. Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
10

---

Under review as a conference paper at ICLR 2022
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. 2016.
Chih-Hui Ho and Nuno Vasconcelos. Contrastive learning with adversarial examples. arXiv preprini arXiv:2010. 12050, 2020.
Qianjiang Hu, Xiao Wang, Wei Hu, and Guo-Jun Qi. Adco: Adversarial contrast for efficient learning of unsupervised representations from self-trained negative adversaries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1074-1083, 2021.
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. arXiv preprint arXiv:2010.010088, 2020.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint
arXiv:1312.611 14, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Tongyu Liu, Ju Fan, Yinqing Luo, Nan Tang, Guoliang Li, and Xiaoyong Du Adaptive data augmentation for supervised learning over missing data. Proceedings of the VLDB Endowment, 14 (7):1202-1214, 2021.
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013.
Ishan Misra and I. Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6707-6717, 2020.
M-E Nilsback and Andrew Zisserman. visual vocabulary for flower classification. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 06), volume 2, pp. 1447-1454. IEEE, 2006.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. European conference on computer vision, pp. 69-84. Springer, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1807.0388, 2018
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012.
Joshua Robinson, Ching Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.0453 045992 2020.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi aaiisspervised learning with consistency and confidence. arXiv preprint arXiv:2001.07685, 2020.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision-ECCV 2020: 16th European Conference, Glusgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pp. 776-794. Springer, 2020a.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? arXiv preprint a0X14:2005.10243, 2020b.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29:3630-3638, 2016.
Welinder. S. Branson, T. Mita, C. Wah, Schroff, S. Belongie, and P. Perona. Maltech-UCSS) Birds 200. Technical Report CNS-TR-2010-001, Califomia Institute of Technology, 2010.
11

---

Under review as a conference paper at ICLR 2022
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-37722 2018.
Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan Yuille, and Quoc V Le. Adversarial examples improve image recognition. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 519-828, 2020.
Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. In Proceedings of the IEEE/CVT Conference Computer Vision and Puttern Recognition, pp. 6210-6219, 2019.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6002-6012 2019.
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 PROOF FOR PROPOSITION
Proposition 1. (Sample-identffiction likelihood lower bound of T(x; y)). If is random mini-batch of size and the y i be ply i| Fi), the mutual information (2;; y) can be lower bounded by
likelihood of D; on its correct identification label
T(:;y) 2 log N Ex
logp(y f/ zi)
(12)
Proof. To relate mutual information and sample-identification task, we take the expectation by first sample mini-batch x = {21, (2,, "2,""* : [ of size N and then sample pairs (z, y) from the mini-batch. Then have:
(x, v) (v) (g]=))
N
= loog N Ex
Σ
dxply, x) logp(y(x)
y=1
=log 1xx
Σ
(xp(:))(o/=))
N
2 log N + Ex
Σ
N
= log N Ex
Σ
dxp(y, x) log ((yl))))))))))))))()))))))))<<
(13)
N
-logNN N 1x
Σ
dxp(=ly))(v) loga(u1z))
N
logg N Ex
Σ
dxp(=/l= i)p(y i) log i|xx)
N
llogNN N Ex
Σ
dxo(x (I - -x,) 1 log q4v )|0)
N
= log N Ex
Σ
log q(v 1| = x3.)
N
where p(-) denotes the true probability and q(.) denotes arbitrary estimation. The inequality in O.
Equation (13) comes from DRI. (p)))))))))
A.2 PROOF FOR THEOREM 1
We start by proving Lemma 2 and then we can prove Theorem by combing Lemma and Lemma 2. Lemma 2. (Identity-disentangled data generation). For data generative model described above, T(R(x):)) 2 T(x;v) T(2;
(14)
the data generative model in Fig. 3, we have [(: v) [(R(z), z; = (((())))) [2: By the definition of conditional mutual information, we have
Proof. Due to the Markov chain y - (R(x),
(15)
[(z; R())) = (2; + (R())))z - H (R())) + [H(((z)lv) - H (R(x)ly, Since 1 R(x) in the generative model, the last two terms the above equation are zeros and T(2; y [())) T(2; v). Substituting it to Eq. (15) completes the proof.
(16)
13

---

Under review as a conference paper at ICLR 2022
CIFARIO
CIFARI00
Birdsnap
Aireraft
DTD
Pets
Flower
CUB-200
SimSiam 49.75
25.59
7.32
16.29
40.07
31.33
58.20
9.96
SimSiam+CLAL 53.05
26.78
8,80
20.01
43.97
33.62
63.54
12.39
SimSiam+IDAA 55.74
29.63
9.16
20.16
47.32
36.22
65.63
12.84
Table 4: Transfer learning performance (in test accuracy %) on 8 datasets for CLAE and IDAA applied to SimSiam (pretrained on ImageNet100).
Theorem 1. Iddenitt-diieenanglee data augmentation). If we use VAE in the identity-dienttangle data generative model for Lemmn 2, and if we define an augmentation x' = R(x) + G' (2) with G' (z) (x(222)) and z' + 8 (a ó 5-perturbed 2), we can lower bound I [xx v)
1/ (LVAE ((;2))).
1
M(': (x: v)
(17)
3
Proof.
Applying the results from Lemma and Lemma we have
[(((z) + G'():)) (R(()))) 2 [(x; - 1 (23 v) 2 1 (z; 3 (LVAE + (z;
(18)
The first inequality is due to the identifiability assumption extend from Theorem 1 Robust PCA Can- dès et al. (2011), i.c., the identity-disentanllee disentangled part G(z) and the identity-revelant part R(z) are separable give æ. Under this assumption, we have the conditional entropy H (R(x)) [(x) ((x)++((()) =
because R(x) + (z) can be separated into R(x) and G' (z) again in an unique and exact way using VAE. This is a natural extension because VAE is an extension of robust PCA, as pointed out by Dai
et al. (2018).
A.3 EXPERIMENTAL DETAILS OF SELF-SUPERVISED LEARNING
We evaluate data augmentations on four self-supervised learning methods: Plain (InfoNCE in Eq. (I), UEL (Ye et al., 2019). SimSiam (Chen & He, 2021) and SimCLR (Chen et al., 2020) on three datasets, i.e., CIFAR10 (Krizhevsky al., 2009), CIFAR100 (Krizhevsky et al., 2009), and minilmagenet (Vinyals et al., 2016). The four SSL methods cover different contrastive learning methods (Plain, UEL, SimCLR) and consistency regularization based method (SimSiam). The training/test splitting of minilmagenet follows Ebrahimi et al. (2020). Unless otherwise noted, ResNet-18 18 with two separate BNs (Xie al., 2020) for clean and adversarial samples utilized as the representation encoder. The regularization weight 3 in VAE is set to 0.1 and the bottleneck dimension is 512(3072) for CIFARIO/CIES I miiiimmmage)))).)
A.4 TRANSFER LEARNING PERFORMANCE USING SIMSIAM AS BASELINE
We further add new experiments of SimSiam, SimSiam+CLAE, and SimSiam+IDAA. and report their results in Table 4 as an extension Table 2. Unlike SimCLR, SimSiam Chen & He (2021) adopts another popular idea of self-supervised learning based on the consistency regularization and Siamese network.
We evaluate the three methods on ImageNet- 100 we did for SimCLR, i.c., by running each method for 100 epochs and evaluating their transfer learning performance on the 8 datasets as in Table Similar to SimCLR, IDAA consistenly improves the transfer learning performance of SimSiam and outperform CLAE on all the datasets.
We select SimCLR and SimSiam for this study because they represent the two most widely studied self-supervised learning strategies, i.e., contrastive learning and consistency learning respectively.
14

---

Under review as a conference paper at ICLR 2022
A.5
NEW BASELINE: DATA AUGMENTATION WITHOUT DECOMPOSITION (G(2") ONLY)
Here we compare with an additional baseline: simply attacking the z of a standard VAE (with beta=l) to produce an x' = G(z') as augmentations for contrastive learning. We report its results on CIFAR in Table 5, denoted as "SimCLR=IDAA(NW"" decomposition)". It shows that the performance significantly declines once the decomposition removed and it performs even worse than the original SimCLR.
Therefore,
is critical and preserving the identity-relevan part R(x:) in augmentations is essential to seff-suue learning. This is illuestrated in Figure 1. On the contrary, we simply use the Dhentity-distorier part G(2') for augmentation with R(z) removed, the positive and negative assignments in contrastive learning can be wrong. With many wrong identity labels, the identification task of contrastive learning can easily fail, resulting in poor representations and performance degradation on downstream tasks.
kNN
Linear Evaluation
Method
CIFARIO
CIFAR100
CIFARIO
CIFARI00
SimCLR
80.79
41.11
86.40
57.81
decomposition)
67.25
32.77
75.82
47.41
SimCLR+IDAA
83.41
46.78
88.07
60.90
Table 5: Test accuracy on downstream classification tasks: comparing representations learned with (x' = G(2z) + R(x)) and without dlentit--disentanglee decomposition (a' G(2')).
A.6 EXPERIMENTS ON LOWER RESOLUTION VERSION (64 x 64) OF IMAGENET- 1K
We add new experiments that train a ResNet- 18 on ImageNet 64x64 by SimCLR, SimCLR+CLAE and SiMCLR+IDAA. We run each method for 100 epochs with a batch size of 1024. We report the results in Table 6. It shows that SimCLR+IDAA still outperforms SimCLR+CLAE and original SimCLR this more challenging dataset. Moreover, IDAA can improve both the kNN and linear evaluation accuracy while CLAE can only improve the kNN accuracy.
kNN
Linear Evaluation
SimCLR
14.72
30.47
SimCLR+CLAE
15.44
28.77
SimCLR-IDAA
16.24
30.96
Table 6: Test accuracy of downstream classification on ImageNet 64 64.
A.7 COMPARISON WITH MORE BASELINES
In this section, we compare IDAA with two additional baselines published recently, i.e., InfoMin (Tian et al., 2020b) and Debiased (Chuang et al., 2020). InfoMin (Tian et al., 2020b) studies how to obtain good augmentation for contrastive learning, here we compare IDAA with InfoMin (Tian et al., 2020b) using their official code' on ImageNet they used in their paper (we reduced the resolution to 64 x 64 due to our limited computational resources). We apply each method to train ResNet-18 18 for 100 epochs and compare their linear evaluation accuracy on the test set. The results reported in the Table 7, which shows that IDAA outperforms InfoMin (Tian et al., 2020b) by a large margin.
Debiased (Chuang al., 2020) proposed new objective, which aims avoid sampling negatives of the same class as the anchor in contrastive learning. Our method as an augmentation strategy is complementary to Debiased. We add new experiments comparing with Debiased using their official
15

---

Under review as a conference paper at ICLR 2022
Linear Evaluation
InfoMin (Tian al., 2020b)
27.11
SimCLR+IDAA
30.96
Table 7: Comparison with InfoMin for downstream classification on Imagenet 64 x 64.
code2 on CIFARIO, the dataset used in their original paper. Specifically, we apply Debiased and "Debiased+IDAA" to train ResNet-18 18 for 300 epochs and compare their kNN an linear evaluation accuracy. The results are reported in Table 8, which shows that Debiased outperforms SimCLR but IDAA further improve Debiased when applied Debiased. Moreover, only "SimCLR+IDAA" can outperform Debiased by a large margin, i.c., 1.6% linear evaluation accuracy, which clearly demonstrate the effectiveness IDAA.
kNN
Linear Evaluation
SimCLR Debiased(Chuang al, 2020)
80.79
86.40
81,96
86.47
SimCLR+IDAA
83,41
88.07
Debiased(Chuang al, 202A
84.44
88.55
Table 8: Copmarison with Debiased for downstream classification on CIFARIO.
16


Reviews:
[{"review_id": "jF91lTdkfEK", "reviewer": "Reviewer_kfHb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work studies an important area of machine learning that is important to a vast literature in self-supervised learning. In particular, they propose a method that utilizes the strengths of deep variational autoencoders and their information theoretic properties to inherit identity disentanglement. \n\nThe main strengths of the paper is the utility and simplicity of such a method. As illustrated on the extensive applications, the method is readily applicable to schemes and I see the work presented in this paper to be extremely useful for the machine learning community at large for applications in data augmentation beyond those described in this work. In particular, this work may be useful for adversarial training via augmentation, which is a closely related area and given the 'adversarial' motivation hinted in this paper, can be developed into such a scheme.\n\nThe simplicity of the paper also serves to some extent as a weakness. This is due to the fact that there are several questions that may be asked albeit not taking too much merit away from the idea. The first is how specific is this method to the VAE? While the main reason for using VAEs is the link to information theory (the appearance of mutual information), which is the reason the main theorems work, I wonder how applicable this result would be at least heuristically with other deep generative methods such as the Wasserstein Autoencoder for example. I think such avenues can be deferred to future work however I would be interested to know the author's position on this.\n\nThe paper is very well presented and I found the level of exposition very high with little or no issues understanding the contribution of this work. I am a non-expert in this area however one small question that does emerge is the motivation of using an adversary for this task. While I understand the adversary takes the role to strengthen the identity preservation, I do feel that this paper may be conflated with adversarial training, which as mentioned earlier is a potential application yet not explored. This is however a minor point as a non-expert, it may be a common phrase used for disentanglement.", "summary_of_the_paper": "This work presents a new technique for disentanglement based inspired by the premise of an adversary that generate diverse samples that preserve identities via augmentation. The main motivation of the paper is for self-supervised learning which is an important area that benefits a lot from data augmentation. In particular, they use the reconstructed sample of a VAE and its distorted difference (G(x) - x) as an adversary to perform data augmentation and show its effectiveness via success on a number of different self-supervised learning tasks.", "summary_of_the_review": "I believe the paper presents a new method and serves to a useful problem of machine learning for a number of tasks. Due to the simplicity and utility of the method, I lean towards acceptance of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LM2f_MuGzbg", "reviewer": "Reviewer_rByQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea to disentangle the identity with VAE is technically correct and the analysis in section 3 with the information bounds and ELBO provides a clear theoretical support method. The proposed IDAA seems to be able to improve the current augmentation strategy by removing the identity attributes when generating augmentations. A weakness here is IDAA seems to rely upon existing pre-defined augmentation combinations. The authors better tune down their claim or provide more support that IDAA can achieve on par performance even with random augmentation or without any augmentation. \n\nAlthough I like the analysis in section 3, I still have some concerns about the claim that IDAA can improve different SSL methods. As known recent SSL methods can be generally classified as contrastive and non-contrastive ones. The contrastive methods, such as SimCLR, mainly use InfoNCE loss, and the analysis in section 1 well supports the relation of IDAA to these methods. However, the relation to non-contrastive methods like SimSiam, SWAV, etc. is not clear since they do not use InfoNCE loss. Thus, I am curious whether IDAA really improves the performance of these methods. If the authors would like to claim IDAA generally improves existing SSL methods, I think it is also necessary to show how IDAA works with recent SSL methods such as MoCo, SWAV, BYOL, DINO, etc. \n\nI also have two main questions about the experiment part:\n\n- The results reported in the paper seem not consistent with existing papers in SSL area. For example, SimCLR and SimSiam are known to achieve over 90% accuracy in linear classification when ResNet18 backbone with 800 epochs, and the results in Table 1 seem to have significant differences. In Table 1 CLAE seems to harm the performance of SimCLR and SimSiam, which is not consistent with the conclusion in their paper either. Based on these results, it is hard to judge whether IDAA improves existing SSL methods' performance and outperforms other augmentation strategies.\n\n- According to $\\beta$-VAE, the disentanglement is affected by choice of $\\beta$, and they have better disentanglement in the latent space when $\\beta$ is bigger than 1. According to Eqn (5) in the paper, it also seems the identity is better preserved from wrong augmentation when $\\beta$ is bigger (please correct me if I am wrong). The result in Fig. 7 seems to suggest the best choice is $\\beta=0.1$, which is a little bit out of my expectation and I am worried this could violate the lower bound shown in Eqn (8). Could the authors provide additional justifications with some analytical examples or provide some possible explanation?", "summary_of_the_paper": "This work presents an augmentation strategy leveraging the latent space of VAE and adversarial samples, named Identity-Disentangled Adversarial Augmentation (IDAA). IDAA perturbs the latent space of a pretrained variational auto-encoder to produce adversarial samples that preserve the identity attributes. Experiments show that IDAA improves the performance of existing SimCLR and SimSiam baselines. ", "summary_of_the_review": "This paper provides a novel augmentation strategy that leverages the latent space of pretrained VAEs to produce adversarial samples. The technique part is generally correct but is limited to InfoNCE loss, and the experiment results are not completely consistent with existing papers. Based on these concerns, I had difficulty in judging the value of the proposed method. I am currently inclined to recommend a reject and I suggest the authors keep the same configuration of the baselines to justify the improvement of their method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "K9mdsopO0W1", "reviewer": "Reviewer_Augv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Finding better augmentation schemes for self-supervised learning is undoubtedly and important and challenging problem. While the proposed method does seem to offer empirical gains with respect to baselines, I am not completely convinced about the motivation and some of the theoretical claims.\n\nIn particular, in Theorem 1, the authors claim that they can perturb z to produce x' without hurting the lower bound. However, it seems odd that this theorem does not have a dependency on $\\delta$ (that is used to perturb z). After all, one could construct a z' that is arbitrarily different from z (using a large delta). In this case, x' could be very different from x and I(x', y) would be very small (possibly violating the lower bound). The proof utilizes I(R(x) + G'(x); y) ≥ I(R(x); y), without explaining why this is the case. Overall, I do not see how this is true for arbitrary $\\delta$.\n\nOther comments:\n\n* The claims made in Section 4.1 regarding the qualitative differences between augmentation samples (in Figure 4) are subjective and hand-wavy. Based on visual inspection alone, the difference in quality between these augmentations is not apparent and thus should be removed.\n* Baselines:\n    - Why do the authors only consider SimCLR in Table 2?\n    - In Table 3, why are CLAE and random augmentations missing?\n    - I am curious whether simply attacking the z of a standard VAE (with beta=1) to produce an x' = G(z') would lead to vastly different performance. In general, I am not convinced that the decomposition into the generated and residual components is necessary. This seems like an essential baselines to compare to.\n* All the plots in the paper need to be made larger and more legible.", "summary_of_the_paper": "This paper proposes a novel augmentation method for contrastive SSL. To this end, the authors leverage VAEs—generating an augmentation for an image x by adversarially perturbing it's encoding (z = E(x)) to generate a perturbed reconstruction (G(z')), which is then added back to the original residual (R(x) = x - G(x)). The motivation for this approach is to maintain the identity preserving information via R(x), while creating challenging augmentations. The authors empirically evaluate their approach for downstream classification, transfer learning and semi-supervised learning.", "summary_of_the_review": "While the empirical results of this paper are interesting, I still have concerns about the theoretical claims and the comparison to baselines. I would be happy to increase my score if the authors addressed these.\n\n\n#### Post-rebuttal Update ####\n\nI thank the authors for their detailed response. Based on the additional experimental results that the authors presented to all the reviewers, I have decided to increase my score to a 6. I still have two comments:\n\n- The authors' response regarding theorem 1 does not completely address my concern. Theorem 1 as stated right now allows z to be perturbed by an arbitrary $\\delta$, regardless of x. (In contrast, in adversarial training x itself is perturbed by $\\delta$ which is constrained by some $\\epsilon$.) Thus, it is still not clear how the bound in Theorem 1 can hold for some arbitrary $\\delta$. For instance, imagine we have two images $x_1$ and $x_2$ and their corresponding latent $z_1$ and $z_2$. I could simply define $z_1' = z_2$ by selecting $\\delta = z_2 - z_1$. In the case, I would expect that the bound is violated. I believe that there should either be an assumption on $\\delta$, or it should feature in the resulting bound.\n\n- In the caption for Table 8, there is a typo: `copmaring`.\n\n- The authors should incorporate all the additional results they reported during the rebuttal into the paper (or appendix).\n\n\n#### Post-discussion Update ####\n\nOver the course of the discussion between the authors and the reviewers, it has become apparent that several of the experimental settings chosen in the paper are non-standard, which make it hard to verify the validity of the results. I do share the concerns of my fellow reviewers in this regard, and thus will downgrade my score to a 5.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ISJzX-QoOsu", "reviewer": "Reviewer_dMSm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n+ The idea to decouple the identity from view generation seems to be novel and not explored in the CL literature before.\n+ The presented method is simple and straightforward. I can see the potential of extending such formulation to a broader range of tasks and data domains.\n+ The evaluation is relatively thorough, spanning across self-supervised & semi-supervised settings, and demonstrates the improvement over the vanilla and the CLAE method.\n\n## Main Weaknesses\n1. __Lack of comparisons to methods within the same category__ \n- The comparison is rather limited. The authors only compare to the vanilla (no adversarial augmentation introduced) and one method based on adversarial augmentation (CLAE). However, the basic idea is still _how to generate effective views / hard negative or positive samples_. Such problems have been extensively studied in the literature for CL [1-5]. How does IDAA perform against these methods? If not thoroughly evaluated, the performance gains are not justified and hence not interpretable.\n\n2. __The definition of \"identity-preserving\"__ \n- In Fig.1, how is the distance measured? Is it measured in the feature space? How is the \"Identity preserving boundary\" defined? For CLAE, if the noise added to the input is bounded by a l_inf purturbation, then by definition the input should preserve the identity of the image. The information that this figure delivers is unclear.\n- Continue on this point, to me it is very confusing on how you define identity-preserving. In fact, from Figure 4 in the paper, all augmentation methods actually preserve the identity. This also holds for CLAE. The only difference is that the perturbation generated by IDAA is more interpretable, but this has nothing to do with \"identity-preserving\". The augmented images still preserve their identities.\n- Further, the motivation is not well explained. Given that the visualization does not directly reflect the point of \"identity-preserving\", the claims that current methods/augmentations \"could change sample-identity and result in poor representations\" is not justified.\n\n3. __The IDAA is not self-contained as universal augmentation, but still needs a base set of augmentations__\n- This might be a smaller issue, but IDAA still requires base augmentations (i.e., color jittering or random cropping/resizing) to work. In the experiments, all results of IDAA are based on some exsiting augmentations (e.g., those from SimCLR). The description of IDAA tends to sell it as a \"universal\" augmentation methods regradless of inputs, but domain knowledge (e.g., how you augment image data) is still needed from experiements. What is the actual performance of IDAA without using existing augmentations (those in SimCLR)?\n\n## Other issues / questions\n- Since the overall objective (i.e., generating better views for contrastive learning) is agnostic to the data modality, is the proposed scheme also be extended to other data domains beyond images, like time series / texts? What would be the most challenging parts when extending to other modalities?\n- What is the actual computational cost (training time) compared to the vanilla SimCLR?\n\n## References\n[1] Hard Negative Mixing for Contrastive Learning.\n\n[2] Adco: Adversarial contrast for efficient learning of unsupervised representations from self-trained negative adversaries.\n\n[3] What makes for good views for contrastive learning.\n\n[4] Debiased contrastive learning.\n\n[5] Contrastive learning with hard negative samples.", "summary_of_the_paper": "The paper studies contrastive self-supervised learning. In particular, it proposes an adversarial augmentation method for different view generation in contrastive learning. It claims that using such adversarial mechanism, the generated views are identity-preserving while being hard positives/negatives. Empirical experiments on several benckmarks validate the improved performance and efficiency.", "summary_of_the_review": "Overall, the idea is interesting, however the current draft is not well-justified on the actual algorithm design, the motivation, and (perhaps the most important) empirical comparisons to existing CL methods that considers hard view mining. Further comments/questions are listed in the weaknesses / questions part.\n\nThe paper has its potential to the field, but issues need to be addressed first. I'm happy to change my score if the feedback addresses my concerns. Please refer to the points in the weaknesses / questions part. I would like to see feedbacks on these comments/questions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IDaRv6sf7v3", "reviewer": "Reviewer_uy6S", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- The idea is novel;\n- The main hypothesis that other augmentation strategies result in pair of samples that are either too easy or run the risk of corrupting the instance-level information is backed up by empirical evidence;\n- This work provides a information theory based interpretation of the IDAA strategy with sufficient proofs;\n- The claim that better hard positives/negatives result in more sample-efficient models w.r.t the batch size and the number of epochs is highlighted through various ablations;\n- IDAA provides significant performance improvements on all reported benchmarks;\n\nWeaknesses:\n\n- Skipping the conclusion is not an acceptable practice to gain more space. One could move some of the training hyperparameters to the appendix to gain some extra space in the main paper for that;\n- The ablation on the batch size raises the concern that IDAA does not scale up too while. It is unclear whether SimCLR will surpass SimCLR+IDAA with a sufficiently large minibatch. Indeed, we observe that increasing the batch size is beneficial for SimCLR only on Figure 6-a;\n- The datasets explored in the experiments of this work are somewhat limited in terms of diversity. Training a contrastive self-supervised model on the full imagenet dataset can be resource intensive, however the authors could have explored the use of lower resolution versions of it while keeping all the classes (e.g. ImageNet 64x64);\n\nMinor improvements:\n- Typo in the introduction: 'ullustrated in Fig. 2'", "summary_of_the_paper": "This work introduces a novel adversarial augmentation strategy. This augmentation, nammed Identity-Disentangled Adversarial Augmentation (IDAA) aims at producing better hard positives/negatives while preserving the identity of the sample. This is achieved by perturbing the latent space of a variational auto-encoder while preserving the identity relevant features through the residual term. This allows for the training of better and more sample efficient self/semi-supervised.", "summary_of_the_review": "This work introduces a novelty augmentation strategy with sufficient empirical and theoretical arguments. Although some concerns remain about its scalability to larger batch sizes and more challenging datasets, the authors clearly show significant performance improvements when using IDAA across all experimental settings and empirically justify all claims/hypotheses made.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: To improve the data augmentation for an existing self-supervised learning framework, the paper presents identity-disentangled adversarial augmentation (IDAA) that utilizes a pretrained VAE and adversarial perturbation in the VAE latent space to generate identity-preserving hard negative/positive samples. The proposed method has been clearly described and is of interest to the community. A wide variety of experiments have been done to illustrate the effectiveness of IDAA. Given the concerns on the validity and generalizability of the experimental results presented in the paper, the AC does not consider the paper of its current form to be ready for publication, as discussed below.

The AC appreciates the amount of effort that the authors have spent responding to the reviewers. However, thees very detailed rebuttals do not appear to be that effective in directly answering several key concerns of the reviewers.

1. While many experiments have been added to improve the paper, most of the settings are still considered by the reviewers to be unconvincing due to the use of a small number of epochs, unusual network backbones, and/or non-standard datasets (e.g., downsampled 64*64 ImageNet with about 30% accuracy).

2. The comparison to the literature is considered to be insufficient. While the basic idea is still how to generate effective views or hard negative/positive views, the paper chooses to narrowly focus its comparison with CLAE that relies on adversarial perturbation in the pixel space, ignoring a careful comparison with many existing methods that boost the performance by mining hard negatives/positives and/or using a memory bank to accommodate a large number of negatives. 

3. Related to point 2, a main justification of IDAA is that it outperforms CLAE, which, however, is shown by the authors to be a weak baseline that barely improves an existing self-supervised learning (SSL) framework in a variety of different settings. For example, the results in Table 1 show CLAE hurts SimCLR's performance. The weak performance of CLAE makes it even more important to include stronger baselines.

4. A reviewer also pointed out that the use of a pretrained VAE introduces additional model parameters and increases the computation cost, and hence a careful discussion on how to ensure a fair comparison with the baselines is desired. 

5. The presentation could be somewhat misleading in giving the impression that IDAA is a stand-alone domain-agnostic data augmentation technique. While it is totally fine that IDAA is a complementary data augmentation technique, the authors need to carefully revise the presentation to minimize the risk of giving the reader a wrong impression.  

6. The authors dismissed some of the negative reviews by arguing them to be “mostly incorrect, wrong, makes no sense”. These strong arguments, if not supported with clear evidence asked by the reviewers, may convince neither the reviewers nor the AC and lead to unnecessarily tedious lengthy discussions. From the AC's reading of the paper and reviews, it appears that there are several cases where a reviewer was asking the answer to Question A, but the authors were providing an answer to Question B and pushing the reviewer to accept that as a satisfactory answer to Question A. For example, while many experiments have been done on several different datasets, there is a legitimate concern on why these computing resource has not been spent on getting results on ImageNet with ResNet 50. When the experiments are only being done on ImageNet-100 or the downsampled full ImageNet with a small network, there could be clear concerns on whether the observed performance gains are only applicable to relatively small/low-resolution datasets or small networks. For example, the VAE and/or the adversarial attack may not work that well on large-scale/high-resolution images and hence IDAA may break down when scaling it to a larger dataset/model. That type of concern can only be addressed if the authors have taken the time to follow a standard setting, such as ResNet50 on ImageNet, and make comparisons with a wide variety of baselines whose results on these standard settings are readily available.
Decision: Reject