Adversarial perturbation based latent reconstruction for domain-agnostic self-supervised learning
Authors: Sijie Tian
Kuilin Chen
Chi-Guhn Lee
Venue: Submitted to ICLR 2023 (2023)
Keywords: self-supervised learning
representation learning
domain-agnostic

Abstract: Most self-supervised learning (SSL) methods rely on domain-specific pretext tasks and data augmentations to learn high-quality representations from unlabeled data. Development of those pretext tasks and data augmentations requires expert domain knowledge. In addition, it is not clear why solving certain pretext tasks leads to useful representations. Those two reasons hinder wider application of SSL to different domains. To overcome such limitations, we propose adversarial perturbation based latent reconstruction (APLR) for domain-agnostic self-supervised learning. In APLR, a neural network is trained to generate adversarial noise to perturb the unlabeled training sample so that domain-specific augmentations are not required. The pretext task in APLR is to reconstruct the latent representation of a clean sample from a perturbed sample. We show that representation learning via latent reconstruction is closely related to multi-dimensional Hirschfeld-Gebelein-Rényi (HGR) maximal correlation and has theoretical guarantees on the linear probe error. To demonstrate the effectiveness of APLR, the proposed method is applied to various domains such as tabular data, images, and audios. Empirical results indicate that APLR not only outperforms existing domain-agnostic SSL methods, but also closes the performance gap to domain-specific SSL methods. In many cases, APLR also outperforms training the full network in a supervised manner.

Under review as a conference paper at ICLR 2023
ADVERSARIAL PERTURBATION BASED LATENT RECON- STRUCTION FOR DOMAIN-AGNOSTN SELF-SUPERVISED LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Most sulf-sup learning (SSL) methods rely on domain-specific pretext tasks and data augmentations to learn high-quality representations from unlabeled data. Development of those pretext tasks and data augmentations requires expert domain knowledge. In addition, it is not clear why solving certain pretext tasks leads to useful representations. Those two reasons hinder wider application of SSL to dif- ferent domains. To overcome such limitations, we propose adversarial perturbation based latent reconstruction (APLR) for domain-agnostic self-supervised learning. In APLR, a neural network is trained to generate adversarial noise to perturb the unlabeled training sample so that domain-specific augmentations are not required. The pretext task in APLR is to reconstruct the latent representation of a clean sample from a perturbed sample. We show that representation learning via latent reconstruction is closely related to muuti--immensioall (HGR) maximal correlation and has theoretical guarantees on the linear probe еrror. To demonstrate the effectiveness of APLR, the proposed method is applied to various domains such as tabular data, images, and audios. Empirical results indicate that APLR not only outperforms existing domain-agnosti SSL methods, but also closes the performance gap to domain-speciffe SSL methods. In many cases, APLR also outperforms training the full network in supervised manner.
Serrchfeld-Gebelein--
INTRODUCTION
Unsupervised deep learning has been highly successful in discovering useful representations in natural language processing (NLP) (Devlin et al., 2019; Brown et al., 2020) and computer vision (Chen al., 2020; He al., 2020). These methods define pretext tasks on unlabeled data so that unsupervised representation learning can be done in self-supervised manner without explicit human annotations. The success of self-suuprvvsed learning (SSL) depends on domain-specific pretext tasks, well as Nomainn-peecine data augmentations. However, the development of semantic-pre semantic-preserving preserving data augmentations requires expert domain knowledge, and such knowledge may not be readily available for certain data types such tabular data (Ucar et al., 2021). Furthermore, the theoretical understanding of why certain pretext tasks lead to useful representations remains fairly elusive Tian al. (2021). Those two reasons hinder wider applications of SSL beyond the fields of NLP and computer vision.
Self-supervised algorithms benefit from inductive biases from domain-specific designs but they do not generalize across different domains. For example, masked language models like BERT (Devlin al., 2019) are directly applicable untokenized data. Although contrastive learning does not require tokenized data, its success in computer vision cannot be easily leveraged in other domains due its sensitivity to image-specific data augmentations (Chen et al., 2020). Furthermore, in contrastive learning, the quality of representations degrades significantly without those hand-crafted data augmentations (Grill et al., 2020). Inspired by denoising auto-encoding (Vincent et al., 2008; 2010; Pathak et al., 2016), perturbation of natural samples with Gaussian, Bernoulli, and mixup noises (Verma et al., 2021; Yoon et al., 2020) has been utilized as domain damain-agnostic data augmentations applicable for self-supervised representation learning of images, graphs, and tabular data. However, random noises may not be as effective since uniformly perturbing uninformative features may not lead to the intended goal of augmentations. Specifically, convex combinations in mixup noises (Zhang

---

Under review as a conference paper at ICLR 2023
al., 2018; Yun et al., 2019) could generate out-of-distributinn samples because there is no guarantee that the input data space is convex (Ucar et al., 2021). In this article, we use generative adversarial perturbation as semantic-preseet preserving data augmentation method (Baluja & Fischer, 2018; Poursaeed et al., 2018; Naseer et al., 2019; Nakkal & Salzmann, 2021) applicable to different domains of data. Adversarial perturbation is constrained by the lp norm distance the natural sample that it is semantic-preservve and does not change the label (Goodfellow et al., 2015; Madry et al., 2018).
With remantic-preserving perturbation. the pretext tasks in domain-agnostic SSL could be reconstruc- tion of clean samples (Yoon et al., 2020) or instance discrimination of perturbed samples (Verma al., 2021). Nevertheless, reconstruction of clean samples in the input space is computationally expensive because the input data dimension is high. Therefore, we present adversarial perturbation based latent reconstruction (APLR), a simple and intuitive domain-agnostie self-supervised pretext task derived from linear generative models, to learn representations from unlabeled data in domain- agnostic manner. Contrary to the pretext task of instance discrimination, our method does not require comparison to a large number of negative samples to achieve good performance. The proposed APLR not only achieves strong empirical performance on SSL in various domains, but also has theoretical guarantees the linear probe error downstream tasks.
The contributions of this article are summarized below:
We present adversarial perturbation based latent reconstruction for domain-agnostic self- supervised learning.
The proposed APIR achieves strong linear probe performance on various data types without using domain-specific data augmentations.
We provide theoretical guarantees on the linear probe error on downstream tasks.
2 BACKGROUND
Learning representations from two views of an input, x' and x², is appealing if the learned representations do not contain the noises in different views. This assumption can explicitly encoded into the following generative model (Bach & Jordan, 2005) with one shared latent variable z:
p(z) = N I)
pxx² z) = - (v z, E!)
xxx 2) N (n 2, E²)
where the model parameters vb, n. Σ' and Σ² can be learned by maximum likelihood estimation. Reconstruction of input data via maximum likelihood estimation is computationally expensive when the dimension of the input data is high. Instead, the probabilistic generative model can be reinterpreted as latent reconstruction, which has the benefit of direct representation learning while retaining the properties of generative modeling.
To convert generative modelling into latent reconstruction, two assumptions need to be met. First, the assumption in generative modeling is that both datasets have similar low-rank approximations. In latent reconstruction, this can be achieved by correlating the pair of latent embeddings vxx² and ypx?. Secondly, it is assumed in generative modelling that the latent variables follow isotropic Gaussian distribution p(z)= = / (0, I). In latent reconstruction, the covariance matrix of the latent variables is diagonal, meaning that there is covariance between different dimensions of the latent variable. This orthogonality constraint is equivalent to the assumption of isotropic Gaussian prior and avoids the trivial solution. As a result, by correlating the latent embeddings v/x² and rpx² and enforcing a diagonal covariance matrix, the properties of generative modelling can be retained in latent reconstruction.
The key principle behind latent reconstruction is that the latent representation of x is good predictor for that of x² Given two datasets X and X of N observations, the projection directions are found by maximizing the regularized correlation between the latent scores of x and x
Cov( (X 22,, x²
(2)
mmm T 7) Var 1b 7 + (1 Var X²m
2

---

Under review as a conference paper at ICLR 2023
where v and 77; are the i-th directions of the projection matrices and 0 1 y S is regularization coefficient (Rosipal & Krämer, 2005; Hardoon et al., 2004). When 7 = 0. it is unregularized canonical correlation analysis (CCA) (Hotelling, 1936). When Y 1, it corresponds to partial least squares (PI.S) (Wold, 1975; Wold et al., 1984). Solving the optimization problem in Eq. non-linear iterative methods (NIPLAS algorithm) to make projection directions orthogonal to each other. The computation costs of both methods are prohibitively expensive when the data dimension is high or the number of samples is large. Therefore, it is more desirable to alternatively update a/5 and 17 in an iterative manner (Breiman & Friedman, 1985).
equation 2 requires singular value decomposition
ADVERSARIAL PERTURBATION BASED LATENT RECONSTRUCTION
3.1 LATENT RECONSTRUCTION
Let x' be a perturbed sample with a certain type of noise, which is adversarial noise in our case. Our pretext task in SSL reconstruct the latent representation of the clean sample x² from the perturbed sample x-. We use deep neural networks ()))) and 17(-) to project x² and x² into latent spaces, respectively. The reconstruction in the latent space can be achieved by maximizing the inner product between &b (x') and 17 (x"), when di (x') and (x²) have zero mean and unit variance. Based on discussions in Section 2, latent reconstruction must be done with orthogonality constraints to avoid the trivial solution, in which e²(-) and n7(-) projects all input data into constant vector. Latent reconstruction with orthogonality constraints is equivalent to finding the multi-dimensional Hirschfeld Hirschfeld-Gebeleinn-. Gebelein-Reer Rényi (HGR) maximal correlation (Renyi, 1959; Makur al., 2015) between two random views. It is defined as follows
p(x;;x² x2)
E & (x!) 1 (x²)]
sup
(oxxx))(>'''')) =B n )nxx)) ===()))==(((>*))
where zero mean constraints can be easily satisfied using batch normalization layer (loffe & Szegedy, 2015) and the constraints on identity covariance matrices can be achieved by forcing the off-diagonal elements to be zero. In practice, the constrained optimization problem in Eq. equation 3 is solved by minimizing the following loss
LLR
n (x2) + 2 (IV (x') db (x)) 11
II (x²) nx - -112:)
x² xIB
(4)
where y is a Lagrange multiplier, || || denotes the Frobenius norm, and is a mini-batch.
3.2 ADVERSARIAL PERTURBATION
Adversarial perturbation creates input samples that are almost indistinguishable from natural data but causes the deep learning models to make wrong predictions (Szegedy et al., 2014). We use generative model to generate adversarial perturbation because capable of creating diverse adversarial perturbations very quickly (Baluja & Fischer, 2018; Poursaeed et al., 2018; Naseer et al., 2019; Li al., 2020).
A generator G is trained to produce an unbounded adversarial ((x2) 5. The perturbation is then clipped to within an bound of x2 under the f norm. Let x² x² + the perturbed view of the clean sample x². The vast majority of adversarial perturbation methods rely on the classification boundary of the attacked neural network (1²()) and 17()) to train the generator via maximizing cross-entropy loss. However, it is not possible obtain the generative adversarial perturbation via maximizing a cross cross-entropy loss in our case because no label is available. In addition, existing generative adversarial perturbation methods explicitly relying on the classification boundary of the attacked model tend to over-fit to the training data (Nakkal & Salzmann, 2021). Instead of using cross-entropy loss, we train ((-) by maximizing the & distance between (i(x)) and r7x²))
Ladv
E. x1 x'EB
[|n(x2) (xx
(5)
where (-)) and n7()) are frozen.
3

---

Under review as a conference paper at ICLR 2023
3.3 ADVERSARIAL TRAINING
Our model is trained in an adversarial manner. Given a mini-batch of data, we train g(+) by maximizing Ldv while freezing v0()) and v7()) Then update the parameters in + and n7()) alternatively by minimizing LLR while freezing G(-). This process is illustrated in Fig. and Algorithm I.
Update g
Generator g
Encoder +p
|+|
8
x!
Ladv
Momentum Update "
Update ve
LLR
x²
Encoder
Figure I: Illustrative diagram for adversarial perturbation based latent reconstruction. The adversarial generator network takes a clean sample as input and outputs 6, an adversarial perturbation of the same shape. We constrain the perturbation by its ép norm and control its strength with the perturbation budget e. This constrained perturbation allows perturhed samples to appear unchanged to a human evaluator, making the perturbations sementic-preservingg
Algorithm 1 Adversarial perturbation based latent reconstruction
Natural sample x, encoders v6(+) and 77-), noise generator G(-), perturbation budget c, latent reconstruction loss LTRR, adversarial loss Ladv
for sampled minibatch
Train (()) (freeze v-)) and 17(-)
Generate an unbounded adversarial perturbation = ( ô has the same shape as x
Clip adversarial perturbation 6 = co/lô|p
Obtain the perturbed sample x' = x + and the clean sample x²
Obtain latent representations (xx1) and m)x")
Compute Ladv and update G(-) using SGD
Train (-)) and 17(-) (freeze G()))
Generate unbounded adversarial perturbation 8 = G(x)
8 has the same shape
Clip adversarial perturbation = c5/(5]p
Obtain the perturbed sample x3 = x and the clean sample x² x
Obtain latent representations (x')) and n7x²)
Compute LLR and update v-()) using SGD
Update 77+)) using the exponentially moving average of parameters in (-(-)
end for
THEORETICAL ANALYSIS
Let x be data sample without perturbation and y(x) be its downstream task label. The quality of the representation (x) is evaluated by the linear probe error, which is the linear classification
4

---

Under review as a conference paper at ICLR 2023
error of predicting (x) from using linear model parameterized by B R Let fB(x)
arg max4e(r) (1((x) B); be the prediction of the linear model. The linear probe error on (x) is
defined as
Err.
min Pr (x(x) + fB(x)]
(6)
x~P(x)
where P(x) is the data distribution.
We have to make two assumptions to bound the linear probe error on the learned representations. First, we assume that some universal minimizer of Eq. equation 3 can be realized by th(-) and n(-)- When the nonlinear mapping to find nulti-dimensional HGR maximal correlation is realizable by neural networks, we can analyze the quality of the learned representations using the properties in estimating the HGR maximal correlation.
Assumption 4.1 (Realizability). Let H. be hypothesis class containing functions vb X + R and ?) X T We assume that at least one the global minima of C(v,1) belongs to H.
In addition, it is also reasonable to assume that optimal classifier (*) can predict the label of x almost deterministically under semantic- preautiiii-rrrringgg perturbation. The assumption about the classification error of f-)+) provides baseline to quantify the linear probe error because part of the error from approximating f*(+) using a linear model.
Assumption 4.2 (a-bounded Error of the Optimal Classifier). Let x be a data sample without perturbation and y(x) be its downstream lask label. 8 is semantic-preserving perturbation. Then, assume that there a classifter I* such that Pr(S* (x) v(x)) S a and Pr(f*(x 5) xx)) S
Given assumptions 4.1 and 4.2, we provide the following main theorem on the generalization bound when learning a linear classifier with finite labeled samples on the representations learned by maximizing the HGR maximal correlation.
Theorem 4.3. Let ,
nf" E H be a minimizer of L(v, n)- The linear classification parameter B is estimated with nz i...d. random samples {(x;; y/) 1:)) With probability at least 1 Ç over the randomness of data, we have
k
[xx) + xx
Pr
f (x) S
(7)
Ak+1
122
where A4t1 is the k + 1-th Seapeeldd--eeeeii--eeeeee maximal correlation between X and X²2
We hide universal constant factors and logarithmic factors of k in Ō(-). The first term the right hand side of Theorem 4.3 guarantees the existence of linear classifier that achieves a small downstream classification error. It indicates whether the downstream label is linearly separable by the learned representation, thus measuring the expressivity of the learned representation. The second term on the right hand side reveals the sample complexity learning from finite labeled samples in the downstream task. It measures the data-efficiency of learning the downstream task using the learned representation. The proof is presented in the Appendix.
EXPERIMENTS
We demonstrate the effectiveness of the proposed method by application on three different data domains: tabular data, images, and audios. Additionally, we include ablation studies and sensitivity analysis in Appendix C. For all datasets, we follow the widely used linear evaluation protocol in self-supervised learning as a proxy to examine the quality of the learned representations (He et al., 2020; Chen & He, 2021). I). After the feature extractor pretrained with unlabeled training data, we discard the projection-head. and learn linear classifier on top of the frozen backbone network using the labeled training data.
For tabular and audio experiments, search the perturbation budget hyperparameter from the set (0.05, 0.1, 0.15). For image experiments, we fix e to 0.05 for a direct comparison with Viewmaker networks (Tamkin et al., 2021). We find that constraining the perturbations an & norm distance
5

---

Under review as a conference paper at ICLR 2023
achieves the best results. For all experiments, train the feature extractor and the adversarial generator in an alternating fashion. The feature extractor vh()) is trained with the SGD optimizer with momentum of 0.9 and weight decay of 5e-4. The learning rate is 0.03 without decay. The momentum coefficient in exponential moving average is set to 0.99 when updating 17()). The generator is trained with the Adam optimizer with initial learning rate of 1e-3 and its architecture is descrihed in the Appendix. Both the feature extractor and the generator are trained for 200 epochs with a batch size of 256. After self-supervised training on unlabeled data, linear classifier is trained using SGD with batch size of 256 and no weight decay for 200 epochs. The learning rate starts at 30.0 and is decayed after 200 epochs with cosine schedule.
5.1 TABULAR DATA
For tabular data, we follow existing works (Yoon et al., 2020; Verma et al., 2021) and use MNIST and MasSion-MNIST as proxy datasets by flattening the images into I-dimensional vectors. In addition, we use two real tabular datasets from the UCI repository to evaluate the proposed method (Dua & Graff, 2017).
MNIIST/Fashion-MNNN are two image datasets of handwritten digits and Zalando's article images, respectively (LeCun et al., 2010; Xiao et al., 2017). The images of size 28 x 28 are flattened into vectors with 784 features. Both datasets have 10 classes, and contain 60,000 training examples and 10,000 test examples.
Gas Concentrations is a dataset containing chemical sensor measurements of 128 features when exposed to 6 different gases (Vergara al., 2012: Rodriguez-Lajan al., 2014). The classification task is to identify the target gas. We perform a 80/20 train/test split to obtain 11,128 training examples and 2.782 test examples.
Gesture Phase is dataset containing 32 features extracted from videos of people in 5 different gestures (Wagner et al., 2014; Dua & Graff, 2017). We perform a 80/20 train/test split obtain 7,898 training examples and 1,975 test examples.
For all tabular data experiments, we adopt 10-layer MLP with residual connections Gorishniy et al. (2021) as the feature extractor. The generator is adapted from Nakkal & Salzmann (2021) by replacing convolutional layers with linear layers. The linear evaluation results on test datasets are reported in Table 1. APLR outperforms VIME-Self (Yoon et al., 2020), which corrupts tabular data and uses mask vector estimation and feature vector estimation as pretext tasks, on three out of four datasets. aligns with the empirical observations that reconstruction of high-dimensional data in the input space not necessary for learning high-quality representations. APLR outperforms the domain- aagnosic benchmark DACL (Verma et al., 2021), which uses mixup noise, on all four datasets. Mixup noise is less effective than adversarial noise because mixup noise perturbs informative and uninformative dimensions in the input space uniformly. Furthermore, convex combinations in the input space via mixup may result in augmented views off the data manifold. Interestingly, our proposed APLR also outperforms training the full architecture in supervised manner the two real tabular datasets, Gas and Gesture.
Table 1: Linear evaluation accuracy on tabular data
MNIST
Fashion-MNIST
Gas
Gesture
Tabular-specific
VIME-Self' (Yoon et al., 2020)
96.62
87.26
93.17
38.99
Domain-agnostic
DACL (Verma el al., 2021)
94.70
79.78
95.39
38.56
Ours
97.11
87.12
97.98
42.97
Fully supervised training
98.67
90.00
94.91
42.32
'oor experiments with VIMF-Self, we align the experimental setup with APLR use the same backbone architecture and perform pretraining the entire training set.

---

Under review as a conference paper at ICLR 2023
5.2 IMAGE DATA
We use four benchmark image datasets to evaluate the effectiveness of the proposed method, including LEFAR-10/100 (Krizhevsky. 2009). STL-10 (Coates et al., 2011) and Tiny-ImageNet (Le & Yang. 2015). The details of the datasets are described in the Appendix.
ResNet18 (He al., 2016) is adopted as the backbone network in the feature extractor. We adopt the generator in Nakkal & Salzmann (2021) for image data. We present the results for self-supervised representation learning on image data Table 2. It observed that APLR outperforms DACL (Verma et al., 2021) by large margin, indicating that adversarial noise is a more effective semantic-preserv perturbation than mix-up noise in DACL. Interpolation of input samples via mix-up could lead to out-of-distribution training samples because the input data space may not be convex. Our method also achieves better performance than Viewmaker (Tamkin et al., 2021), which is a Nomainn-gnossie sulf-supervieed learning method by discriminating adversarially perturbed data. The adversarial noise in APIR is more robust because the training process of adversarial noise APLR does not rely on the classification boundary between augmented samples (Nakkal & Salzmann, 2021). Furthermore, we also compare APLR against methods that use image augmentations (e.g. cropping, rotation, horizontal flipping), such as SimCLR (Chen et al., 2020). It is found in previous studies (Grill et al., 2020; Chen et al., 2020) that random crop is a crucial data augmentation towards learning high-quality representations for image data. However, it is impossible to create cropped views of images using adversarial perturbation because the adversarial noise is additive to the natural sample. Given the importance of random crop and the inability to create cropped views with adversarial perturbations, achieving comparable accuracies between APLR and SimCLR indicates that adversarial noise is highly effective data augmentation method.
Table 2: Linear evaluation accuracy on image data
CIFAR-10 CIFAR- 100
STL-10
Tiny-ImageNet
Image-specific
SimCLR (Chen al., 2020)
86.47
54.86
85,49
43.27
Domain-agnostic
DACL (Verma et al., 2021)²
60.49
35.28
57.34
22.69
Viewmaker (Tamkin et al., 2021)
84.51
52.28
82.73
40.51
Ours
85,92
55,83
86.21
42,93
5.3 AUDIO DATA
We use three benchmark audio datasets to evaluate APLR and describe the dataset details below.
ESC-10/50 are two environmental sound classification datasets containing 5-seconds of environmental recordings (Piczak, 2015). ESC-10 10 and ESC-50 have 10 and 50 classes, and contain 400 and 2000 examples, respectively. We use the original fold settings from the authors (Piczak, 2015), and follow the experimental setup in Al-Tahan & Mohsenzadeh (2021) to use the first fold for testing and the rest for training.
LibriSpeech-100 100 is corpus of read English speech (Panayotov al., 2015). We use speaker identification as the downstream classification task. We follow the experimental setup from Tamkin al. (2021) to pretrain with the LibriSpeech- hour corpus which contains 28,539 examples, and perform linear evaluation on the LibriSpeech development set which contains 2,703 examples.
For audio experiments, generator in Nakkal & Salzmann (2021) with one input channel. The time-frequency representation a 2D log mel spectrogram, normalized to zero mean and variance. We report the results on audio data benchmarks in Table 3, and visualize examples from LibriSpeech-100 100 in the Appendix. APLR performs significantly better than CLAR (Al-Tahan & Mohsenzadeh, 2021). which experimented extensively with combinations of audio-specific augmentations and uses fade in/out and time masking as their fest-performing audio augmentations. Compared to image augmentations, the data augmen-
1-D ResNet18 (He al., 2016) the feature extractor and adopt the
PACL reports better results using larger hackbones and more training epochs. We just report the reproduced results under our experiment settings.
7

---

Under review as a conference paper at ICLR 2023
tations for audio data are relatively underexplored. Our results demonstrate the advantage of learning audio augmentations over manually designed augmentations. Our proposed method also outperforms methods, DACL and Viewmaker. DACL performs close to APLR on the simple yet small ESC-10 and ESC-50 datasets. However, it is unable to learn effective representations complex LibriSpeech- 100 dataset. Even though both APLR
both domain-agnone
the larger and significantly and Viewmaker use adversarial noise, APLR outperforms Viewmaker by a large margin across all benchmark datasets. This indicates the effectiveness of learning augmentations by maximizing the discrepancy between latent representations.
In Table 3, we also report results on training the full architecture supervised manner. We find that linear classifiers trained on top of the representations learned by APLR outperforms the supervised model on ESC-10, and closes the gap to ESC-50 compared other benchmarks, demonstrating the ability for APLR to learn useful latent representations. Current ofate-o--hheane supervised approaches report high accuracies (over 94%) on the ESC-50 dataset (Gong et al., 2021; Kumar & Ithapu, 2020). However, these methods perform pretraining using large datasets such as AudioSet and ImageNet, and use multiple audio-specific data augmentations. With the supervised training experiments, we do not perform pretraining with large datasets, and as augmentations. Our goal is to simply compare APLR against training the same architecture in a supervised manner.
time masking and frequency masking
Table 3: Linear evaluation accuracy on audio data
ESC-10
ESC-50
LibriSpeech-100
Audio-specific
CLAR (Al-Tahan & Mohsenzadeh, 2021)
68.70
40.40
62.14
Domain-agnoste
DACL (Verma et al., 2021)
77.75
48.50
37.30
Viewmaker (Tamkin et al., 2021)
70.00
35.75
88.30
Ours
81.25
57.75
96.29
Fully supervised training
76.25
59.14
N/A²
RELATED METHODS
Unsupervised representation learning can be categorized into two classes based on the type of the pretext task: generative or discriminative. Generative approaches learn to generate reconstruct unla- beled data in the input space (Higgins et al., 2017; Donahue et al., 2017: Donahue & Simonyan, 2019). Reconstructing the masked portion of data is highly successful in discovering useful representations in natural language processing (Mikolov et al., 2013; Brown et al., 2020; Devlin et al.. 2019). Before the success of masked language models, variants of denoising or masked autoencoders are developed for computer vision tasks (Vincent et al., 2008; 2010; Pathak et al., 2016) but the performance is worse than discriminative SSL methods. was not until recently that masked image models are revived in unsupervised visual representation learning by discretizing image patches via tokenizers (Bao et al., 2022; Zhou et al., 2022). MAE (He al., 2022) further simplifies masked image models by directly inpainting masked images without tokenizers or image-specific augmentations. Although masking is a simple data augmentation that can be flexibly applied to different domains of data, computationally expensive generation or reconstruction in the input space may not be necessary for representation learning. Our method is derived from a generative model and shares the idea of reconstructing the corrupted samples in masked autoencoding. Instead of reconstructing discrete tokens or raw inputs, our method reconstructs the continuous latent representation, which is related discriminative SSL methods using data augmentations.
Augmentation-basee based discriminative SSL methods not limited to contrastive learning) augmented views of unlabeled data in the latent space. This line of work involves a contrastive framework with variants of InfoNCE loss (Gutmann & Hyvärinen, 2010;
learn representation by comparing (including but
Following the experimental setup in Tamkin et al. (2021), the training set in LibriSpeech has 251 classes and the testing set has 40 classes. Since we train the fully supervised model in an end-to-end manner without linear evaluation phase, we cannot report result for LihriSpeech-104
8

---

Under review as a conference paper at ICLR 2023
Oord et al., 2018) pull together representations of augmented views of the same training sample (positive sample) and disperse representations of augmented views from different training samples (negative samples) (Wu et al., 2018; Hénaff et al., 2020; Wang & Isola, 2020). Typically, contrastive learning methods require a large size of negative samples to learn high-quality representations from unlabeled data (Chen al., 2020; He et al., 2020). Meanwhile, non-contrastive methods train neural networks to match the representations of augmented positive pairs without comparison to negative samples or cluster centers. However, non-contrastive methods suffer from trivial solutions where the model maps all inputs to the same constant vector, known as a collapsed representation. Various methods have been proposed to avoid a collapsed representation on an ad hoc basis, such as asymmetric network architecture (Grill et al., 2020), stop gradient (Chen & He, 2021), and feature decorrelation (Ermolov et al., 2021; Zbontar et al., 2021; Hua et al., 2021). Interestingly, our method also includes feature decorrelation constraint, which is adapted from generative model. Recently, adversarial perturbations are combined with image augmentations to create more challenging positive and negative samples in self-supervised learning (Ho & Nvasconcelos, 2020; Yang et al., 2022). APLR does not require domain-speciff augmentations and can be applied to different domains of data.
Learning augmentations has been investigated in supervised learning to obtain data-dependent augmentation policies for better generalization (Cubuk et al., 2019; Hataya et al., 2020). In parallel, adversarial perturbation can be treated as a special form of learnable augmentations to enhance the robustness of models with adversarial training (Goodfellow et al., 2015; Madry et al., 2018), The domain-agnostic augmentations in our method are closely related to generative adversarial perturbation, where data augmentations are obtained through a forward pass of learnable generative models (Li et al., 2020; Baluja & Fischer, 2018; Poursaeed et al., 2018; Naseer et al., 2019). The vast majority of adversarial perturbation methods rely on the classification boundary of the attacked neural network to train the generator via maximizing cross-entropy loss. Those ideas have been extended to SSL to get adversarial perturbation by maximizing the InfoNCE loss in SimCLR (Kim et al., 2020; Tamkin et al., 2021). However, existing generative adversarial perturbation methods rely explicitly on the classification boundary or the instance discrimination boundary of the attacked model and tend to make them over-fit the source data (Nakkal & Salzmann, 2021). Instead of maximizing a cross-entropy loss, we maximize the Eg distance between mid-level feature maps to obtain generative adversarial perturbations.
Theoretical understanding of SSL has been studied under the assumption that augmented views of the same raw sample are somewhat conditionally independent given the label or hidden variable (Arora et al., 2019; Tosh et al., 2021 b;;; Lee et al., 2021). However, those assumptions do not hold in practice because augmented views of natural sample are usually highly correlated. Augmented views are unlikely to be independent given the hidden label. Recent studies in contrastive learning provide theoretical guarantees of the learned representation without the assumption of conditional independence (HaoChen et al., 2021; Wang et al., 2022). In parallel, Tian et al. (2021) investigates the training dynamics of non-contrastive SSL methods to show how feature collapse is avoided but lacks guarantees for solving downstream tasks. Note that our proposed method does not involve an explicit comparison between positive and negative samples. Our theoretical analysis relies on the divergence transition matrix without the assumption of conditional independence.
CONCLUSIONS
In this article, we introduce APLR, a domain-agnostic SSL method by reconstruction of adversarial perturbed samples in the latent space. The adversarial perturbation is created by generative network, which is trained concurrently with the feature encoder an adversarial manner. Our empirical results show that the proposed method is better than the existing domain-agnoste SSL methods and achieves comparable performance with SOTA domain-specifi SSL methods. In many cases, APLR also outperforms training the same architecture in fully supervised manner, demonstrating its strong ability to learn useful latent representations. In addition, the proposed latent reconstruction linked to Hirschfeld Geeidd-ebeeeiineee classification tasks. We believe that the proposed method can be applied to applications beyond classification, such as reinforcement learning.
maximal correlation and thus has theoretical guarantees of downstream
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Haider Al-Tahan and Yalda Mohsenzadeh. Clar: contrastive learning of auditory representations. In International Conference on Artificial Intelligence and Statistics, pp. 2530- 2538. PMLR, 2021.
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning. In Proceedings of the 36th International Conference on Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 5628-5631 PMLR, 2019.
Francis R Bach and Michael I Jordan. A probabilistic interpretation of canonical correlation analysis. Technical Report, Department of Statistics, University of Califormia, 2005.
Shumeet Baluja and Ian Fischer. Learning to attack: Adversarial transformation networks. In Thirty-second aaai conference on artificial intelligence, 2018.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers. In International Conference on Learning Representations, 2022.
Leo Breiman and Jerome H Friedman. Estimating optimal transformations for multiple regression and correlation. Journal of the American statistical Association, 80(391):580-595 1985.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33: 13:177711 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2021.
Adam Coates, Andrew Ng, and Honglak Lee. An Analysis of Single Layer Networks in Unsupervised Feature Learning. In AISTATS, 2011.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 113-123, 2019.
Jacob Devlin, Ming- Ming-Weii Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HILT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pp. 4171-4186. Association for Computational Linguistics, 2019.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. Advances in Neural Information Processing Systems, 32, 2019.
Jeff Donahue, Philipp Krähenbühh, and Trevor Darrell. Adversarial feature learning. In 5th Interna- tional Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
Dheeru Dua and Casey Graff. UCI machine learning repository. 2017. URL AtLLttp///acciiee. ics. uci medi.eedumm
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nieu Sebe. Whitening for self- supervised representation learning. In International Conference on Machine Learning, pp. 3015- 3024. PMLR. 2021.
Yuan Gong, Yu-An Chung, and James Glass. AST: Audio Spectrogram Transformer. Proc. Interspeech 2021, pp. 571-575, 2021. doi: (0.21437/nterrspecch..00 2021-698.
10

---

Under review as a conference paper at ICLR 2023
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In 3rd International Conference on Learning Representations. ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learn- ing models for tabular data. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 18932-18943. Curran Associates, Inc., 2021. URL. https::tt//.tc pwtp//ooooeeeeeeeoooooooo iggs. . cc/paper wttpppppe//2211///////// 2ttp://2212/fiee 9d8 6d8 3008839955222 /ttphhtts://proeeeee
papp:ea92222p-pppeee.e. .df.
William H Greene. Econometric analysis. Prentice Hall, 2003.
Jean-Bastir Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to sulf--suee learning. In Proceedings of the 33nd International Conference on Neural Information Processing Systems, 2020.
Michael Gutmann and Aapo Hyvärinen. contraccnt estimation: A new estimation principle for unnormalized statistical models. Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 297-304, 2010.
Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems, 34, 2021.
David Hardoon, Sandor Szedmak, and John Shawe-Taylor. Canonical correlation analysis: An overview with application to learning methods. Neural computation, 16(12):2639-288 2664, 2004.
Ryuichiro Hataya, Jan Zdenek, Kazuki Yoshizoe, and Hideki Nakayama. Faster autoaugment: Learning augmentation strategies using backpropagation. In European Conference on Computer Vision, pp. 1-16. Springer, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Olivier Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. In International Conference on Machine Learning, pp. 4182-4192, PMLR, 2020.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick,
Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. ICLR, 2(5) 2017.
Chih-Hui Ho and Nuno Nvasconcelos. Contrastive learning with adversarial examples. Advances in Neural Information Processing Systems, 23:17081-17095 2020.
H Hotelling, Relations between two sets of variates. Biometrika, 1936.
Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao. On feature decor- relation in self-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9598 000889908,-881 2021.
Shao-Lun Huang and Xiangxiang Xu. On the sample complexity of hgr maximal correlation functions for large datasets. IEFE Transactions Information Theory, 67(3):1951-1999 2020.
11

---

Under review as a conference paper at ICLR 2023
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning. pp. 448-444455 456.. PMLR. 2015.
Minseon Kim, Jihoon Tack, and Sung Ju Hwang. Adversarial superviee contrastive learning. Advances in Neural Information Processing Systems, 33:2983- 2994, 2020.
Krizhevsky. Learning Multiple Layers of Features from Tiny Images. PhD thesis, University of Toronto, 2009.
Anurag Kumar and Vamsi Krishna Ithapu. sequential self teaching approach for improving generalization in sound event recognition. In Proceedings of the 37th International Conference on Machine Learning, ICML'200. JorR.org, 2020.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. 2015.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecmn www.n. cww.ex/b/////iss 2, 2010.
Jason D Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo Predicting what you already know helps:
Provable self-supervised learning. Advances Neural Information Processing Systems, 34, 2021.
Yingwei Li, Song Bai, Cihang Xie, Zhenyu Liao, Xiaohui Shen, and Alan L. Yuille. Regional homogeneity: Towards learning transferable universal adversarial perturbations against defenses. In Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI, volume 12356. pp. 795- 813. Springer, 2020.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net 2018
Anuran Makur, Fabián Kozynski, Shao-Lun Huang, and Lizhong Zheng. An efficient algorithm for information decomposition and extraction. In 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp. 972-979. IEEE, 2015.
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representa- tions in vector space. In Yoshua Bengio and Yann LeCun (eds.), 1st International Conference on Learning Representations, ICLR 2013, 2013.
Krishna Kanth Nakkal and Mathieu Salzmann. Learning transferable adversarial perturbations.
Advances in Neural Information Processing Systems, 34, 2021.
Muhammad Muzammal Naseer, Salman H Khan, Muhammad Haris Khan, Fahad Shahbaz Khan, and Fatih Porikli. Cross-domain transferability of adversarial perturbations. Advances in Neural Information Processing Systems, 32, 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1rXiv:1807..33:
2018.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on, pp. 5206-5210. IEEE, 2015.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2016.
Karol J. Piczak. ESC: Dataset for Environmental Sound Classification. In Proceedings of the 23rd Annual ACM Conference Multimedia, pp. 1015-1018. ACM Press, 2015. ISBN 9-5-3- 3459-4. doi: 10.11455773333332206663
URL http: /dl. acm. owggciiaaiinnoooooooooooooo cfm?
doid=2733373 (021773333332288833 2806390.
12

---

Under review as a conference paper at ICLR 2023
Omid Poursaeed, Isay Katsman, Bicheng Gao, and Serge Belongie. Generative adversarial perturba- tions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4422 1422-4431, 2018.
A. Renyi. On measures of dependence. Acta Mathematica Academiae Scientiarum Hungarica, 10: 441-451, 1959.
Irene Rodriguez-LLujan Jordi Fonollosa, Alexander Vergara, Margie Homer, and Ramon Huerta. On the calibration of sensor arrays for pattern recognition using the minimal number of experi- ments. Chemometrics and Intelligent Laboratory Systems, 130:123-1344 2014. ISSN 0169-7439. doi: http:///rrr......0ooooooeemmoabb com/ cotp///ccennneerrrrrreee//////1/
URL https: wwww.wwwwom...oom...........wwo/ www.::ww.w.....................w sciencecirect..
Roman Rosipal and Nicole Krämer. Overview and recent advances in partial least squares. In International Statistical and Optimization Perspectives Workshop" Subspace, Latent Structure and Feature Selection", , pp. 34-51. Springer, 2005.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In 2nd International Conference on Learning Representations, ICLR 2014, Bansf, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Alex Tamkin, Mike Wu, and Noah Goodman. Viewmaker networks: Learning views for unsupervised representation learning. In International Conference Learning Representations, 2021.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learming Research, pp. 10268-10278. PMLR, 2021.
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redun- dancy, and linear models. In Algorithmic Learning Theory, pp. 1179-1206. PMLR, 2021a.
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive estimation reveals topic posterior information to linear models. Journal of Machine Learning Research, 22(281):1-2 2021b.
Talip Ucar, Ehsan Hajiramezanali, and Lindsay Edwards. Subtab: Subsetting features of tabular data for self-supervised representation learning. Advances in Neural Information Processing Systems, 34, 2021.
Alexander Vergara, Shankar Vembu, Tuba Ayhan, Margaret A. Ryan, Margie L. Homer, and Ramón Huerta. Chemical gas sensor drift compensation using classifier ensembles. Sensors and Actuators B: Chemical, 166-167:320-32,, 2012. ISSN 0925-4005. doi: https///doi.orr/10..100 2012.01.074. URL https:/ www. sciencedirect. com/ wwwence/article///eee s0 50925400512002013
Vikas Verma, Thang Luong, Kenji Kawaguchi, Hieu Pham, and Quoe Le. Towards domain-agnostic contrastive learning. In International Conference Machine Learning, pp. 10530-10541. PMLR, 2021.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103. 2008
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of machine leurning research, 11(2)), 2010.
Priscilla Wagner, Sarajane Peres, Renata Madeo, Clodoaldo Lima, and Fernando Freitas. Gesture unit segmentation using spatial-temporal information and machine learning. 01 2014.
13

---

Under review as a conference paper at ICLR 2023
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. 9929-9939, PMLR. 2020.
Yifei Wang. Qi Zhang, Yisen Wang. Jiansheng Yang, and Zhouchen Lin. Chaos is ladder: A new theoretical understanding of contrastive learning via augmentation overlap. In International Conference on Learning Representations, 2022.
Hans S Witsenhausen. On sequences of pairs of dependent random variables. SIAM Journal on Applied Mathematics, 28(1):100-11 1975.
Herman Wold. Soft modelling by latent variables: the non non-linear iterative partial least squares (nipals) approach. Journal of Applied Probability, 12($1):1 1975.
Wold, A. Ruhe, Wold, and W.J. Dunn, III. The collinearity problem in linear regression. The partial least squares (PI.S) approach to generalized inverses. SIAM J. Sci. Comput., 5:735-743, 1984.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametrie instance discrimination. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3733-3742, 2018.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Rashion-mnise a novel image dataset for benchmarking machine learning algorithms, 2017.
Kaiwen Yang, Tianyi Zhou, Xinmei Tian, and Dacheng Tao. augmentation for self-supervised learning. In International Conference on Machine Learning, pp. 25364 25381. PMLR, 2022.
adversarial
Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar. Vime: Extending the success of self-and semi supervised learning to tabular domain. Advances in Neural Information Processing Systems, 33:1 11111003-11000 11043, 2020.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6023-6032, 2019.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self Self-supervised learning via redundancy reduction. In Proceedings of the 38th International Conference Machine Learning, ICMI. 2021, 18-24 July 2021, I, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 12310-12320 PMLR, 2021.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmenta- tion. CoRR, abs/ 71111708.0888888 04896, 2017. URL /tttt:://aaatte org/ aaaaabss1// 04896.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image BERT pre-training with online tokenizer. In International Conference on Learning Representations, 2022.
A PROOF OF THE MAIN THEOREM
The HGR maximal correlation can be estimated from divergence transition matrix A E R [x whose entries are defined by the joint and marginal distributions x and x² (Witsenhausen, 1975). Let P.. and Pa be the marginal distribution and Patz= be the joint distribution. Pe (x)) can be viewed as the probability mass of x being randomly sampled from x Then, each entry A is given by
Pat Ext, x3)
Aij ij
(8)
Par ) (x²)
14

---

Under review as a conference paper at ICLR 2023
The solution Eq. equation is the of the top k singular values of A with left singular vectors Z E IRNx and right singular vectors Z IR Nx defined as
z V (x(x)))(x),
i= I,.,., N
(9)
z (=((x))((()) i= 1=.,..,
where z! and z are the i-th row of embedding matrices Z and Z respectively (Huang & Xu, 2020).
This is essentially a rank-k approximation of A via minimizing ||A Z- || clean sample and z² is the representation of clean sample. We use clean samples in downstream tasks. We drop the superscription to avoid cluttered notation.
Note that x² is
The first term the right hand side of the main theorem (theorem 4.3) measures the approximation error of the optimal classifier f by a linear classifier parameterized by B. It amounts to residual of the least squares problem |[f* ZB| 1 in Fig. 2, where the representation matrix Z E IR N contains the top-k left singular vectors of A and f* E {0, 1} the vector that contains the predicted labels of all the data by the optimal classifier f*. The approximation error is bounded if f has limited projection into the residual subspace that is perpendicular to the column space of the representation.
f"
2 B
Figure 2: Geometric interpretation of least squares. fx) X + 10,1} is the Bayes optimal
classifier for predicting the label given x with error most a according to Assumption 4.2. The green panel is the subspace spanned by the columns of the representation z. B is the parameters of the linear classification model.
the first step, we construct a quadratic form f' to quantify its projection into residual space based on singular value decomposition of A. The largest singular value of A is 1, with constant left and right singular vectors being and 1 (Huang & Xu, 2020). Therefore, is more convenient to subtract the top singular mode and introduce À = I- A. À can be factorized as = EMr ? via singular value decomposition, where % the i-th singular value of À with the left singular vector
15

---

Under review as a conference paper at ICLR 2023
u, and the right singular vector v,. The quadratic form is given as follows
f" Af
P
yuvy
yu.v
r
(10)
k
T
muu
+
Σ
f
(11)
1=
2 T
Nuu
fr
(12)
2 T
*k+1
Σ i=++1
iiu
f
(13)
-k+1 f"
Pf
7+1ff" " PP = =1+11PPf
(14)
where Eq. equation 11 is due to the fact that the left and right singular vectors are the same in the symmetric matrix A, the inequality in Eq. equation 12 is because of dropping a quadratic term, and Eq. equation 13 is due to 7 I 7x+2 S 7N- P & Ei i defines a projection matrix that projects f into a residual subspace spanned by singular vectors Ug+1,, .... u,y Eq. equation 14 is obtained because P is an idempotent matrix (P² = P) (Greene, 2003). In addition. (I - f) is in the subspace spanned by singular vectors u1..... ug. which the column space of Z. Based on the geometric interpretation of the least squares problem || ZBI|² there exists B that such that (I P)) ZB the projection of f* onto the column space of Z.
In the second step, we upper bound 11+1 IP Based on Eq. equation 14, we have 74+1 | <
p* T Af.. is more convenient upper bound f* Af*
Af Af
f" If ft
Af
N
N
ft (x)
px my
Σ
Paza
iJ=1
1
N
F
f
ft -2
Σ
> f f))
2
VP v (X))(X))
1
N
f
2
N
ft
Pz
(15)
Pai (xi)
2
P Pataa 2x)==(X;; x3,;
2
VPal (xx)
/ P2.(x,)) v VP
1,=1
N
2
+
(x)(
(Pae(Xg)
2 ΣΣ
1
N N
pa (x;,
Pz
f:
Paa (x;)
where f = fx(x), (x,, = Σ₁ Pataa "x;=*(;;,;, (x,,,)) I/N and P,= (xx) = EN Pa (x,,x,)
1/N. Note that we only sample a pair of samples (x,, x,) where x; is created from semantic- preserving perturbation of x, to train the model. The probability mass Paiza (x,, x,) > 0 only if (x,, are generated from a shared latent variable. Let (x, x+) be a positive pair to denote pair of samples created from semantic- preeerviip perturbation. We rewrite equation equation 15 as
N Af = 2 Ez (( - (1
(16)
where Eg, [(C: - P: } quantifies the probability that the optimal classifier f*(-) predicts different
labels for (x, xt) When f * f. there must be fx() + y(x) or fx()) * y(x). With
Assumption 4.2, we have Pr(f* (x) # fx+))) = 2a. As such, the quadratic form in Eq. equation 16 can be upper bounded:
Pr Tp < No.
(17)
16

---

Under review as a conference paper at ICLR 2023
With Eq. equation 14 and equation 17, we have
Na
| 2B| |||Pf|
(18)
76+1
N f ZBI S a/7k+1
(19)
The k-th singular value of A is A 1 maximal correlation by definition. Therefore, we have Prx=P(xx)x(x) + [n(x]] S Ō T-
7k. which is also the k-th Hirschfeld-Gehelein-Rkii
The second term on right hand side the main theorem the estimation error that measures sample complexity of learning B with access to n2 i.i.d. training samples in the downstream task. It can be upper bounded using the Rademacher complexity of linear models. Let 74 = {z - | C.J. We have the Rademacher complexity of the linear model
COVC.
Bag (H1)
(20)
Vn2
where [ I Cz. By definition of Eq. equation 3, B||l[|*] captures the summation of first k HGR maximal correlation. |[[|2|]] < k because the HGR maximal correlation less equal than 1. Therefore, have
Pr [e(()) + [B(()] Sō x P(x)
1-1x+1
Ak+1
V
112
B IMAGE DATASETS
CIFAR-10/100 are two datasets of tiny natural images with a size of 32 x 32 (Krizhevsky, 2009). CIFAR-I0 and CIFAR-100 have 10 and 100 classes, respectively. Both datasets contain 50,000 training images and 10,000 test images.
STL-10 10-class image recognition dataset for unsupervised learning (Coates al., 2011). Each class contains 500 labeled training images and 800 test images. In addition, it also contains 100,000 unlabeled training images. Both labeled and unlabeled training images are used for feature extractor pretraining without using labels. The linear classifier is learned using the labeled training images.
Tmay-ImageNet subset of the ILSVRC-2012 classification dataset (Le Yang, 2015). consists of 200 classes, with 500 training images, 50 validation images, and 50 test images in each class. The size of each image is 64 64.
C ADDITIONAL RESULTS
To understand the effectiveness of adversarial perturbations within APLR, perform several additional experiments. First, we compare perturbation by adversarial noise against perturbation by Gaussian noise and random masking. For image datasets, we additionally compare the proposed adversarial perturbations against common image augmentations used in supervised learning, including CutMix (Yun al., 2019), GandAutment (Cubuk et al., 2019), and Random Erasing (Zhong et al., 2017). Next, we explore the sensitivity of APLR to different perturbation strengths and Lagrange multipliers. Lastly, we compare our framework against SOTA SSL methods on image data.
C.I ABLATION STUDY
First, for all datasets, perform ablations compare perturbations with adversarial noise against Gaussian noise and masking. To obtain a sample augmented with Gaussian noise, we use x = N (0, For each dataset, we search the standard deviation from the set (1, 5, 10) and report the best linear evaluation accuracy. For experiments with masking, randomly mask a proportion of the clean sample x². We search the proportion masking from the set (20%, 40%, 50%, 60%. 70%} and report the best linear evaluation accuracy.
x" + 6, where
17

---

Under review as a conference paper at ICLR 2023
Tables 4 - 6 summarize the results. The adversarial noise outperforms the Gaussian noise and random masking on all datasets, except MNIST. Random noises may not be as effective since uniformly perturbing uninformative features may not lead to the intended goal of augmentations. That is why APLR leads to significant improvement over random perturbations on complex data, such as images and audios.
Table 4: Ablation study on tabular data.
MNIST
Gas
Gesture
Gaussian noise
97.43
85.77
96.25
41.46
Masking
97.58
86.95
95.70
42.30
APLR
97.11
87.12
97.98
42.97
Table Ablation study on image data.
CIFAR-10
CIFAR-100
STL-10
Tiny-ImageNel
Gaussian Noise
53.58
28.43
52.76
12.07
Masking 48.79
27.43
50.39
11.29
APLR
85.92
55.83
86.21
42.93
Table 6: Ablation study audio data.
ESC-10
ESC-50
LibriSpeech-100
Gaussian noise
75.00
41.75
78.52
Masking
77.50
45.00
76.76
APLR
81.25
57.75
96.29
Additionally, we compare adversarial noise against image augmentations in supervised learning, namely CutMix (Yun al., 2019), RandAugment (Cubuk et al., 2019), and Random Erasing (Zhong et al., 2017). The results are summarized in Table 7. Random Erasing results in the worst performance among all methods, while CutMix is on par with Mixup SSL. This is expected because CutMix performs slightly better similar to MixUp in supervised learning. Rand/ugment leads to better performance than CutMix and Mi because RandAAgment contains a wide range of image augmentations. However, RandAugment does not outperform SimCLR. The studies in SimCLR show that careful selection of image augmentations necessary for good performance in SSL. Some effective image augmentations in supervised learning do not lead to good performance in SSL..
Table 7: Additional ablation study on image data,
CIFAR-10
CIFAR-100
STL-10
Tiay-em
SimCLR
86.47
54.86
85,49
43.27
CutMix
61.27
35.14
58.33
22.83
RandAugment
84.34
51.92
84.37
40.68
Random Erasing
56.71
28.89
55.91
18.45
Ours
85.92
55.83
86.21
42,93
C.2 SENSITIVITY ANALYSIS
We perform experiments understand how sensitive APLR is to different strengths of the adversarial perturbation and Lagrange multiplier.
We experiment with perturbation strengths of c (0.05, 0.1, 0.15). and report the results in Table 8. The sensitivity analysis indicates that our method is robust to the adversarial perturbation strengths.
For the Lagrange multiplier, we experiment with y E {0.1, 0.5, 0.1} and report the results in Table 9. We find that our method is robust to y and achieves strong performance. For APLR, we selected y 1 as the default value since it performed well consistently.
18

---

Under review as a conference paper at ICLR 2023
Table 8: Sensitivity to adversarial perturbation strengths.
0.05
0.1
0.15
Tabular Data
MNIST
97.11
96.15
93.73
Fashion- -NNIST
87.04
86.40
84,14
Gas
97.50
Gesture
42.97
40.90
41,46
Image Data
CIFAR-10
85,92
84.66
85,26
CIFAR-100
55.83
54.37
54.77
STL-10
86.21
85.04
85.64
Tiny-ImageNet
42.93
42.42
41,47
Audio Data
ESC-10
78.75
81.25
77.50
ESC-50
54.25
54.50
57.75
LibriSpeech-100
93.55
96.29
96.29
Table 9: Sensitivity to Lagrange multiplier.
== 0.1
7y
0.5
7y
Tabular
MNIST
96.54
96.93
97.11
"astion-MNIST
86.83
86.69
87.12
Gas
84.61
97.97
97.98
Gesture
40.35
40.35
42,97
Audio Data
ESC-10
80.00
75.00
81.25
ESC-50
47.75
44,50
57.75
LibriSpeech- 100
89.45
87.30
96.29
Our sensitivity analyses indicate that our method is robust hyperparameters such as and y- The proposed APLR achieves strong performance as long as the hyperparameter values are within reasonable ranges.
C.3 APLR AGAINST SOTA IMAGE-SPECIFIC SSL METHODS
We perform an analysis compare the proposed framework against SOTA SSL methods on images, namely SimCLR (Chen et al., 2020), Barlow Twins (Zbontar et al., 2021), and BYOL (Grill et al., 2020). For this experiment, for a fair comparison against image-specific SSL methods. We train each model for 200 epochs and summarize the results in Table 10. Our method achieves comparable performance BYOL and Barlow Twins.
use the image augmentations described in SimCLR (Chen et al., 2020)
Table 10: APLR SOTA SSL methods image data
CIFAR-10
CIFAR-100
STL-10
Tiny-ImageNet
SimCLR
86.47
54.86
85.49
43.27
Barlow Twins
89.02
62.84
85.43
45.33
BYOL
88.54
61.76
85.59
42.75
Ours
89.63
62.55
86.41
44.76
19

---

Under review as a conference paper at ICLR 2023
D VISUALIZATIONS OF ORIGINAL AND PERTURBED SPECTROGRAMS
In Figure 3, we visualize random spectrograms from LibriSpech- and the deltas between the original and perturbed spectrograms. The perturbations indistngguishable and thus semantic- preserving.
Figure 3: Examples triplets of original spectrograms (left), perturbed spectrograms (middle) and their differences (right) from LibriSpeech- 100. The color scales for original and perturbed spectrograms are set to the scale of the original spectrogram. The color scale for the differences is set to -2.5 (red) to 2.5 (blue), though some values exceed this range. Best viewed when zoomed.
E ADVERSARIAL GENERATOR ARCHITECTURE
The architecture of the generator is described in Table 11. For experiments tabular data, replace the convolution layers with fully connected layers.
Table 11: Architecture of the adversarial generator.
Layer
Number of Filters
Kernel Size
Convolution Layer
9
Convolution Layer
64
3
Convolution Layer
128
3
Residual Block
128
3
Residual Block
128
3
Residual Block
128
3
Upsampling Convolution I Layer
64
3
(Upsample 2)
Upsampling Convolution Layer
3
Upsample==2 2)
Convolution Layer
F VISUALIZATIONS OF TRAINING OBJECTIVES DURING TRAINING
To understand how well the final objective Equation approximates the HGR correlation, we plot the two loss terms over training in Figure 4. We find that the orthogonal loss approaches zero during training.
20

---

Under review as a conference paper at ICLR 2023
Losses During Training
0.075
Orthogonal Loss
0.050
0.025
LOSS
0.025
0.050
-0.075
-0.100
200
400
800
800 Iteration
1000
1200
1400
1600
Figure Losses during training.
21


Reviews:
[{"review_id": "_UbfDOFFev", "reviewer": "Reviewer_JiWn", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a domain-agnostic method to learn augmentations in self-supervised learning.", "summary_of_the_review": "The paper proposed a simple and novel paradigm for self-supervised learning. Unlike contrastive learning, the proposed method can learn augmentation without domain knowledge. However, the paradigm still suffers from some limitations, as detailed in Weaknesses.", "strength_and_weaknesses": "Strength:\n1. The paper proposes a new paradigm other than contrastive learning for self-supervised learning.\n2. The proposed algorithm is theoretically justified.\n3. The paper is written and easy to follow in general.\n\nWeakness:\n1. The algorithm still requires domain-dependent hyperparameters (e.g., epsilon and L_p norm). I wonder if there is a way to automate the choices from the data which makes the whole algorithm agnostic.\n2. The attack model is limited to additive attacks. However, some natural augmentation may require a very epsilon to realize using an additive model. For example, suppose a model would like to learn a translation-invariant representation (a shift in image/audio leads to the same semantic). In that case, a proper augmentation may be shifting or cropping if it is domain-specific, which is hard to realize by an additive model.\n3. The theory seems a bit useless to me. The error bound depends on some factors that are not measurable in practice (e.g., alpha and lambda). I wonder if it is possible to obtain some numerical bound that measures whether the learned model succeeds or not (before evaluating it on test dataset).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IluxS_ZvBXT", "reviewer": "Reviewer_Hv4e", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for self-supervised learning based on adversarial sample generation. The proposed method is based on the probabilistic interpretation of PLS and/or CCA based on graphical models and does not require class labels or classifiers. It performs best among the domain-agnostic methods on tabular, image, and audio datasets. They also provide theoretical discussions as well as experimental comparisons.", "summary_of_the_review": "Overall, the reviewers are inclined to judge this paper as acceptable. The score will be improved more if the above weaknesses and questions are adequately answered.\n\n----\n\n**Updated review**\n\nResponses from the authors and additional experiments were sufficient to answer the reviewers' questions, thus improving the score. If there is room in the paper, the following responses from the authors could be added, even in summary.\n\n> The fundamental motivation behind contrastive learning is maximizing the mutual information between two augmented views (Oord et al., 2018). However, it is inconsistent with the actual behavior in practice, e.g., optimizing a tighter bound on mutual information can lead to worse representations (Tschannen et al., 2019). Theoretical understanding of contrastive loss is still elusive. Analysis based on the assumption of latent classes provides nice theoretical insights (Saunshi et al., 2019), but the theoretical result that representation quality suffers with more negative samples is inconsistent with empirical results in SimCLR and popular contrastive learning methods. A recent study interprets contrastive loss as alignment and uniformity on the hypersphere (Wang et al, 2020), but they do not provide any theoretical guarantees on the downstream performance of the learned representation.\n\n> For the reasons presented above, we decide not to employ contrastive learning and develop a self-supervised learning method based on well-known statistical learning methods such as CCA and PLS. Our goal is not to beat SOTA performance. We want to give some insights into why training models with unlabeled data using certain loss functions leads to useful representations in the downstream task. We believe such insights are very important for the development of unsupervised representation learning in the long term.", "strength_and_weaknesses": "Strengths\n1. The proposed method is a simple but straightforward design based on probabilistic CCA and PLS and performs better than related methods.\n1. Experiments have been conducted on data sets from multiple modalities, which is sufficient to demonstrate versatility.\n1. Rather than ending with a heuristic proposal of a method, there is also a theoretical discussion.\n\nWeaknesses\n1. The abstract and introduction also mention data augmentation, but the experiments are limited to comparisons among self-supervised learning methods. For example, in the case of images, Cutmix, RandAugment, and Random Erasing are used in addition to Mixup, but no comparison with such data expansion methods is made. The reviewer agrees that there is no de-facto standard data augmentation for tabular data, but some comparisons using image and audio datasets would be possible.\n1. Approaches that maximize correlation, such as CCA and PLS, are also reasonable, while recent trends, such as (Verma et al., 2021) and (Ho and Nvasconcelos, 2020), use contrast learning. A contrastive loss with original features x, x+δ, and features x' with different semantics seems natural. The reviewer would like to know if there is any motivation for the proposed method not to employ contrastive learning.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CM-AodctOs", "reviewer": "Reviewer_3qmD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a new self-supervised learning objective that uses a notation of multi-dimensional Hirschfeld-Gebelein-Rényi (HGR) maximal correlation as the similarity measure for different views of the same sample. The paper gives some theoretical insights and experimental study on some relatively small datasets.", "summary_of_the_review": "I generally enjoy reading the paper and find the idea an interesting one. In terms of technical novelty, it is a bit incremental since the introduction of the new correlation measure is not well justified, as well as the adversarial training. In terms of empirical significance, I find it hard to evaluate based on the current toy-ish experiments.\n\nMy overall concerns are listed as follows:\n\n- Since the final objective is a lagrangian relaxation, whether the final objective can well approximate the HGR correlation is unclear. I think the natural to evaluate this is to compute the individual loss terms in the objective and see how each term is optimized. For example, if the network output indeed tends to be element-wise independent, then the constraint for HGR correlation can be satisfied.\n\n- For the novelty, as I mentioned above, it is a bit incremental. Just to add more details, I think the underlying motivation of the paper is very similar to Barlo Twins, but differently, the paper introduces a stronger regularization for the encoder network, ie, the network output is element-wise independent, which is essentially requiring the network to conduct something conceptually similar to nonlinear ICA. This is already a difficult task, and I don't understand why this will be beneficial for general data.\n\n- For the experiments, I highly suggest the authors to conduct experiments on ImageNet and compare the results to Barlo Twins. Ideally, you can directly follow the settings of Barlo Twons, which can serve as a fair comparison.\n\nPost-rebuttal:\n\nBased on the comparison to Barlo Twins, the gain can be barely observed. I can see there are differences to Barlo Twins, but the authors do not really justify why these differences are generally beneficial to generalizability, and most importantly, the empirical evidence does not support the authors' argument. Given the above reason, I am still leaning towards rejection for the time being.", "strength_and_weaknesses": "For the strength:\n\n- To the best of my knowledge, the idea seems to be new (although it sitll falls into the large category of non-contrastive learning, e.g., BYOL). The introduction of adversarial training seems interesting as well.\n\n- The formulation is interesting, especially considering the similarity / correlation measure is a variational formulation which involves a constrained maximization as an inner optimization. \n\n- The experiments show some improvement on a number of tasks including tabular data, vision data and audio data.\n\n\nFor the weakness:\n\n- The novelty is incremental given that both the introduction of multi-dimensional Hirschfeld-Gebelein-Rényi (HGR) maximal correlation and adversarial traininng can not be well justified (both theoretically and empirically). It seems to be the idea is directly connected to Barlo Twins [https://arxiv.org/pdf/2103.03230.pdf] since the final objective is essentially similar. I think a proper comparison to Barlo Twins is necessary.\n\n- The theoreical part does not inform why the proposed method is better than the others (say cross-entropy).\n\n- Althoguh the paper presents a number of tasks, they are a bit toy-ish. It could much convincing if the authors can conduct some experiments on larger datasets (say ImageNet).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "5Hw0YQHUYqP", "reviewer": "Reviewer_RgE2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a domain-agnostic self-supervised learning algorithm using adversarial perturbations in the latent space. The paper argues that existing self-supervised learning algorithms either have domain-dependent pretext tasks or have domain dependent augmentations. For instance, the masked image/language models approach requires data to be tokenized while the contrastive learning approaches require domain-dependent augmentation. Thus, to propose a domain-agnostic approach, the authors devise a pre-text task where the objective is to reconstruct the latent representation of a clean sample from the perturbed sample. For perturbation, instead of using an augmentation the paper rather uses adversarial perturbation, where the perturbation is produced by a generator. The proposed algorithm iteratively switches between learning a generator to produce the adversarial perturbation and learning the encoder to predict the latent representation of the clean sample from the adversarially perturbed sample. To avoid learning the trivial solution by the encoder, latent reconstruction is done with an orthogonality constraint where the encoders project all input data into a constant vector. To show the generality of their approach, the authors present results over tabular data, audio data, and image data.\n\n\n", "summary_of_the_review": "While I like the direction followed in the paper about proposing a domain-agnostic self-supervised learning approach I have concerns about the experimental setup, particularly proper comparisons with BYOL and also the method for doing adversarial perturbation. Furthermore, the proposed approach is still difficult to apply over text-domain and hence it is not domain-agnostic.\n", "strength_and_weaknesses": "Strength - \n\n1. The paper is generally easy to follow along and well-motivated. \n2. The approach also show theoretical guarantees on the linear probe error for their approach.\n3. Empirically the results are good across different domains and datasets.\n\nWeakness - \n\n1. I feel the paper's approach is very similar to BYOL, which also has a similar pretext task of reconstructing the latent representation of one augmented sample from another augmented image. The main difference is that BYOL uses domain augmentations while the proposed approach uses adversarial perturbation. However, there is no comparison with BYOL, at least for the image data. Further to avoid learning a trivial solution BYOL uses a predictor network on top of only one encoder(and hence asymmetric). In contrast, the paper injects an orthogonality constraint on the latent reconstructions.  How do these two approaches differ? Can authors do an ablation study between these two approaches? In particular, why would one want to use the orthogonal projection constraint as mentioned in the paper over the asymmetric networks approach as done in BYOL?\n\n2. For learning the adversarial perturbation, the paper learn a generator to produce the perturbation. However, we could also instead just use the gradient of L_adv to compute the perturbation direction, similar to single-step PGD attack[1] done for adversarial attacks. This is much simpler than learning a generator. Can the authors comment on this, and show some ablation to validate that learning a generator is indeed better?\n\n3. The proposed approach is still not domain agnostic as it cannot be applied over the text domain as it would be hard to define a perturbation over the input. This questions the domain-agnostic claim made in the paper. Masked learning approaches in contrast can be applied over both text and image data[2]. Can the authors comment on this?\n\nReferences - \n[1] Towards Deep Learning Models Resistant to Adversarial Attacks. Madry et al.\n[2] Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks. Wang et al.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject