Augmentation Component Analysis: Modeling Similarity via the Augmentation Overlaps
Authors: Lu Han
Han-Jia Ye
De-Chuan Zhan
Venue: ICLR 2023 poster (2023)

Abstract: Self-supervised learning aims to learn a embedding space where semantically similar samples are close. Contrastive learning methods pull views of samples together and push different samples away, which utilizes semantic invariance of augmentation but ignores the relationship between samples. To better exploit the power of augmentation, we observe that semantically similar samples are more likely to have similar augmented views. Therefore, we can take the augmented views as a special description of a sample. In this paper, we model such a description as the augmentation distribution, and we call it augmentation feature. The similarity in augmentation feature reflects how much the views of two samples overlap and is related to their semantical similarity. Without computational burdens to explicitly estimate values of the augmentation feature, we propose Augmentation Component Analysis (ACA) with a contrastive-like loss to learn principal components and an on-the-fly projection loss to embed data. ACA equals an efficient dimension reduction by PCA and extracts low-dimensional embeddings, theoretically preserving the similarity of augmentation distribution between samples. Empirical results show that our method can achieve competitive results against various traditional contrastive learning methods on different benchmarks.

Published as a conference paper at ICLR 2023
AUGMENTATION COMPONENT ANALYSIS:
MODELING SIMILARITY VIA THE AUGMENTATION OVERLAPS
Lu Han, Han-Jia Yo=, De-Chuan Zhan
State Key Laboratory for Novel Software Technology, Nanjing University [hanlu, yehj j @elamda..... edu.ed, zhandc@nju. edu. on
ABSTRACT
Self-supervised learning aims to learn embedding space where semantically simi- lar samples are close. Contrastive learning methods pull views of samples together and push different samples away, which utilizes semantic invariance of augmen- tation but ignores the relationship between samples. To better exploit the power of augmentation, we observe that semantically similar samples are more likely to have similar augmented views. Therefore, special description of a sample. In this paper, we model such description as the augmentation distribution, and we call it augmentation feature. The similarity augmentation feature reflects how much the views of two samples overlap and is related to their semantical similarity. Without computational burdens to explicitly estimate values of the augmentation feature, we propose Augmentation Component loss to learn principal components and
can take the augmented views as a
Analysis (ACA) with a contrastive-lie on-the-fly fly projection loss to embed data. ACA equals an efficient dimension reduction by PCA and extracts oow-dimensional embeddings, theoretically pre- serving the similarity of augmentation distribution between samples. Empirical results show that our method can achieve competitive results against various tra- ditional contrastive learning methods different benchmarks. Code available https: htt//gttttu..... com han CAgCA
INTRODUCTION
The rapid development of contrastive learning has pushed self-supervised representation learning to unprecedented success. Many contrastive learning methods surpass traditional pretext-basee methods by large margin and even outperform representation learned by supervised learning (Wu et al.) 2018; van den Oord et al. 2018: Tian et al. 2020a; He et al., 2020 Chen et al., 2020a c). The key idea of self-supervised contrastive learning is to construct views of samples via modern data 2020a). Then discriminative embeddings are learned by pulling together
augmentations (Chen et al. views of the same sample the embedding space while pushing apart views of others.
Contrastive learning methods utilize the semantic invariance between views of the same sample, but the semantic relationship between samples is ignored. Instead of measuring the similarity between certain augmented views of samples, we claim that the similarity between the augmentation distributions of samples can reveal the sample-wise similarity better. In other words, semantically similar samples have similar sets of views. As shown in Figure [ left. two images of deer create many similar crops, and sets of their augmentation results, i.e., their distributions, overlap much. In contrast, car image will rarely be augmented to the same crop as deer, and their augmentation distributions overlap little. Figure rightt we verify the motivation numerically. We approximate the overlaps between image augmentations with a classical image matching algorithm (Zitova & Flusser (2003), which counts the portion of the key points matched in the raw images. We find samples of the same class overlap more than different classes on average, supporting motivation. Therefore, we establish the semantic relationship between samples in an unsupervised manner based on the similarity of augmentation distributions, i.e., how much they overlap.
In this paper, we propose to describe data directly by their augmentation distributions. We call the feature of this kind the augmentation feature. The elements of the augmentation feature represent

---

Published as a conference paper at ICLR 2023
Approdimated Oxtrlop
0.28
Same
Diraamnt
0.15-
CIBAR000
Imeppke
sdand dbda
Aageenbic dade
axyneteise Ekthlattnn
Doloset
Figure 1: Left: semantically similar samples (e.g., those in the same class) usually create similar augmentations. The right figure indicates the same class images have higher averaged augmentation overlaps than those from different classes on four common datasets. For this reason, we learn embeddings by preserving the similarity between augmentation distributions of samples.
the probability of getting certain view by augmenting the sample as shown in the left of Figure The augmentation feature serves as an "ideal" representation since it encodes the augmentation information without any loss and not only its elements are hard to calculate, but also such sigh-dimensionall embeddings are impractical
easily obtain the overlap of two samples from it. However,
Inspired by the classical strategy to deal with nigh-dimensional data, we propose Augmentation Component Analysis (ACA), which employs the idea of PCA (Hotelling 1933) perform dimension reduction on augmentation features previously mentioned. ACA reformulates the steps of extracting principal components of the augmentation features with a contrastive- loss. With the learned principal components, another on-the-fly loss embeds samples effectively. ACA learns operable low-dimensional embeddings theoretically preserving the augmentation distribution distances.
In addition, the similarity between the objectives of ACA and traditional contrastive loss may explain why contrastive learning can learn semantic-related embeddings they embed samples into spaces that partially preserve augmentation distributions. Experiments on synthetic and real-world datasets demonstrate that our ACA achieves competitive results against various traditional contrastive learning methods. Our contributions are as follows:
We propose new self-supervised strategy, which measures sample-wise similarity via the similarity augmentation distributions. This new aspect facilitates learning embeddings.
We propose ACA method that implicitly employs the dimension reduction over the augmentation feature, and the learned embeddings preserve augmentation similarity between samples.
Benefiting from the resemblance to contrastive loss, our ACA helps explain the functionality of contrastive learning and why they can learn semantically meaningful embeddings.
2 RELATED WORK
Self-Superviedd Learning. Learning effective visual representations without human supervision is a long-standing problem. Self-supervised learning methods solve this problem by creating supervision from the data itself instead of human labelers. The model needs to solve pretext task before
is used for the downstream tasks. For example, in computer vision, the pretext tasks include
colorizing grayscale images (Zhang al. 2016), inpainting images (Pathak al. 2016).
predicting
relative patch (Doersch et al. al, 2015), solving jigsaw puzzles (Noroozi & Favaro 2016
predicting
rotations (Gidaris al.) (0018) and exploiting generative models (Goodfellow al.
2014 Kingma &
Welling 2014] Donahue & Simonyan 2019). Self-supervised learning also achieves great success in
natural language processing (Mikolov al. 2013 Devlin al. 2019).
Contrastive Learning and Non-Contrastive Methods. Contrastive approaches have been one of the most prominent representation learning strategies in self-supervised learning. Similar to the metric learning in supervised scenarios (Ye et al.] 2019 2020), these approaches maximize the agreement between positive pairs and minimize the agreement between negative pairs. Positive pairs are commonly constructed by co-occurrence (van den Oord et al. 2018 Tian et al., 2020a Bachman et al. 2209) augmentation of the same sample (He et al. 2020 Chen et 2020a Li al. 2021
Ye et al.] al.] (202), while all the other samples are taken as negatives. Most of these methods employ the InfoNCE loss (van den Oord et al. 2018). which acts lower bound of mutual information between
2

---

Published as a conference paper at ICLR 2023
views. Based this idea, there several methods that attempt to improve contrastive learning,
including mining nearest neighbour [Dwibedi et al. 2021; ?: Azabou et al. 202T] I and creating extra
views by mixing up (Kalantidis et al.) 2020) adversarial training (Hu et al. 202Tp. Another stream of methods employs a similar idea of contrastive learning to pull views of a sample together without using negative samples (Grill et al., [020)] Chen & He, (02T). ). Barlow Twins (Zbontar et al. 2021) minimizes the redundancy within the representation vector. Tsai et al.] (2021) reveals the relationship among Barlow Twins, contrastive and non-contrastive methods. Most of these methods only utilize the semantic invariance of augmentation and ignore the relationship between samples. Different from them, we propose a new way to perform self-supervised learning by preserving the similarity of augmentation distribution, based on the observation that a strong correlation exists between the similarity of augmentation distributions and the similarity of semantics.
Explanation of Contrastive Learning. Several works provide empirical or theoretical results for explaining the behavior of contrastive learning. Tian et al. (2020b); Xiao et al. (2021) explore the role of augmentation and show contrastive model can extract useful information from views but also can be affected by nuisance information. /Zhao et al. (202T) empirically shows that contrastive learning preserves low-level or middle-level instance information. In theoretical studies, |Saunshi et al.) (2019) provide guarantees of downstream linear classification tasks under conditionally independence assumption. Other works weaken the assumption but are still unrealistic (Lee et al. 2021 [oosh] et al. 202T). [HaoChen et al. (2021) focus on how views of different samples connected by the augmentation process and provide guarantees with certain connectivity assumptions. [Wang et al. (2022) notice that the augmentation overlap provides a ladder for gradually learning class-separated representations. In addition to the alignment and uniformity as shown by Wang & Isola (002)). Huang et al. (202T) develop theories on the crucial effect of data augmentation on the generalization of contrastive learning. Hu et al. (2022) explain that the contrastive loss is implicitly doing SNE with "positive" pairs constructed from data augmentation. Inspired by the important role of augmentation, we provide novel self-supervised method that ensures preserving augmentation overlap.
NOTATIONS
The set of all natural data (data without augmentation) is denoted by X. with size |X] = N. We assume that the natural data follow uniform distribution p(œ) on X, i.e., p(œ) N. va E X. By applying an augmentation method A, natural sample z E X could be augmented to another sample x with probability (a T)), so we use p(- i to encode the augmentation distribution. | For example, if is an image, then A can be common augmentations like Gaussian blur, color distortion and random cropping (Chen et al., 2020a).. Denote the set of all possible augmented data as X. We assume X' has finite size | = L and L > N for ease of exposition. Note that N and are finite, but can be arbitrarily large. We denote the encoder as fo. parameterized by 0, which projects a sample x to an embedding vector in I
4 LEARNING VIA AUGMENTATION OVERLAPS
As we mentioned in Section [ measuring the similarity between the augmentation distri- butions, i.e., the overlap of the augmented results of the two samples reveals their se- mantic relationship well. For example, in natural language processing, we usually gen- erate augmented sentences by dropping out some words. Then diffcrent sentences with similar meanings are likely to contain the same set of words and thus have high probability of creating similar augmented data. With the help of this self-supervision, formulate the embedding learning task to meet the following similarity preserving condition:
dg (So- (ā1), , So- (ā2)) x d @1), (2)))
dys is a distance measure in the embedding space R$, and d measures the distance between two augmentation distributions. Equation ( requires the learned embedding with the optimal parameter O" has the same similarity comparison with that measured by the augmentation distributions. In this section, we first introduce the augmentation feature for each sample, which is manually designed embedding satisfying the condition in Equation (L). To handle the high dimensionality and complexity of the augmentation feature, we further propose our A Augmentation Component Analysis (ACA) that learns to reduce the dimensionality and preserve the similarity.
Note that p(- #) is usually difficult compute and we can only sample from it. We omit the subscript A and directly use p(. () in the following content for convenient.
3

---

Published as a conference paper at ICLR 2023
Augmentation Component Analysis
Principal Componeat Analysis
Augmented
Insatie!
Marrix d)
Marric
Emboodingg
ACA-PC Loss
(x,)
(Lamm
W
(Prdlett
XW
R. ((x p(x.))))
pang
: (x,)
X2 p(x,(2,) (xx,
Coman Kil
Natual
Projection Loss
Embbeddings
Embooking Space
Inport Spoce
Nx (xx (8x)
X;
X,
Preserve
prat AFaag
Augmentation Feature L.).
(Pagnua
fo(x) = Fp(xx,166688]
Augmenniation Simmlariyy
The idea of learning embeddings via Augmentation Component Analysis (ACA). The upper right figure demonstrates the process of PCA. It learns PCs and projects the input feature get emheddings of data. Similarly, ACA performs PCA on the augmentation feature, which encodes all the information about the augmentation distribution. To overcome the dimensional and computational complexity, ACA employs ACA-PC loss and projection loss to learn PCs and embeddings tractably. Via ACA, our model can learn embeddings that preserve augmentation similarity for natural data.
Figure 2:
4.1 AUGMENTATION FEATURE
To reach the goal of similarity preserving in Equation ( direct way is to manually construct the feature by the augmentation distributions of each natural sample, i.e., f(ø) = [p(z) i)..... p(æ, T))T, where each element p(æ; 3 represents the probability of getting certain element æ; in space X' by augmenting æ. We omit 0 in f()) since such augmentation feature does not rely on any learnable parameters. In this case, any distance dgi defined in the space of is exactly valid distribution distance, which reveals the augmentation overlaps and is related to the semantic similarity.
Although the constructive augmentation feature naturally satisfies the similarity preserving condition (Equation (T)) (because directly use the augmentation distribution without loss of information), is impractical for the following reasons. First, its dimensionality is exponentially high, which is up to I., the number of possible augmented results. For example, even on CIFAR1O, the small-scale dataset with image size 32 x 32 x 3. L is up to 2563072 (3072 pixels and 256 possible pixel values). Second, the computation of each element is intractable, We may need an exponentially large number of samples to accurately estimate each p(æ: æ). The dimensionality and computation problems make the augmentation feature impractical both at inference and training time. Such inconvenience motivates to (1) conduct certain dimension reduction to preserve the information in low dimensional space (Section 4.2 and (2) develop an efficient algorithm for dimension reduction (Section 4.3
4.2 DIMENSION REDUCTION ON AUGMENTATION FEATURES
To deal with the high-dimensional property, employ the idea of PCA (Hotelling 1933) which reconstructs the data with principal components/ For convenience, we denote the design matrix of augmentation feature by A, where I As, æ p(æ x) (see Figure 2). We perform PCA on a transformed augmentation feature called normalized augmentation feature:
Å = AD-
where D = diag([de, dæz....., drr.), de = E. p(x 3). Based on normalized augmentation feature, we can develop an efficient algorithm for similarity preserving embeddings.
Assume the SVD of Å T with U e NNN, Ee NXL, V E 4 PCA first learns the projection matrix consisting of the top-k right singular vectors, which can be denoted as V R..kk
The vectors V are called Principal Components (PCs). Then, it projects the feature by AV to get the embeddings for each sample. The overall procedure illustrated at the top-right of Figure 2 But performing PCA on the augmentation feature will encounter many obstacles. The element of augmentation feature is not possible to estimate accurately, not to mention its high dimensionality.
Following the common knowledge in dimension reduction, call the raw high dimensional representation as "feature", and learned low-dimensional representation as "emhedding". In this paper, we use the non-centred version (Reyment & Jvreskog 1996). which is more appropriate for observations than for variables, where the origin matters more.
4

---

Published as a conference paper at ICLR 2023
Even if we somehow get the projection matrix V, it is also impractical to project the high- dimensional matrix Å. For this reason, we propose ACA make PC learning and projection process efficient without explicitly calculating elements of augmentation feature.
4.3 AUGMENTATION COMPONENT ANALYSIS
Although there are several obstacles when performing PCA on the augmentation features directly, fortunately, it is efficient to sample from the augmentation distribution p(æ [i), L.e. by performing augmentation on the natural data E and get an augmented sample æ. Being aware of this, our ACA uses two practical losses to simulate the PCA process efficiently by sampling. The first contrastive- like loss leads the encoder to learn principal components of Á, which can be efficiently optimized by sampling like traditional contrastive methods. The second loss performs on-the-fly projection of Á through the training trajectory. which solves the difficulty of high dimensional projection.
Learning principal components. ACA learns the principal components by an efficient contrastive- like loss. Besides its projection functionality, these learned principal components can also serve as embeddings that preserve a kind of posterior distribution similarity, as we will show later.
In the SVD view, UE serves the PCA projection results for samples and V contains the principal components (Jolliffe 2002). However, if changing our view, VE can be seen as the representation of each column. Since each column of Á encodes the probability of the augmented data given natural data, V preserves certain augmentation relationships, as we will show in Theorem 4.2 later. To leverage the extrapolation power of encoders like deep neural networks, we choose to design a loss that can guide the parameterized encoder fo learn similar embeddings PCA. Inspired by the rank minimization view of PCA (Vidal et al. ). we employ the low-rank approximation objective with matrix factorization, similar to HaoChen et al. (2021):
min Lmy = |
PERL*k
where columns of F store the scaled version of top-k right singular vectors, and each row can seen as the embedding of augmented data as will show in Lemma|4.1] According to Eckart- Eckart-Young Young-Mirsk theorem (Eckart & Young 1936). by optimizing f.., we can get the optimal V. which has the form VÊQ, Q R kx an orthonormal matrix. Σ and V contains the top-k singular values and right singular vectors. By expanding Equation (3), we get Augmentation Component Analysis Loss for learning Principal Components (ACA-PC) in the following lemma:
Lemma 4.1 (ACA-PC loss). Let Fx.: vd (æ), Vx E X. Minimizing Lmgf is equivalent to minimizing the following objective:
LACAPC = 2E
s, fo(a.) fo(x;)
amp(a) 0y
+ NT N pe
The proof can be found in Appendix I In ACA-PC, the first term is the common alignment loss for augmented data and the second term is form of uniformity loss (Wang & Isola 2020). Both terms can be estimated by Monte-Carlo sampling. ACA-PC is kind of contrastive loss. But unlike most of the others, it has theoretical meanings. We note that the form of ACA-PC differs from spectral loss (HaoChen et al. 2021) by adding a constant N before the uniformity term. This term similar to the noise strength in NCE (Gutmann & Hyvärinen 2010) the number of negative samples in InfoNCE (van den Oord et al. 2018). It can be proved that the learned embeddings by ACA-PC preserve the posterior distribution distances between augmented data:
Theorem 4.2 (Almost isometry for posterior distances), Assume fo is a universal encoder, ok+1
is the (l + 1)-th largest singular value of Å. dmin = min dz- and 6 I( = xz2) the minimizer 6* of LACA-PC satisfies:
20 (1 bwjaz) S (1fo-(221) - for (222)1 S (21,
dzooat (x1, x22
(21,
Ve,, E
dmnn
where the posterior distance
d2 upuan (811222) =2)==
Σ
(p x2) p x²))²
(5)
2E.
5

---

Published as a conference paper at ICLR 2023
measures the squared Euclidean distance between the posterior distribution (a( x)) plae a m))())
We give the proof in Appendix G Theorem 4.2 states that the optimal encoder for ACA-PC preserves the distance of posterior distributions between augmented data within an eπrоr related to embedding size k. As increase to NN, the error decrease to O. corresponds to the phenomenon that larger embedding size leads to better contrastive performance (Chen et al. 2020a). The posterior distribution ( x)) represents the probability that a given augmented sample æ is created by a natural sample T. Augmented data that are only produced by the same natural sample will have the smallest distance, and embeddings of those in overlapped areas will be pulled together by ACA-PC. Since the overlapped area are usually created by two same-class samples, ACA-PC can form semantically meaningful embedding space.
It is also noticeable that the optimal encoder meets the similarity preserving condition (Equation (I) but concerning the posterior distribution for augmented data not the augmentation distribution for natural data. Since what care about is the distribution of natural data, further propose projection loss that helps learn good embeddings for all the natural data.
On-the-fly Projection. As stated in the previous part, the learned embeddings by ACA-PC not only serve as certain embeddings for augmented data but also contain principal components of normalized augmentation feature. Based on this, we propose to use these embeddings to act projection operator to ensure meaningful embeddings for all the natural data. To be specific, denote the embedding matrix for all augmented data as Fouu(E RR Lx where each row F2. = So- (z)).
From Equation
3 and Fz.:
ddef f- (z), it can be easily seen that:
= Fang = =
Similar to PCA (Hotelling. 1933) that projects the original feature by the principal components V, we propose use F project the augmentation feature to get the embeddings for each natural sample. Denote the embedding matrix for natural data as Frat(E RN**) where each row F represents the embeddings of æ. We compute Fuot as follows:
Fnot_ AF ÁD - = (ČS)EQ,
where Ê,U contain the top-k singular values and corresponding left singular vectors. It is noticeable that Frat is exactly the PCA projection result multiplied by an additional matrix ÊQ. Fortunately, such additional linear transformation does not affect the linear probe performance (HaoChen et al. 2021). With Equation (6). the embedding of each natural sample can be computed as follows:
Enot a.::
Ag. pang Σ p( a) f)fo (x) = Ex-p(= (a) fo (x)
which is exactly the expected feature over the augmentation distribution. Similar Theorem 4.2 the embeddings calculated by Equation ( also present certain isometry property:
Theorem 4.3 (Almost isometry for weighted augmentation distances). Assume fo universal encoder, 1 the (k: + 1)-th largest sigular value of Å,ōæy T[ = 22), let the minimizer of LACA PC be 0 and g g(ō) Bz=p(z]a) So- (a)) as in Equation (7). then:
de derangg (11 - 20 (1 d2,02) S (lg(ā)) - g( I der (01, x2), Vx 32 E X
where 11-112, represent the Mahalanobis distance with matrix " diag([o1, 02, ..., ,k)) is the diagonal matrix containing top-k singular values and the weighted augmentation distance
1
(p(x @1)
pp(r x2))²
d2 w--aug (1..
2222
(8)
N
PA(
DEX
measures the weighted squared Euclidean distance between the augmentation distribution p( xx)
Different from Theorem 4.2 which presents isometry between Euclidean distances in embeddings
and augmentation distribution, Theorem [43 presents isometry between Mahalanobis distances. The weighted augmentation distances weigh the Euclidean distances by P=(x)) dw- can be regarded as valid augmentation distance measure d as in Equation (1) and moaa preserve such a distance. So our goal is to make embeddings of æ approaches Ep (a) J (z:). However, as stated before, the additional projection process is not efficient, i.e., we need exponentially many samples from p(æ a). We notice that samples during the training process of ACA-PC can be reused. For this reason,

---

Published as a conference paper at ICLR 2023
propose an on-the-fly projection loss that directly the current encoder for projection:
proj Em~p(a)
(1) Byto) ((
(9)
Full objective of ACA. Based on the discussion of the above parts, ACA simultaneously learns the principal components by ACA-PC and projects natural data by an on-the-fly projection loss. The full
objective of ACA has the following form:
LACA-Full LACA-PC Proj
(10)
where is trade-off hyperparameter. We also find N in Equation (4) too large for stable training, so we replace it with a tunable hyperparameter K. Here, we only display the loss in expectation forms.
The details of the implementation described in Appendix
5 A PILOT STUDY
In this section, we experiment with our Augmentation Component Analysis method on a synthetic mixture component data with a Gaussian augmentation method. In this example, we aim to show the relationship between semantic similarity and posterior/weighted augmentation distances. We also show the effectiveness of our method compared to traditional contrastive learning. In this example, the natural data & are sampled from a mixture gaussian with component:
p a
We Gaussian noise the data augmentation of a natural data sample, i.e., A(x) = where & ~ (O, snl), Concretely, we conduct our experiment on 2-D data with c = 4, प. ,, 1 and sample 200 natural data
N4 uniformly distributed on a circle with radius 2. For each component, with the index of the component as their label. For each natural datum, we augment 2 times with 8a = 4, which results in totally 1600 augmented data. We compute the augmentation probability for between æ and æ by p(æ T) and we normalize the probability for each æ.
First, we plot the distribution of posterior distances (Equation (5) for pairs of augmented data and weighted augmentation distances (Equation (8) for pairs of natural data in Figure left. The two distances appear have similar distributions because the synthetic data are Gaussian. It can be seen that data from the same component tend to have small distances, while from different components, their distances are large. low-distance areas, there are pairs of the same class, which means that the two distances reliable metrics for judging semantic similarity. In all, this picture reveals the correlation between semantic similarity and postcrior/weighted augmentation distances.
Second, we compare methods with SimCLR (Chen et al. 2020a), the traditional contrastive method and Spectral (HaoChen et al. 202I), which similarly learns embeddings with spectral theory. We test the learned embeddings using a Logistic Regression classifier and report the error rate of the prediction in Figure [ right. We also report performance when directly using augmentation feature (AF). First, AF has discriminability for simple linear classifiers. SimCLR and Spectral tend to underperform AF as the embedding size increases, while our methods consistently outperform. may be confusing since our method performs dimension reduction on this feature. But we note that as the embedding size increases, the complexity of the linear model also increases, which affects the generalizability. All the methods in Figure 3 right show degradation of this kind. However, our methods consistently outperform others, which shows the superiority of ACA. Additionally, by adding projection loss, ACA-Full improves ACA-PC by margin. Additionally, traditional contrastive learning like SimCLR achieves similar performance our methods. We think it reveals that traditional contrastive learning has the same functionality as our methods.
EXPERIMENTS
6.1 SETUP
Dataset. In this paper, conduct experiments mainly on the following datasets with RTX-3090 x4. CIFAR-10 and 1-00 (Krizevsky et al., 2009): two datasets containing totally 500K images of size 32 x 32 from 10 and 100 classes respectively. STL-10 (Coates et al. 2011): D): derived from ImageNet (Deng et al. 2009). with 96 x 96 resolution images with 5K labeled traming data from 10 classes. Additionally. l, 100K unlabeled images are used for unsupervised learning. Tiny
7

---

Published as a conference paper at ICLR 2023
Sume
SimCLR
Density
Spectral
0.14
ACA-PC
ACA-Full
0.13
0.12
Posterior Dislance
0.11
Same
Density
0.0
AF
0.08
0.07
60
0.2
Weighted Augmentttion Distance
Embedding Size
Figure 3: Synthetic experiments on mixture Gaussian data with Gaussian noise as augmentation. (a) The posterior distance and weighted augmentation distances among data sampled from the same component and different components. reveals the correlation between semantic similarity and the two distances, especially when the distance is small. (b) Comparison of linear classification performance among SimCLR, Spectral and our methods with various embedding dimensions ranging from 4 to 200. The dashed line represents the result when directly using Augmentation Feature (AF). ACA-PC outperforms SimCLR and Spectral. ACA-Full further improves.
ImageNet: a reduced version of ImageNet (Deng et al. 2009), composed of 100K images scaled down to 64 x 64 from 200 classes. ImageNee-1000 (Tian et al. (020a): subset of ImageNet, with 100-classes. ImageNet (Deng et al., 2009). the large-scale dataset with IK classes.
Network Structure. Following common practice (Chen al. 2020a bjc), we use the encoder- projector structure during training, where the projector projects the embeddings into low- dimensional space. For CIFAR-10 and CIFAR-100, we use the CIFAR variant of ResNet-18 (He) et al. 2016: Chen & He 2021) the encoder. We hidden dimension is half of the input dimension and output dimension is 64. For STL-10 and Tiny ImageNet, only the max-pes max-pooling layer is disabled following (Chen & He 2021 Ermolov et al. 2021). For these two datasets, we use the same projector structure, except that the output dimension is 128. For ImageNet, we use ResNet-50 with the same projector Chen al. (2020a).
two-layer MLP as the projector whose
Following the common practice of contrastive learning (Chen al. 2020a), we apply the following augmentations sequentially during training: (a) crops with a random size; (b) random horizontal flipping: (c) color jittering; (d) grayscaling. For ImageNet-100 and ImageNet, we use the same implementation as (Chen et al. 2020a).
Image Transformation.
Optimizer and other mizer (Kingma & Ba 2015) is used for all datasets. For CIFAR-10 and CIFAR-100, we use 800 epochs with a learning rate of 3 x 10 3, For Tiny ImageNet and STL-10, we train 1,000 epochs with a learning rate 2 10-3 3, We use a 0.1 learning rate decay at 100. 50, 20 epochs before the end. Due to hardware resource restrictions, we use mini-batch of size 512. The weight decay is 1 x 10-6 if not specified. Following common practice projected feature into sphere. For CIFAR-10, 10, we use a = 1. For the rest datasets, we use = 0.2. By default, K is set to 2. For ImageNet, we use the same hyperparameters as (Chen et al. 2020a) except batch size being 256, = 0.2 and K = 2.
papeerparammters. For datasets except for ImageNet, adam opti-
contrastive learning, we normalize the
Evaluation Protocol. We evaluate the learned representation on two most commonly used pro- tocols - linear classification (Zhang et al.) 2016 Kolesnikov et al. 2019) and k-nearest neighbors classifier (Chen & He 2021). In all the experiments,
train the linear classifier for 100 epochs. The learning rate exponentially decays from 10 to 10 6. The weight decay is 1 x 10-6, 6 We report the classification accuracy on test embeddings well as the accuracy of 5-Nearest Neighbors classifier for datasets except for ImageNet.
6.2 PERFORMANCE COMPARISON
In Table I we compare the linear probe performance on various small-scale or mid-scale bench- marks with several methods including SimCLR (Chen al. 2020a), BYOL. (Grill et al.) 2020).
8

---

Published as a conference paper at ICLR 2023
Table 1: Top-1 linear classification accuracy and 5-NN accuracy on four datasets with ResNet- 18 encoder. We use bold to mark the best results and underline to mark the second-best results. means the results are reproduced by our code.
CIFAR-10
CIFAR-100
STL-10
Tiny ImageNet Linear
method
Linear
5-NN
Linear
5-NN
Linear
5-NN
5-NN
SimCLR:
90.88
88.25
65.53
55.32
89.27
85.44
47.41
31.95
BYOL# #
91.20
89.52
64.85
54.92
88.64
86.73
47.66
32.96
SimSiam
91.06
89.43
65.41
54.84
90.04
85.48
45.17
30.41
Spectral
90.28
87.25
65.42
55.05
89.16
84.23
45.69
29.32
ACA-PC (ours)
90.35
87.38
65.69
54.57
90.08
85,86
46.08
30.97
ACA-Full (ours)
92.04
89.79
67.16
56.52
90.88
86.44
48.79
33.53
Table Left:
Top-1 classification accuracy and 5-NN accuracy on ImageNet- with ResNet- 18.
I: results are taken from (Wang & Isola 2020). *: results are taken from (Tian et al. 2020b).* means the results are reproduced by our code. Right: Top-1 classification accuracy on ImageNet with
ResNet-50, results are taken from (Chen & Hc 2021; HaoChen et al. 2021). We use bold to mark the best results and underline to mark the second-best results.
ImageNet-100
Linear
5-NN
ImageNet
Linear (100 epochs)
MoCot
72.80
SimCLR
66.5
Lalign Luniform
74.60
MoCo v2
67.4
InfoMin
74.90
BYOL
66.5
SimCLRE
75.62
62.70
SimSiam
68.1
Spectral"
75.52
61.80
Spectral
66.97
ACA-PC (ours)
75.80
62.54
ACA-PC (ours)
67.21
ACA-Full (ours) 76.02
63.20
ACA-Full (ours)
68.32
2021 and Spectral (HaoChen et al. 2021). For transfer learning benchmarks, please refer to Appendix D and Appendix [.] SimCLR uses method that uses contrastive loss. BYOL and SimSiam do not use negative samples. Spectral is similar loss derived from the idea of spectral clustering. From Table | we can see that our ACA-Full method achieves competitive results small- mid-scale benchmarks, achieving either the best the second-bes results all benchmarks except the 5-NN evaluation on STL-10. Also, ACA-PC differs from ACA-Full in the that the projection loss improves performance.
SimSiam (Chen & He
projection loss. In all the benchmarks,
For large-scale benchmarks, we compare several methods on Image 100 and ImageNet. On ImageNet-100,
compare method additionally MoCo (He al. 2020), Latign. funiform (Wang & Isola 2020) and InfoMin (Tian et al.] 20205). Note that the results of the other three methods are reported when using the ResNet-50 encoder, which has more capacity than ResNet18. Our method can also achieve state-of-the-are results among them. This means that our method is also effective with relatively small encoders even on large-scale datasets. On ImageNet, the-art art contrastive meth- HaoChen et al. 2021 and ACA-Full
we see that ACA-PC achieves competitive performance against state-of- saate-of-thh-art
ods (Chen al., 2020a c Grill et al. 2020 Chen He 202
achieves the best.
7 CONCLUSION AND FUTURE WORK
In this paper, we provide a new way of constructing self-supervised contrastive learning tasks by modeling similarity through augmentation overlap, which is motivated by the observation that seman- tically similar data usually creates similar augmentations. We propose Augmentation Component Analysis to perform PCA on augmentation feature efficiently. Interestingly, our methods have similar form as the traditional contrastive loss and may explain the ability of contrastive loss. We hope our paper can inspire more thoughts about how to measure similarity in self-supervised learning and how to construct contrastive learning tasks. Future studies may be explorations of applying ACA to learn representations of other forms of instances, such as tasks (Achille et al. 2019) and models (Wu et al. 2023).

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGE
This research was supported by NSFC (61773198, 12,669922,11111100 Collaborative Innovation Center of Novel Software Technology and Industrialization, NSF of Jiangsu Province (BK20200313)
REFERENCES
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C. Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-leariing In ICCV, pp. 6429-6438. IEEE, 2019.
Mehdi Azabou, Mohammad Gheshlaghi Azar, Ran Liu, Chi-Heng Lin, Erik C. Johnson, Kiran Bhaskaran-Natt Max Dabagia, Keith B. Hengen, William R. Gray Roncal, Michal Valko, and Eva L. Dyer. Mine your view: Self-supervised learning through across-sample prediction. CoRR, abs/2102.1 10.10000... 10106, 2021.
Philip Bachman, R. Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In NeurIPS, pp. 15509-15519, 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. ICML, pp. 1597-1607, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E. Hinton. Big self-supervised models are strong semi-supervised learners. In NeurIPS, 2020b.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, pp. 15750- 2021.
Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. CoRR, abs/2003.04297, 2020c.
Adam Coates, Andrew Y. Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In AISTATS, pp. 215-223, 2011.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255, 2009.
Jacob Devlin, Ming-Wor Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, pp. 4171-4186,, 2019.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV, pp. 1422-1430, 2015.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In NeurlPS, pp- 10541-10551, 2019.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With little help from my friends: Nearest-neighhor contrastive learning of visual representations. In ICCV, pp. 9588-9597, 2021.
Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychome- trika, (33:211-2116 1936.
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nieu Sebe. Whitening for self- supervised representation learning. In ICML, pp. 3015-3024, 2021.
Mark Everingham, S. M. Ali Eslami, Luc Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman. The pascal visual object classes challenge: retrospective. IJCV 111(1): 98-136, 2015.
10

---

Published as a conference paper at ICLR 2023
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. 2018.
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, pp. 2672-2680, 2014.
lean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent- A new approach self-supervised learning. In NeurIPS, 2020.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In AISTATS, pp. 297-304, 2010.
Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. In NeurIPS, pp. 5000-5011, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pp. 9729-999999999 90389978..220 2020.
Harold Hotelling. Analysis of complex of statistical variables into principal components. Journal of educational psychology, 24(6):417, 1933.
Qianjiang Hu, Xiao Wang, Wei Hu, and Guo-Jun Qi. Adco: Adversarial contrast for efficient learning of unsupervised representations from self-trained negative adversaries. In CVPR, pp. 1074-1083, 2021.
Tianyang Hu, Zhili Liu, Fengwei Zhou, Wenjia Wang, and Weiran Huang. Your contrastive learning is secretly doing stochastic neighbor embedding. CoRR, 108122055.18144 2022.
Weiran Huang, Mingyang Yi, and Xuyang Zhao. Towards the generalization of contrastive self- supervised learning. CoRR, abs/2111 11.00743, 2021.
lan T Jolliffe. Principal component analysis for special rypes of data. Springer. 2002.
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. NeurIPS, 13:21798-21809 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Diederik Kingma and Max Welling, Auto-encoding variational bayes. In ICLR, 2014.
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual representa- tion learning. In CVPR, pp. 1920-1929, 2019.
Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash. Mean shift for self- supervised learning. In ICCV, pp. 10306-10315. IEEE, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Jason D. Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo Predicting what you already know helps: Provable self-supervised learning. In NeurlPS, pp. 309-323, 2021.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven C. H. Hoi. Prototypical contrastive learning of unsupervised representations. In ICLR, 2021.
Tomás Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In NIPS, pp. 3111-3119, 2013.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, pp. 69-84. Springer, 2016.
11

---

Published as a conference paper at ICLR 2023
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR, pp.
2016.
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. In Corinna Cortes, Neil D. Lawrence, Daniel Lee, Masashi Sugiyama, and Roman Garnett (eds.), NIPS, pp. 91-99, 2015.
Richard A Reyment and KG Jvreskog. Applied factor analysis in the natural sciences. Cambridge University Press, 1996.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar, theoretical analysis of contrastive unsupervised representation learning, In ICML, pp. 5628-5637, 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV, pp. 776-794, 2020a.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In NeurIPS, 2020b.
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redun- dancy, and linear models. In Algorithmic Learning Theory, pp. 1179-1206, 2021.
Yao-Hung Hubert Tsai, Shaojie Bai, Louis-Philippe Morency, and Ruslan Salakhutdinov. A note on connecting barlow twins with negative-sample-free contrastive learning. CoRR, abs/2104.13712, 2021.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, 00011107003740 2018
René Vidal, Yi Ma, and S. Shankar Sastry. Generalized Principal Component Analysis, volume 40 of Interdisciplinary applied mathematics. Springer, 2016.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- 2020.
ment and uniformity on the hypersphere. ICML, pp.
Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is ladder; A new theoretical understanding of contrastive learning via augmentation overlap. CoRR, abs/2203.13457, 2022.
Xi-Zhu Wu, Wenkai Xu, Song Liu, and Zhi-Hua Zhou. Model reuse with reduced kernel mean embedding specification. TKDE, 35(1):699-710, 2023.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, pp. p 3733-3742, 2018.
Tete Xiao, Xiaolong Wang, Alexei A. Efros, and Trevor Darrell. What should not be contrastive in contrastive learning. In ICLR, 2021.
Han-Jia Ye, De-Chuan Zhan, Yuan Jiang, and Zhi-Hua Zhou. What makes objects similar: A unified multi-metric learning approach. TPAMI, 1155:11257 2019.
Han-Jia Ye, De-Chuan Zhan, Nan Li, and Yuan Jiang. Learning multiple local metrics: Global consideration helps. TPAMI, 42(7):1698-177) 2020.
Han-Jia Ye, Lu Han, and De-Chuan Zhan. Revisiting unsupervised meta-learning via the characteris- tics of few-shot tasks. TPAMI. 45(3):3721-3737, 2023.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In ICML, pp. 12310-12320. PMLR, 2021.
Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful image colorization. In ECCV, pp. 649-666, 2016.
12

---

Published as a conference paper at ICLR 2023
Nanxuan Zhao, Zhirong Wu, Rynson W. H. Lau, and Stephen Lin. What makes instance discrimina- tion good for transfer learning? In JCLR, 2021.
Barbara Zitova and Jan Flusser. Image registration methods: a survey. Image and vision computing, 21(11):977-1000 2003.
13

---

Published as a conference paper at ICLR 2023
fo
+ fo
Sample from
Sample P(XEE)
Lppooj
fo
Lat
=
=
fa
Miri Jutch
sigrese
Luni
(fō
fe
Figure 4: The implementation of ACA. Like traditional contrastive learning methods, ACA samples mini-batch from the whole natural dataset, then performs random augmentations on the mini-batch. The mini-batch sampling and augmentation create samples from p(ž) and p(æ x)) respectively. Then we use the samples estimate the values of Lum and Guooj as the figure.
A IMPLEMENTATION AND FURTHER DISCUSSION
In the previous section, we have presented the expected form of ACA. Thanks to its form, we can efficiently optimize ACA by Monte-Carlo sampling, making the problem tractable.
For convenience of illustration, we decompose the ACA-PC loss into two parts, i.e. LACA fali + Lun- For the first part, Laii serves as the alignment loss in traditional contrastive learning, which maximizes the inner product similarity between augmented samples from the same natural data:
Cali 2E
#>-p(m) pippla; [m) fo( fo(
(11)
we use the mini-batch of natural sample to estimate Em--(a)- And we just use one sample to estimate (a) and Enyopp(a respectively. This leads to the traditional contrastive learning procedure sample a mini-batch of natural data, augment it twice, compute and maximize the similarity of two augmented data.
For the second part, Luni minimize the inner product similarity of augmented data from the marginal distribution:
Luni = NE
[fo(z)) fe(x)))
(12)
We use the in-batch augmented data to estimate -mp-p)(r)) Notably, two augmented samples randomly sampled are hardly augmented by the same natural sample. Therefore. following common practice (Chen et al. 2020a), we use two augmented data that are created by augmenting two different natural data to compute this term. Additionally, we find that N in Luni too large to perform stable numerical computation. Thus in our implementation, replace the N with a tunable noise strength K.
For Lpojj, it is not efficient to fully sample from p(æ Φ). However, it is notable that:
[lo(z) fo(x,) fo (x,)
Cpro Em)((()) [[) E [op()
--p(a) mamplmma)) E
It has the same expectation subscript as Cali- So the samples. Pooj is computed along with Cali during training, i.e., the principal component learning and projection are done simultaneously. That is why call Epoj "on-the-fly projection".
the same strategy LACA-PC and reuse
The overall implementation of ACA is illustrated in Figure And the algorithm is illustrated in Algorithm
Discussion on the relation with traditional contrastive learning. As is described in this section. ACA-PC takes a similar form as traditional contrastive learning methods. Similar to them, ACA- PC maximizes the inner product similarity of two views from the sample by Equation (I]) and minimizes the squared inner product similarity of views from different samples. Note that we have proved that the learned embeddings by ACA-PC function the principal components of the augmentation feature and preserve the posterior augmentation distances (Theorem 4.2). We believe traditional contrastive loss also has the similar functionality ours. Due to the strong correlation
14

---

Published as a conference paper at ICLR 2023
between augmentation overlap and semantic similarity, this may explain contrastive learning can learn semantically meaningful embeddings even though they ignore the semantic relationship between samples.
Discussion approximation in implementations. There are several approximations to stabilize the training. First, we replace the factor in Equation (12) with a tunable noise strength Usually, complex model like DNN,
the number of samples very large in common datasets. When it is unstable to involve such a large number in the loss. Therefore, we tune small and find it works well. But we also note that use N in our synthetic experiment in Section for the dimensionality and model is not too complex. The superior results proves effectiveness of our theory. Second, we normalize embeddings to project them into sphere, which equals replacing the inner product with cosine similarity. We find this modification improves the performance from 81.84% 84% to 91.58%.
Algorithm Augmentation Component Analysis Algorithm
Require: Unlabeled natural dataset { Augmentation function A Encoding model so param- eterized with 0; projection parameter at Noise Strength K; Batch size B.
1: for sampled minibatch { do
2: for 1, B do
(1) A(;.), (2) A(;:)
3: m
4: end for
5: LACA-FaR = -B - fo (a) (2)
K + B(B-1) Eies(fo (x) (2) (f((a
(o) [o(aa")/0o00""
11
6: update with w.r.t 2 LACA Full.
7: end for
8: return fe
B EFFECT OF AUGMENTATION OVERLAPS
Like contrastive learning, method relies the quality of augmentation. Therefore, we in- vestigate the influence of different augmentations and reveal the relationship between distribution difference and the linear probe performance CIFAR10. The augmentation distribution is estimated by augmenting 10 times for a subset of random 2000 pairs of samples with the number intra-class and inter-class pairs being 1000 respectively. Note that is stated in Section 4.1 even CIFARI 10, the actual value of L is exponentially large (up to 2563072).
impossible to accurately estimate a distribution so many possible values. But we notice that for neural networks, many operators can reduce the possible number of values, like convolutions and poolings. Following this observation and make the computation efficient, we descrete the color into 8-bit for each channel and use max pooling operation to get a 4 x 4 picture. by this kind of approximation, the number of L reduces to can noted that the augmentation distribution of each sample covers only small region. It is enough to estimate the distribution by sampling. For memory restriction, we cannot fully estimate the weighted augmentation distance Theorem 4.3 Because we cannot store all possible values for (x)) Instead, use the Hellinger distance as the distribution distance measure:
g48, Seems still too large,
di (o 52) = Σ VVll Z1)
Vplæ T22)
2
N
Hellinger distance ranges [0, 2], making the comparison clear.
We list the experimented augmentation here:
1. Grayscale: Randomly change the color into gray with probability of 0.1.
2. HorizontalFlip: Randomly flip horizontally with probability 0.5.
3. Rotation: Randomly rotate image with uniformly distributed angle in [0, m
4. Color,Jitter: Jitter (brightness, contrast, saturation, huc) with strength (0.4, 0.4, 0.4, 0.1) and probability 0.8.
15

---

Published as a conference paper at ICLR 2023
Table 3: Histogram (HIST) of distribution distances and linear probe accuracy (ACC) when using different augmentations on CIFARIO. Note that HIST is estimated in the input space. It is property of augmentation, regardless of learning algorithm. We aims investigate the different augmentation overlaps caused by different augmentation, and reveal its connection between learned model. "Same" denotes the distance between samples with the same semantic class, and "Different" means different classes. The existence of overlap and relationship between intra-/inter-class distances affects the performance.
Grayscale
HorizontalFlip
Rotation
Sin
D.test
100
HIST
Aspaneiittan Delazoos
ACC
27.05%
29.67%
33.50%
ColorJitter
ResizedCrop
Augs in SimCLR
Vime
Sem
Som
HIST
30.
Agmmrration Distnoss
ACC
59.74%
75.98%
92.04%
5. RrsizedCrop: Extract crops with a random size from 0.2 to 1.0 of the original area and a random aspect ratio from 3/4 4/3 of the original aspect ratio.
6. Augs in SLRCLR: Sequential combination of 5,4,1,2.
In Table we display the histogram (HIST) of intra- and inter-class augmentation distribution distances. ACC displays the linear probe performance on the test set. From the table, the following requirements for good augmentation can be concluded: (1) Existence of overlap. For the upper three augmentations. The "scope" of augmentation is small. As a result, most of the samples do not overlap. This makes embeddings lack the discriminative ability for downstream tasks. On the contrary, the lower three create overlaps for most of the samples, leading much better performance. (2) Intra-class distance is lower than inter-class. Compared to ColorJitter, ResizedCrop makes more intra-class samples have lower distance. So ResizedCrop outperforms ColorJitter, SimCLR augmentation surpasses these two for the same reason. Interestingly, we find that the same phenomena appear when using other contrastive methods like SimCLR. It shows that these methods somehow utilize the augmentation overlap like our method.
C PERFORMANCE CURVE
In this section, we illustrate the performance curve throughout training. We aim to demonstrate the functionality of projection loss and show that ACA method leads to better performance. The compared traditional contrastive learning method is chosen to be SimCLR, for the reason that our method only differs from SimCLR in the loss, with all other things (architecture, optimizer and other shared hyperparameters) identical. Also, we do not introduce extra mechanisms like momentum encoder (BYOL, MoCo) and predictor (BYOL, SimSiam).
Figure 5 shows the performance curve along with the projection loss on the CIFAR-10 dataset. The left figure shows the projection loss. We that in the early stage of training, the projection loss
16

---

Published as a conference paper at ICLR 2023
0.22-
Prijedonn Lo8
0.0---.-
JNN INNN Accuney 10
600
0004
Rm
Epoch
Epoch
Epoch
Figure 5: Values of projection loss and accuracy during training. Left is the projection loss. It rises in the early stage and is stable later, showing that the projection loss prevents the embeddings of natural data from deviating from the center of augmentation distribution. The mid and right figures show the performance curve measured by kNN accuracy and test accuracy. ACA-PC achieves similar performance as SimCLR but ACA-Full is better during the training process.
Table 4: We compare various SSL methods on transfer tasks by training linear layers. Only a single linear layer is trained on top of features. Simple random horizontal flips are used. Results except ours are taken from Koohpayegani et al. (202T). Our method can achieve competitive results with other contrastive learning methods despite short epochs, especially that can surpass 1000-epoch SimCLR.
Method
Epochs
CIFARIO
SUN
Aircraft
DTD
Caltech
Flowers
Mean
Supervised
72.30
93.60
61.90
66.70
61.00
74.90
91.50
94.50
94.70
78.90
SimCLR MoCo v2
1000
72.80
90.50
60.60
49.30
49.80
75.70
84.60
89.30
92.60
74.00
800
72.50
92.20
59.60
50.50
53.20
74.40
84.60
90.00
90.50
74,20
BYOL
1000
75.30
91.30
62.20
67.80
60.60
75.50
90.40
94.20
96.10
79.20
BYOL-asym
54.00
52.10
73.40
86.20
90.40
92.10
74.30
MoCo v2
200
70.40
91.00
57.50
47.70
51,20
73.90
81.30
88.70
91.10
72.60
MSF
200
70.70
92.00
$9.00
60.90
$3.50
72.10
89.20
92.10
92.40
75.80
MSF-w/s
200
71.20
92.60
59.20
55.60
53.70
73.20
88.70
92.70
92.00
75.50
ACA (ours)
100
72.54
90.01
59.17
49.61
56.51
73.78
83.27
89.92
90.18
74.07
will increase. It reveals that the natural data will deviate from the center of augmentation distribution. is harmful to the performance of the model. With the help of projection loss, the embeddings of natural data will be dragged back to their right position, the center. The mid and right figures illustrate the performance curve during training. With only ACA-PC loss, the model can only achieve similar performance during training. But the ACA-Full loss will help improve performance during training. Also, we can see that ACA starts to outperform SimCLR and ACA-PC by a considerable margin from about 50 epochs. This happens to be the epoch in which the projection loss increases to its stable level. Therefore, pulling the natural data to the center of its augmentation helps to learn better embeddings.
D TRANSFER TO OTHER DATASETS
Following Chen et al. (2000a) we evaluate the self-supervised pre-trained models for linear classifi- cation task on 10 datasets as it is conducted in MSF paper (Koohpayegani et al. (001). The results are reported in Table 4 All the results other than ACA are taken from Koohpayegani et al. (202T). Although our method is trained with fewer epochs, it achieves competitive results with contrastive learning methods. Notably, surpasses the 1000-epoch SimCLR which differs from our method only in loss. It shows that the embeddings learned by method also transferable to other downstream tasks. We think it is due to the universality of the correlation between augmentation similarity and semantical similarity across these benchmarks.
17

---

Published as a conference paper at ICLR 2023
Table 5: We compare models the transfer task of object detction. We find that given similar computational budget, our method is better than SimCLR, with shorter training time. The models trained on the VOC trainvall 17++12 and evaluated on the test07 set. We report average over 5 runs.
Method
Epochs
APs0
AP
AP75
Sup. IN
81.30
53.50
58.80
Scratch
60.20
33.80
33.10
SimCLR (Chen et al.
2020a
200
81.80
55.50
61.40
MoCo v2
Chen al.
2020c
200
82.30
57.00
63.30
BYOL Grill et al.
2020)
200
81.40
55,30
61.10
SwAV CCron et al.
2020
200
81.50
55.40
61.40
SimSiam (Chen & He
[221
200
82.40
57.00
63.70
MSF (Koohpayegani et al.
200
82.20
56.70
63.40
MSF w/s (Koohpayegani et al.
2021
200
82.20
56.60
63.10
ACA-Full (ours)
100
82.25
56.14
63.05
E TRANSFER TO OBJECT DETECTION
Following the procedure outlined in ?, we use Faster-RCNN Ren et al. (2155) for the task of object detection on PASCAL-VOC (EGELLL-V((Vvrrnghaaa Everingham et al. 2015). We use the code provided at MoCo repository" with default parameters. All the weights are finetuned on the t trainval07 +22 set and evaluated on the test07 set. We report an average over runs in Table 5 Despite the shorter training epochs, method can achieve better results than SimCLR, especially outperform by large margin AP75(> 1%).
F PROOF OF LEMMA
For convenient, we define M = AT Á. The elements of M are:
p(an
1 1))((22 x))
Ma. 22
21 22 X'
(13)
de
des
mEX
Expanding Equation 3 we get:
Lmf Σ
(Mares FT. F
Σ (Majar - Vd V do(x)))))) fo(x)))
#1 EX
const - 2
Σ
der d Meresfo fo(xz) +
Σ (fo(aa) fo fo(x2))²
const
Σ
Σ
p(m 2)p(22 x)fo(an) To((2))))))))))))))
Σ
d de, (fo(a)) (x²2))
18

---

Published as a conference paper at ICLR 2023
multiply by p( N and replace dæ with Σs p(æ x) = Np)(w)). The objective becomes:
min 2 Σ
p(an #)p(2> ()p(m) [o((2)) Σ
N
Σ
2E
(mp(a) A(m ()) A2,
fo( fo (222)
NE spal (()) fa
LACA-PC
G PROOF OF THEOREM 4.2
As in Appendix we define M == A A. By Eckart-Young Mirsky theorem (Eckart & Young 1936), the minimizer F of |M - FF [|g. must have the form VEQ where V. V. contain the top-k singular values and corresoonding right singular vectors of Á, Q R is some orthonormal matrix with Q' Q = I. Since we let Fa V fo (2x), then the minimizer 0" must satisfy
δ 2 (( Q 022)())
fa (a)) Q
vd
dd
where @ is the element-wise multiplication. For convenience, we use to denote i-th largest singular value, u;; to denote the element of i-th left/right singular value corresponding z/x. When p()) = - dæ = NDA(Ex)= = pa(a) Then the posterior distance: p(a)
dpost (x) 2x2) Σ
(P.
an) PA( ! x²)))²²
x
p(
T)p(2
p @ a)p(())
Σ
(A(1)
(a(
xex
p(x
p(æ²
x))
d
daz
sex
Agu
Agus
2
/dzi
Vduz
#cX
N
(Z)
o,u;(
2 (x))
Vden
Vduz
TeP
1=1
(14)
(x(20))
11((22
doe
#eX
1x(21)
Vi vx(202)
oe (x11
Uy (xx2)
Σ
034 02 (1)
)(
da,
ddn,
ddn,
/ da
BER '.3!
(1(1)))
(2((22) ) te pa((a))
ve (2(2)
O.O.
Σ
us (z)).
daz
Vdzi
Vdna
xEX )8i.
U: U1(2))
ve(2z2)
)( ve (21))
ne (2(22))
Vilan
/ daz
Vdng
1
N
(x(1)
11(222) )z
ddzi
Vdza
19

---

Published as a conference paper at ICLR 2023
(I) is due the orthogonality of singular vectors. Note that:
N
Us (2(21))
(x2222
v dae
(x(1
(x22
(x11)
1x(22) )²
V do:
vdæ:
dd=
i=
(2(1)
v:(122)
V dlas
Vdnaz
vf(1) )
(2222
0i ((x1))ux(222)
+
Σ
2
day
daz
v dag v day
1=1
1=1
28
da
day
v Vd V /dz
(2)
+
(1 )m))))))
daz
2 (1
)
dini
(2) can be deduced by considering conditions whether æ; 22 not. Then:
(x)) - So (2:2))1
0i 1x(111
21(222) j
V dos
v /dz
1=1
N
(s( )²
= Ppost (211 202)
Σ
of(
(S depostar (201, 202))
Vdzi
V Vdas
o Σ 12(211)
(2(222) )²
2 doo [I1, (2)
Vdln
dang
im
0(((1))
v2(222) 2
2 dBost (01))22
k+11
V /dz,
Vdæs
20
2 (poottaa (211
(1 dmn)r))
dian
Therefore, we have proved Theorem 4.2
H PROOF OF THEOREM 43
similar Appendix G
20

---

Published as a conference paper at ICLR 2023
drrang (01, (22 Npa(x)
Σ
(p(x) x1) pp(x x22))²
=EX
p(ar Tz1)
p( | Tz))
VNpa(z)
VNpA(œ)
xEN
p( x1)
p(a x11)
vd
Vd
xEX
AAe A
Σ
Σ 0, (1 (ou))))))))
xEx
1=l
Σ
Σ (u, - ( (1))))oo)
DEX
Σ
Σ ((((0))o00( (d us (x2))
sEX
Σ
ur (x2)) Σ (x))x(())
øEX
Σ 0.00 (u. (z u. (3,)))
1,7
Σ (u,(z)) - 2;(x)))²
(1) is due to the orthogonality of singular vectors. And g takes the following form:
g( ( = Q (ofu1(z). oz )xux
Thus.
(9((1)) g( Σ ( uz (I²))²
N Σ
d2 aaa (0). T22)
(u,()))) - ux(x2))² (S dz d (51, 222)
i=k+1
d2 a-g (0), (22 o 1+11
(u; 2x(xx))²
i=1
da awaug (0), x22 20 1 60,02)
I ABLATION STUDY ON PARAMETER AND K
We conduct ablation experiments on the parameter a and K. a is the trade-off parameter between ACA-PC loss and projection loss Equation 110 K act the noise strength for ACA-PC, which replaces N in Equation
Figure 6 shows the effect of a and K on different benchmarks. It can be seen that is necessary to improve the performance of ACA-PC. A certain value of a helps the model to achieve better results. However, a large value of degrades the performance. The same phenomenon is the same on K.
21

---

Published as a conference paper at ICLR 2023
CIEARSO
CITARIO
003
100
142
(a)
(b)
(c)
CIFAROO
(d)
(e)
(I)
Figure 6: Ablation studies the effect of and K. We report linear classification and 5 5-nearest neighbor accuracy on different datasets with the ResNet-18 18 encoder. The upper figures illustrate the effect of on different datasets. The lower figures illustrate the performance K.
COMPARISON OF NEAREST NEIGHBORS
We randomly select 8 samples from the validation of ImageNNe-1100 (Tian al. 2020a). Then use the encoder learned by our ACA method and SimCLR (Chen et al. 2020a) to extract features and investigate their nearest neighbors of them. The left-most column displays the selected samples and the following columns show the nearest neighbors. The samples labeled as different classes are marked by the red box. We also annotate the distance between the samples and their nearest neighbors. First, we can see that even though utilizing the augmentation in a different way, ACA achieves similar results as traditional contrastive learning. Both them can learn semantically meaningful embeddings. However, together images that are similar in the input space, i.e., creating similar augmentation, while SimCLR sometimes has neighbors that seem different.
can see that ACA tends to learn embeddings that pull
22

---

Published as a conference paper at ICLR 2023
ACA
SimCLR
1-5.56
d-1.69
d-7.7
1-1.8
d-1.1
1.8.
2-033
2.04
2-0.5
d-0.93
d 1-1.05
d
1.07
1-1.12
d 1.12
d-1.02
1.36
1.400
1-1.3
1.49
d=1.81
d=1.82
d=1.83
d=1.84
.48
1=1.86
d=1.74
1-1.7 79
1
1=1.98
2
d=1.6
d=1.71
1.80
d=
1.84
d=1.93
d 2-1.10
4
2-1.10
2,20
d 2.21
d=0.92
1=1.28
1=1.29
1-1.29
d= 1.34
d=1.27
1.1.1
d-1.34
d=1.36
1=1.52
d-1.31
1.4.3
d-1.72
1-1.82
1.93
d=1.85
d=1.1
d=1.95
d=1.97
d=2.00
d=1.97
1.9.
2=2.27
d=2.33
2-3.77
d=1.29
1=1.51
1=1.51
1.5.22
1-5.33
d=1.82
1-9.
d
00
d-2.03
2-0.7
d=1.14
d=1.20
d=
1.20
d=
10
d=
1.21
d=1.25
d=1.52
d=1.55
1=1.5
1.57
1-1.38
1-38
1-4.43
d-1.55
1-5.7
1-6.
1.84
d 1.93
d 1.95
1.95
Figure 7: The nearest neighbors of selected samples in the embedding space of ACA and SimCLR. The images were taken from the ImageNet- 100 validation set. Distances between selected samples and their nearest neighbors learned by ACA tend to pull together images that are similar in the input space, i.e., creating similar augmentation. While SimCLR sometimes has neighbors that seem different.
annotated above each picture. We
that the embeddings
23


Reviews:
[{"review_id": "fYgl_McjLm", "reviewer": "Reviewer_fbkG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Augmentation Component Analysis (ACA), which employs the idea of PCA to perform dimension reduction on augmentation features. ACA reformulates the steps of extracting principal components of the augmentation features with a contrastive-like loss. With the learned principal components, another on-the-fly loss embeds samples effectively. ACA learns operable low-dimensional embeddings theoretically preserving the augmentation distribution distances.", "summary_of_the_review": "See Strength and Weaknesses above\n", "strength_and_weaknesses": "**Strength** \n1. The idea of augmentation component analysis is novel and interesting.\n2. The theoretical analysis of this work is promising.\n\n**Weaknesses** \nThe experimental results are too weak.\n\n1. Self-supervised learning aims to transfer the learned representations or whole network parameters into various downstream tasks. However, I do not see any transfer learning experiments in this paper. Could you provide more transfer learning experiments, for example, linear evaluation and fine-tuning in fine-grained classification tasks, semi-supervised learning, and object detection/segmentation?\n\n2. The improvements of this method are very marginal. From Table 1 and Table 2, ACA-Full only surpasses the second-best performance 0.5% in most cases, which needs to be more convincing.\n\n3. The comparison methods in Table 1 and Table 2 are outdated. I highly recommend the author compare ACA-Full with the latest contrastive learning methods. For example, [1][2][3][4][5][6][7]. Moreover, the convergence rate and final accuracy highly depend on the methods. For ImageNet experiments, the authors should at least train the model for 200 epochs (or even longer) to make sure all methods are fully converged.\n\n[1] With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations\n[2] Solving Inefficiency of Self-supervised Representation Learning\n[3] Unsupervised Learning of Visual Features by Contrasting Cluster Assignments\n[4] Mean Shift for Self-Supervised Learning\n[5] Ressl: relational self-supervised learning with weak augmentation\n[6] Barlow Twins: Self-Supervised Learning via Redundancy Reduction\n[7] AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations From Self-Trained Negative Adversaries\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LymICHuWk9y", "reviewer": "Reviewer_1C2x", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new method for self-supervised representation learning based on data augmentation. Unlike prior work in this line of research, the proposed method considers semantic similarity (e.g., class equivalence) between different instances as well as semantic invariance between different views of the same instance. The key idea is that if two training instances are semantically close to each other, their augmented versions (i.e., different views) will be largely overlapped. Conceptually, the idea can be implemented by comparing posterior probabilities of augmented examples given original ones, called augmentation features, which however is computationally intractable due to the prohibitively large number of potential augmented examples and the difficulty of estimating such probabilities. The authors thus propose to simulate principal component analysis (PCA) of augmentation features through a neural network encoder trained by two loss functions; they also prove that the loss functions allow the encoder to produce PCA projection results without computing the augmentation features explicitly. The proposed method outperforms prior work in the same direction and provides theoretical interpretation of contrastive methods for self-supervised representation learning. ", "summary_of_the_review": "The paper presents a new idea for self-supervised representation learning, which enables us to consider both semantic similarity between different instances and semantic invariance between different views of the same instance. The implementation of the idea is solid theoretically and achieves outstanding performance on multiple public benchmarks. These positive points outweigh my concerns on lack of experiments, and thus I believe the paper is worthy to be introduced to the community. \n", "strength_and_weaknesses": "[Strengths]\n- The main idea about relations between semantic similarity and similarity between augmentation distributions is new in self-supervised representation learning and sounds reasonable. The idea also has been justified empirically. \n- The way of simulating the direct yet impractical implementation of the idea using a neural network and two loss functions is interesting, solid (its validity is proven in the appendix), and worth to be introduced to the community. \n- The proposed method achieved best scores on multiple benchmarks for self-supervised representation learning. \n\n\n[Weaknesses]\n- Clarity issues: (1) The third paragraph of Section 1 is hard to grasp; it will be useful if a figure that conceptually illustrates the main idea is added. (2) The notion of augmented sample x is not clear and seems not consistent in Section 3 and 4. In Section 3, x is introduced as if it is a new view of a natural image (i.e., an augmented version of the natural image), but in Section 4 it is coupled with all natural images to form posterior probabilities. \n- Lack of experiments: The proposed method needs to be compared with latest approaches to self-supervised representation learning. Also, I would recommend evaluating the proposed method on other epochs (e.g., 200, 400, 800) like prior work for the ImageNet linear classification benchmark. Moreover, advantages of the learned model have to be also validated for downstream tasks other than image classification, i.e., transfer learning setting for object detection and instance segmentation.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "B2g_MvhZRAJ", "reviewer": "Reviewer_LVcG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper empirically claims that the overlap among the augmentation distributions from a similar category is much higher than those from dissimilar categories. Based on this discovery, this work establishes the semantic relationship between samples in an unsupervised manner based on the similarity of augmentation distributions. Technically, the paper proposes a new self-supervised loss function via maximizing the similarity of samples from the same augmentation distribution and making those from different augmentation distribution as orthogonal as possible for further enhancing the representation learning. Theoretically, this work is intended to explain why contrastive learning can effectively work with its proposed Augmentation Component Analysis.", "summary_of_the_review": "The paper need further clarify the mentioned items before it is accepted for publishing.", "strength_and_weaknesses": "Strengths:\n+\tThe perspective of augmentation overlaps is interesting for contrastive learning. Especially for those samples from the same category, the discovery of augmentation overlaps is much useful for representation learning in an unsupervised way. \n+\tThe loss functions proposed in this paper are verified their effectiveness for self-supervised representation learning. Especially, a projection loss is designed for compacting the representation from the same samples, which shares similar sprit with the prototype learning in supervised learning. The difference lies in the projection loss is mainly for the augmentation distribution of a single sample.\n\nWeaknesses:\n-\tIn this exploration of augmentation overlaps, the similarity between different samples don’t seem to be explained very well in this paper. And, there also seems to have no corresponding loss function or theoretic derivation can prove the model could pull the semantically similar samples close, as showed in the left of Figure 1.\n-\tThe analysis of dimension reduction with the idea of PCA is not suitable to derive the similarity constraint in Eq. (4). Besides, the ACA-PC loss is similar to the function appeared in [1].\n-\tPerformance gain of ACA-PC can be very marginal to prove the effectiveness of the proposed ACA. \n-\tThere exist many sentences in this paper that are hard to digest. E.g., “we claim that it is the similarity between the augmentation distributions of samples, …, that reveals the sample-wise similarity better” (page 1), “It seems that we can obtain the solution directly without further learning, however, not only the elements in the augmentation feature are hard to calculate, but also such high-dimensional target embeddings are impractical to use” (page 1),, “In addition, the resemblance between the objectives of ACA and traditional contrastive loss may explain why the latter can learn semantic-related embeddings” (page 2).\n\n[1] HaoChen J Z, Wei C, Gaidon A, et al. Provable guarantees for self-supervised deep learning with spectral contrastive loss[J]. Advances in Neural Information Processing Systems, 2021, 34: 5000-5011.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AIn4HfOMsC", "reviewer": "Reviewer_gr8r", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presented a self-supervised learning method based on the augmentation distribution of the image. Specifically, the paper is motivated by the intuition that semantically similar images tend to have similar augmented views, and proposed a method called ACA to learn the augmentation principal components of data. The method is theoretically deduced to maintain and measure the discrepancy between the augmentation distribution across samples. The method is validated on many benchmarks (cifar, stl, imagenet), and shows competitive performance when compared with several existing models.", "summary_of_the_review": "I think this paper is borderline and I lean toward borderline rejection. The paper is based on valid motivation, the method seems to be relatively novel and sound, and the experiments are in general nicely organized. However, the paper lacks sufficient experimental results that support the superiority of the presented method, as the majority of the reported performance is lower than previously reported results. Unfortunately, this is the majority component of the experiments section, and as the paper is framed to present a method that could surpass/ provide comparable performance w.r.t existing methods, the experimental results outplay the methodology contribution imho.", "strength_and_weaknesses": "Strength\n- The proposed method is theoretically sound and complete. The deduced loss function seems new and interesting.\n- The paper provided interesting pilot study and also interesting small experiments for motivation purposes.\n- The evaluation experiments are in general complete and nicely conducted. The method is validated on many datasets, with sufficient benchmarking.\n\n\nWeaknesses\n1. Major concern: model performance\n- The model performance on several datasets seems to be lower than previously reported results. Specifically, check [1, 2] where many numbers are greater than the presented method. Unfortunately, as the major experimental result of the presented method is framed to be about surpassing previous state-of-the-art (that is the only experiment), the presented numbers are just not satisfactory enough, which greatly weakens the paper’s contribution.\n\n2. The related works and the references lack more recent works.\n- There are many contrastive learning/self-supervised learning methods in 2021 and 2022 that are not included in related works. For example, check [2-5].\n\n3. The writing and organization of the paper could be improved.\n- The abstract could be improved. Specifically, I suggest rewording the first sentence and the 4th sentence.\n- Section 4.1 and section 4.2 are a bit wordy and might be simplified. The computation overload is mentioned many times in two adjacent sections.\n- Some of the preliminary in section 3 could be moved into related works, especially the second paragraph, which has much overlapping information with section 2.\n\n\n4. Minor comments:\n- I suggest rewording Section 3 first sentence.\n- Related works the Self-Supervised learning section should mention non-contrastive methods as well, like BYOL. Instead, the paper put non-contrastive methods inside the contrastive learning section and implicitly infer them as Contrastive Learning methods without using negative samples, which is a bit controversial because BYOL is not always referred to as contrastive methods though some studies suggest that they intrinsically are.\n- What is the connection between this method and Barlow Twins?\n- It might be more beneficial if the authors could explore other use cases of the presented learning method rather than just performing classification. For example, if the proposed loss function could provide an “augmentation spectrum” of certain data, could that provide more informative latent space based on some other metrics when compared to other methods?\n\n\n[1] Ermolov, Aleksandr, et al. \"Whitening for self-supervised representation learning.\" International Conference on Machine Learning. PMLR, 2021.\n\n[2] Azabou, Mehdi, et al. \"Mine your own view: Self-supervised learning through across-sample prediction.\" arXiv preprint arXiv:2102.10106 (2021).\n\n[3] Zbontar, Jure, et al. \"Barlow twins: Self-supervised learning via redundancy reduction.\" International Conference on Machine Learning. PMLR, 2021.\n\n[4] Tsai, Yao-Hung Hubert, et al. \"A note on connecting barlow twins with negative-sample-free contrastive learning.\" arXiv preprint arXiv:2104.13712 (2021).\n\n[5] Kalantidis, Yannis, et al. \"Hard negative mixing for contrastive learning.\" Advances in Neural Information Processing Systems 33 (2020): 21798-21809.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster