Improving Transferability of Representations via Augmentation-Aware Self-Supervision
Authors: Hankook Lee
Kibok Lee
Kimin Lee
Honglak Lee
Jinwoo Shin
Venue: NeurIPS 2021 Poster (2021)
Keywords: transfer learning
self-supervision
self-supervised learning
contrastive learning
data augmentation

Abstract: Recent unsupervised representation learning methods have shown to be effective in a range of vision tasks by learning representations invariant to data augmentations such as random cropping and color jittering. However, such invariance could be harmful to downstream tasks if they rely on the characteristics of the data augmentations, e.g., location- or color-sensitive. This is not an issue just for unsupervised learning; we found that this occurs even in supervised learning because it also learns to predict the same label for all augmented samples of an instance. To avoid such failures and obtain more generalizable representations, we suggest to optimize an auxiliary self-supervised loss, coined AugSelf, that learns the difference of augmentation parameters (e.g., cropping positions, color adjustment intensities) between two randomly augmented samples. Our intuition is that AugSelf encourages to preserve augmentation-aware information in learned representations, which could be beneficial for their transferability. Furthermore, AugSelf can easily be incorporated into recent state-of-the-art representation learning methods with a negligible additional training cost. Extensive experiments demonstrate that our simple idea consistently improves the transferability of representations learned by supervised and unsupervised methods in various transfer learning scenarios. The code is available at https://github.com/hankook/AugSelf.


Improving Transferability of Representations via Augmentation-Awan Self-Supervision
Hankook Leel
Kibok Lee2 Kimin Lee
Honglak Lee Jinwoo Shin
Korea Advanced Institute of Science and Technology (KAIST)
University of Michigan
Amazon Web Services
University of California, Berkeley LG AI Research
Abstract
Recent unsupervised representation learning methods have shown to effective range of vision tasks by learning representations invariant to data augmentations such as random cropping and color jittering. However, such invariance could be harmful to downstream tasks if they rely on the characteristics of the data augmen- tations, e.g., location- or color-sensitive. This is not an issue just for unsupervised learning; we found that this occurs even in supervised learning because it also learns to predict the same label for all augmented samples of an instance. To avoid such failures and obtain more generalizable representations, we suggest optimize an auxiliary self sul-ssueer loss, coined AugSelf. that learns the difference of augmentation parameters (e.g., cropping positions, color adjustment intensities) between two randomly augmented samples. Our intuition is that AugSelf encour- ages to preserve augmentation- aware information in learned representations, which could be beneficial for their transferability. Furthermore, AugSelf can easily be incorporated into recent state-of-the-ar representation learning methods with a negligible additional training cost. Extensive experiments demonstrate that our simple idea consistently improves the transferability of representations learned by supervised and unsupervised methods in various transfer learning scenarios. The code is available at https:/ gigttu..b.................. com/ haw.ohk/ooooooaaa AugSSe..
Introduction
Unsupervised representation learning has recently shown a remarkable success in various domains, e.g., computer vision [1, 3], natural language [4, 5], code [6], reinforcement learning [7, 8, 9], and graphs [10]. The representations pretrained with a large number of unlabeled data have achieved outstanding performance in various downstream tasks, by either training task-specific layers top of the model while freezing it or fine-tuning the entire model.
In the vision domain, the recent state-of-the the- methods [1, 2, 11. 12, 13] learn representations to be invariant to a pre-defined set of augmentations. The choice of the augmentations plays crucial role in representation learning [2, 14, 15, 16). A common choice is combination of random cropping, horizontal flipping, color jittering, grayscaling, and Gaussian blurring. With this choice, leamee representations invariant to color and positional information in images; in other words, the representations lose such information.
On the contrary, there have also been attempts to learn representations by designing pretext tasks that keep such information in augmentations, e.g. predicting positional relations between two patches of
WWok done while University of Michigan.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Augmentatiinn invariant Learning
Z2
lwg
Supervised I learning,
9
SimCLR, MoCo.
BYOL, SimSiam,
Ours
X
Waite W1
W2
Augmmeatttionn aware Learning
9
Ewos
Z1
Figure I: Illustration the proposed method, AugSelf, that learns augmentation-awan information by predicting the difference between two augmentation parameters w1 and w2. Here, x is an original image, v (x) is an augmented sample by an augmentation lw., f is a feature extractor such as ResNet [20], and is classifier for supervised learning or a projection MLP head for the recent unsupervised learning methods [1. 2, 12, 13].
an image [17], solving jigsaw puzzles [18], or predicting color information from a gray image [19]. These results show the importance of augmentation-specinn information for representation learning, and inspire us to explore the following research questions: when is learning invariance to given set of augmentations harmful representation learning? ? and, how to prevent the loss the recent unsupervised learning methods?
Contribution. We first found that learning representations with augmentation invvrriant objective might hurt its performance in downstream tasks that rely on information related to the augmentations. For example, learning invariance against strong color augmentations forces the representations tain less color information (see Figure 2a). Hence, it degrades the performance the representations in color-sensitive downstream tasks such as the Flowers classification task [21] (see Figure 2b).
To prevent this information loss and obtaín more generalizable representations, we propose an auxiliary self-supervised loss, coined AugSelf, that learns the difference of augmentation parameters between the two augmented samples (or views) as shown in Figure 1. For example, in the case of random cropping, AugSelf learns predict the difference of cropping positions of two randomly cropped views. We found that AugSelf encourages the self-supervised representation learning methods, such SimCLR [2] and SimSiam [13], preserve augmentation-awarc information (see Figure 2a) that could be useful for downstream tasks. Furthermore, AugSelf can easily be incorporated into the recent unsupervised representation learning methods [1, 2, 12, 13] with negligible additional training cost, which is for training an auxiliary prediction head á in Figure 1.
Somewhat interestingly, we also found that optimizing the auxiliary loss, AugSelf, can even improve the transferability of representations learned under the standard supervised representation learning scenarios [22]. This is because supervised learning also forces invariance, i..., assigns the same label, for all augmented samples (of the same instance), and AugSelf can help to keep sugmentation-awaan knowledge in the learned representations.
We demonstrate the effectiveness of AugSelf under extensive transfer learning experiments: AugSelf improves (a) two unsupervised representation learning methods, MoCo [1] and SimSiam [13], in 20 of 22 tested scenarios; (b) supervised pretraining in of 11 (see Table 1). Furthermore, we found that AugSelf is also effective under few-shot learning setups (see Table 2).
Remark that learning augmentation-invarte -nniaiiiattinnnnaaiiannn representations has been a common practice for both su- pervised and unsupervised representation learning frameworks, while the importance of augmentation- awareness is less emphasized. We hope that our work could inspire researchers rethink the under- under-explored aspect and provide new angle in representation learning.
2 Preliminaries: Augmentation-invarin representation learning
In this section, we review the recent unsupervised representation learning methods [1, 2, 11, 12, 13] that learn representations by optimizing augmentation-invee invariint objectives. Formally, let be an

---

70
ClassFicatio Accurasy
Accurary
65
26-
10-
1-
55
SimSiam+Auggel
602
SimSiam+AAugSe
SimSiam AugSel
Augmentatiio Strungth
Augm Strongth
Augmentation Strength
(a) Mutual information
(b) STLIO-Flowers
(C) STLI0-Food
Figure 2: (a) Changes of mutual information, i.e., (CCE(C; z). between color information C(x) and the representation f(x pretrained on STL10 [23] with varying the color jittering strength s. The pretrained representations are evaluated in rolor-sensitive benchmarks, (b) Flowers [21] and (c) Food [24], by the linear evaluation protocol [25].
image, tw be an augmentation function parameterized by augmentation parameter v= 1,2 (x) be the augmented sample (or view) of by fus and f be a CNN feature extractor, such as ResNet [20]. Generally speaking, the methods encourage the representations f(v1) and f(22) to be invariant to the two randomly augmented views V1 = twi (x) and V2 = two (x). i... (vi) Rs f(v2) for w/, W2 ~ S2 where Q is pre-defined augmentation parameter distribution. We now describe the recent methods one by one briefly. For simplicity, we here omit the projection MLP head g(-) which is widely used in the methods (see Figure 1).
Instance contrastive learning approaches [1, 2, 26] minimize the distance between an anchor f(tw, (x)) and its positive sample f(twa (x)). while maximizing the distance between the anchor [lan (x)) and its negative sample [tws (x')). Since contrastive learning performance depends on the number of negative samples, memory bank [26]. large batch [2], or a momentum network with a representation queue [1] has been utilized.
Clustering approaches [11, 27, 28] encourage two representations f(tn, (x)) and [(t,, to be assigned into the same cluster, other words, the distance between them will be minimized.
Negative-free methods [12, 13] learn to predict the representation f(v1) of a view V1 = twr (x) from another view V2 tw. (x). For example, SimSiam [13] minimizes [l((f(v)) sg(f(v1))
where h is an MLP and sg is the stop-gradient operation. In these methods, if h is optimal, then h(f(vz)) [f (f(1))) thus, the expectation of the objective can be rewritten Var (f (v))..
Therefore, the methods can be considered as learning invariance with respect to the augmentations.
Supervised learning approaches [22] also learn Iugmentation-invatie invariant representations. Since they often maximize exp(c,, (((()))) Ey exp(c, exp(c,, (t(x)) where Cy is the prototype vector the label y. (t(x)) is concentrated to Cy. i.c., C, = f(t,, (x)) ze f(tw,(x))
These approaches encourage representations f(x) to contain shared (i.e., nugmentation-innarian information between fu, (x) and tw, (x) and discard other information [15]. For example, if for changes color information, then to satisfy f(t,, (x)) f(t,, (x)) for any W1, w2 ~ Ω fxx will be learned to contain no (or less) color information. To verify this, we pretrain ResNet-18 [20] on STL10 [29] using SimSiam [13] with varying the strength of the color jittering augmentation. To measure the mutual information between representations and color information, we use the InfoNCE loss [30]. We here simply encode color information as RGB color histograms of an image. As shown in Figure 2a, using stronger color augmentations leads to relov-relevant information loss. In classification on Flowers [21] and Food [24], which is color-sensitive, the learned representations containing less color information result in lower performance as shown in Figure 2b and 2c, respectively. This observation emphasizes the importance of learning augmentation-awarc information in transfer learning scenarios.
Auxiliary augmentation-aware self-supervision
In this section, we introduce auxiliary uugmentation--wwen self-supervision, coined AugSelf, which encourages to preserve augmentation-awarc information for generalizable representation learning. To
3

---

Random cropping
Horizontal flipping
cu (Bometer Jemeterr W.)
whi Ily Hipped]
(0.4,0.3,06600.))
Color jitcoiing
Goussian blurring
Original image
waker = (Mrcigat Accutrast: Ause)
wy std, dev. of Gaussian leernel 1.0
(0.3, 1.0,08110)
Figure 3: Examples of the commonly-use used augmentations and their parameters "an
be specific, add auxiliary self-supervision loss, which learns predict the difference between augmentation parameters of two randomly augmented views, into existing augmentation-invant invarrian representation learning methods [1, 2, 11, 12, 13]. We first describe general form of our auxiliary loss, and then specific forms for various augmentations. For conciseness, let 0 be the collection of all parameters in the model.
Since an augmentation function fw is typically a composition of different types augmentations, the augmentation parameter can be written used in pretraining (e.g.. A {crop, flip)), and w an sugmentation-specinn crop decides how to crop image). Then, given randomly augmented views v1 lun (x) and
(w0 )aggEA where A is the set of augmentations
parameter (e.g..
V2 = tw (x), the AugSelf objective is as follows:
Caugseir W1, w2: = Σ BuggEAuggts
Lang 2u (fo warfe)
where AuugSait S is the set of augmentations for augmentation-ave aware learning, waift is the difference between two augmentation-speeinn parameters w and W Laug is an augmentation- specific loss, and dhu is a 3-layer MLP for warhr prediction. This design allows
incorporate AugSelf into the recent state-of-the-art unsupervised learning methods [1, 2, 11, 12, 13] with negligible additional training cost. For example, the objective of SimSiam [13] with AugSelf can be written as Crotal (x, w1.w2; 0) Lsinsian(x, W1, w2; 0) + À LAugseir (x, W1,Www 0), where hyperparameter for balancing losses. Remark that the total objective Liotal encourages the shared representation fxx) to learn both augmentation-invee invariann and augmentation-awant features. Hence, the learned representation fxx also can be useful in various downstream (e.g., augmentation- sensitive) tasks.?
augmentations in the recent unsupervised representation learning methods [1, 2, 11. 12, 13]: random cropping, random horizontal flipping, remark that different types of augmentations can be incorporated into AugSelf (see Section 4.2). In the following, we elaborate on the details of aug and Lugg for each augmentation. The examples of a are illustrated in Figure 3.
In this paper, mainly focus the commonly
color jittering, and Gaussian blurring; however,
vision tasks. A cropping parameter crop contains the center position and cropping size. We normalize the values by the height and width of the original image x, i.e., wrop E [0. 1 Then, we use & loss for Lexop and crop
Random cropping. The random cropping is the most popular augmentation
set Waitt = w 462
crop
Random horizontal flipping. A flipping parameter w 1pp (0, 1} indicates the image is horizontally
flipped or not. Since it is discrete, we use the binary cross-entropy loss for Leiip and set flip |wf w2 [
Color jittering. The color jittering augmentation adjusts brightness, contrast, saturation, and hue of input image in random order. For each adjustment, its intensity is uniformly sampled from a pre-defined interval. We normalize all intensitics into [0, 1), i.c. w color E [0, 1J4. Similarly to color cropping, use & loss for Leotor and set waife w wolor w2
We observe that our augmentation aware objective La does not interfere with learning the augmentation- invariant objective. e.g., Leiasu. This allows f fxx) to learn augmentation-awee aware information with negligible loss of augmentation invariant information. A detailed discussion is provided in the supplementary material.

---

Table I: Linear evaluation accuracy (%) ResNet-50 [20] and ResNet- pretrained on ImageNet110 [31. 32] and STL10 [23], respectively. Bold entries are the best of each baseline.
Method
CIFARIO
CIFARIOO
Food
MIT67
Flowers
Caltech101
Cars
Aircraft
DTD
SUN397
ImageNer ResNer-50
SimSiam
86.89
66.33
68
65.75
74.69
83.06
84.13
48.20
48.63
65.11
50.60
AugSelF
88.80
70.27
65.63
67.76
76.34
90.70
85.30
47.52
49.76
67.29
52.28
MoCo v2
84.60
61.60
59.37
61,64
70.08
82.43
77.25
33.86
41.21
64.47
46.50
AugSell (ours)
85.26
63.90
63.36
78.93
37.35
48.52
Supervised
86.16
62.70
53.89
52.91
73.50
76.09
77.53
30.61
36.78
61.91
40.59
+ AugSell' (ours)
$6.06
63.77
55.84
54.63
74.81
78.22
77.47
31.26
38.02
62.07
41.49
S7L.-O-preraiieed ReaNez-18
SimSiam
$2.35
54.90
33.99
39.
44.90
59.
66.33
16.85
26.06
42.57
29.05
+ AugSelf (ours)
82.76
58.65
41.58
45.67
48.42
72.18
72.75
21.17
33.17
47.02
34.14
MoCo v2
81.18
53.75
33.69
39.01
34
61.01
64.15
16.09
26.63
41.20
28.50
+ AugSelf (ours)
82.45
$7.17
36.91
41.67
43.80
66.96
66.02
17.53
28.02
45.21
30.93
Gaussian blurring. This blurring operation is widely used in unsupervised representation learning. The Gaussian filter is constructed by a single parameter, standard deviation bbur We also normalize the parameter into [0, 1]. Then, we use Eg loss for Loiur and set blur = w ww blur
4 Experiments
Setup. We pretrain the standard ResNet-18 18 [20] and ResNet-50 on STL10 [23] and ImageNet1 [31, 32], respectively. We use two recent unsupervised representation learning methods baselines for pretraining: contrastive method, MoCo v2 [1, 14], and non-contrastive method, SimSiam [13]. For STL10 and ImageNet100, 100, we pretrain networks for 200 and 500 epochs with a batch size of 256, respectively. For supervised pretraining, we pretrain ResNet-50 for 100 epochs with a batch size of 128 on ImageNer100." For augmentations, we use random cropping, flipping. color jittering, grayscaling, and Gaussian blurring following Chen and He [13]. In this section, our AugSelf predicts random cropping and color jittering parameters, i.c., Augseer = { (crop, color}, unless otherwise stated. We set X 1.0 for STL10 and À 0.5 for ImageNet100. The other details and the sensitivity analysis to the hyperparameter À are provided in the supplementary material. For ablation study (Section 4.2), only use STI.10-pretrained models.
4.1 Main results
Linear evaluation in various downstream tasks. We evaluate the pretrained networks in down-
stream classification tasks on 11 datasets: CIFAR10/100 [29], Food [24], MIT67 [36], Pets [37], Flowers [21]. Caltech101 [38], Cars [39], Aircraft [40], DTD [41], and SUN397 [42]. They contain
roughly 1k~70k training images. We follow the linear evaluation protocol [25]. The detailed in-
formation of datasets and experimental settings is described in the supplementary material. Table shows the transfer learning results in the various downstream tasks. Our AugSelf consistently improves (a) the recent unsupervised representation learning methods, SimSiam [13] and MoCo [13], in 10 out of 11 downstream tasks; and (b) supervised pretraining in 9 out of 11 downstream tasks. These consistent improvements imply that our method encourages to learn more generalizable representations.
Few-shot classification. We also evaluate the pretrained networks on various few-shot learning benchmarks: FC100 [33], UCttech-UUSDD Birds (CUB200) [43], and Plant Disease [35]. Note that CUB200 and Plant Disease benchmarks require low-level features such as color information of birds and leaves, respectively, to detect their fine- rree-grained labels. They are widely used cross-domain few-shot settings [44, 45]. For few-shot learning, we perform logistic regression using the frozen representations fxx without toniettning.... Table shows the few-shot learning performance of 5-way 1-shot and 5-way 5-shot tasks. As shown in the table, our AugSelf improves the performance of SimSiam [13] and MoCo [I] in all cases with a large margin. For example, for plant discase detection
'ImageNet100 is 100-category subset of ImageNet [31]. use same split following Tian et al. [32]. 'We do not experiment supervised pretraining on STL10, as it has only 5k labeled training samples, which is enough for pretraining good representation.
5

---

Table 2: Few-shot classification accuracy (%) with 95% confidence intervals averaged over 2000 episodes on FC100 [33], CUB200 [34], and Plant Disease [35]. (N, K) denotes N N-way K Khohot tasks. Bold entries are the best of each group.
FC100
CUB200
Plant Discase
Method
(5. 1)
(5
(5. 1)
(5.
(5, I)
(5, 5)
ImageNer/00-preeteeee ResNet-50
SimSiam
19.0.6
50.36
45. 45.5610.47
62.48.0.48
75.72.0.46
39,9440311
AugSelf (ours)
39.37a0.40
55.27
48.0820.47
66.2720.6
77.9320.46
91.52=0.29
MoCo v2
31.67+0.33
43.88-038
41.67-044
56.92-0.7
65.73+0.49
84.98-0.6
AugSelf (ours)
35.0210.36
48.77s0.39
44.17
57.35
71.80x0.47
8
Supervised
33. 15
46.59=a37
46.57s0.4
63.69s0.4
68. 68.95+2.47
88.77=0.0
AugSelf (ours)
34.70a0.35
48.89=0.8
47.58
65.31=045
70.82+a.46
89.77=0.29
STLIO-pretrained ResNet-18
SimSiam
36. 7240.5
21.49
37.97a0.3
50.6120:4
13.13
75,98s0.0
AugSelf (ours)
40.68e0.39
56,26=0.8
41.60em.42
56.33s044
62.85en.49
81.14-0.77
MoCo v2
35.69 65.6990.34 34
49.26-036
37.62-0.42
50.71-044
57.87-0.48
25.98-0.0
AugSelf (ours)
39.6620.39
55.5820.9
38.3320.41
51.932044
60.782n.50
78.76s0.38
Table 3: Linear evaluation accuracy (%) under the same setup following Xiao al. [16]. The augmentations in the brackets of LooC [16] indicate which augmentatiin-aware information is learned. N is the number of required augmented samples for each instance, that reflects the effective training batch size." indicates that the numbers are reported in [16]. The numbers in the brackets show the accuracy gains compared each baseline.
Method
N
ImageNet100
CUB200
Flowers (5-shot)
Flowers (10-shot)
MoCo* [1]
2
81.0
36.7
67.9-0.
77.3=0.1
LooC' [16] (color)
81.1 (+0.1)
40.1 (+3.4)
68.220. (+0.3)
77.620.1 (+0.3)
LooC* [16] (rotation)
80.2 (-0.8)
38.8 (+2.1)
70.1-04 (+2.2)
79.3+0.1 (+2.0)
LooC* [16] (color, rotation)
79.2 (-1.8)
39.6 (+2.9)
70.920.3 (+3.0)
80.8202 (+3.5)
MoCo [1]
2
81.0
32.2
78.5.0.
81.2s0.3
MoCo [1] + AugSelf (ours)
82.4 (+1.4)
37.0 (+4.8)
81.7:00 (+3.2)
84.5.0.2 (43.3)
SimSiam [13]
2
81.6
38.4
83.6=0.3
85.9=0.2
SimSiam [13] AugSelf (ours)
82.6 (+1.0)
45.3 (+6.9)
86.4.0.2 (+2.8)
88.3.0.1 (+2.4)
[35], we obtain up 6.07% accuracy gain in 5-way 1-shot tasks. These results show that our method is also effective in such transfer learning scenarios.
Comparison with LooC. Recently, Xiao et al. [16] propose LooC that learns augmentation-awan iirteentatio-speciffe contrastive learning objectives. Table shows
representations via multiple head-to-head comparisons under the same evaluation setup following Xiao et al. [16].5 As shown in the table, our AugSelf has two advantages over LooC: (a) AugSelf requires the same number of augmented samples compared to the baseline unsupervised representation learning methods while LooC' requires more, such that AugSelf does not increase the computational cost; (b) AugSelf can be incorporated with non-contrastive methods e.g., SimSiam [13], and SimSiam with AugSelf outperforms LooC in all cases.
Object localization. We also evaluate representations in an object localization task (i.e., bounding box prediction) that requires positional information. We experiment on CUB200 [46] and solve linear regression using representations pretrained by SimSiam [13] without or with our method. Table 4 reports fg errors of bounding box predictions and Figure 4 shows the examples of the predictions. These results demonstrate that AugSelf is capable of learning positional information.
Retrieval. Figure shows the retrieval results using pretrained models. For this experiment, we use the Flowers [21] and Cars [39] datasets and find top-4 nearest neighbors based on the cosine
SSinee LooC's code is currently not publicly available, we reproduced the MoCo baseline as reported in the sixth row in Table 3: we obtained the same ImageNet100 result, but different ones for CUB200 and Flowers.

---

w/ w/SeeiAuu wio SelfAug
Method
Error
SimSiam
0.00462
+ AugSelf (ours)
0.00335
MoCo
0.00487
+ AugSelf (ours)
0.00429
Supervised
0.00520
+ AugSeelf (ours)
0.00473
Table 4: & errors of bounding box predictions on CUB200.
Figure 4: Examples of bounding box predictions CUB200. Blue and red boxes are ground-truth and model prediction, respectively.
Query
Nearest neighbors
Query
Nearest neighbors
(a) SimSiam
(b) SimSiam + AugSelf (ours)
Figure 5: Top-4 nearest neighbors based on the cosine similarity using representations [(x) learned by (a) SimSiam [13] or (b) SimSiam with AugSelf (ours).
similarity between representations fx) where f is the pretrained ResNet-50 ImageNet100. As shown in the figure, the representations learned by AugSelf are more color-sensitive.
4.2 Ablation study
Effect of augmentation prediction tasks. We first evaluate the proposed augmentation prediction tasks one by one without incorporating evariance-learning methods. More specifically, we pretrain fo using only Lang for each aug E {crop, flip, color, blur). Remark that training objectives are different but we use the same set of augmentations. Table 5 shows the transfer learning results in various downstream tasks. We observe that solving horizontal flipping and Gaussian blurring prediction tasks results worse or similar performance to a random initialized network in various downstream tasks, i.c. the augmentations do not contain task-relevant information. However, solving random cropping and color jittering prediction tasks significantly outperforms the random initialization in all downstream tasks. Furthermore, surprisingly, the color jittering prediction task achieves competitive performance in the Flowers [21] dataset compared to a recent state-of-the-ar method, SimSiam [13]. These results show that augmentation-awee information task-relevant and learning such information could be important in downstream tasks.
Based on the above observations, we incorporate random cropping and color jittering prediction tasks into SimSiam [13] when pretraining. More specifically, we optimize Lsinssan + Narop Acolorfcolor where Aerop: Acolor E {0, 1). The transfer learning results are reported in Table 6. As shown the table, each sulf-supervision task improves SimSiam consistently (and often significantly) in various downstream tasks. For example, the color jittering prediction task improves SimSiam by 6.33% and 11.89% Food [24] and Flowers [21] benchmarks, respectively. When incorporating both tasks simultancously. Furthermore, as shown in Figure 2, our AugSelf preserves augmentation-ave -wwaee information as much possible; hence gain is consistent regardless of the strength of color jittering augmentation.
achieve further improvements in almost all the downstream tasks.
Different augmentations. We confirm that our method can allow to use other strong augmentations: rotation, which rotates an image by 0%, 90", 180 270" degrees randomly; and solarization, which inverts each pixel value when the value larger than a randomly sampled threshold. Based on the default augmentation setting, i.c. Auugser {crop, color), we additionally apply each
7

---

Table 5: Linear evaluation accuracy (%) of ResNet- 18 [20] pretrained by each augmentation prediction task without other methods such as SimSiam [13]. We report SimSiam [13] results as reference. Bold entries are larger than the random initialization.
Pretraining objective
STL.10
CIFARIO
CIFARIO0
Food
MIT67
Pets
Flowers
Random Init
42.72
47.45
23.73
11.54
12.29
12.94
26.06
Corop
68.28
70.78
43.44
22.26
26.17
27.68
38.21
Lrip
46.45
53.80
24.89
9.69
11.99
10.71
13.04
Lcoioz
61.14
63.39
40.38
28.02
25.35
24.49
54.42
Coler
48.26
46.60
20.44
8.73
11.87
13.07
17.20
SimSiam [13]
85.19
82.35
54.90
33.99
39.15
44.90
59.19
Table 6: Linear evaluation accuracy (%) of ResNet-18 [20] pretrained by SimSiam [13] with various
combinations of our augmentation prediction tasks. Bold entries are the best of each task.
Aaugseit
STL.10
CIFARIO
CIFARTO0
Food
MIT67
Pets
Flowers
0
85.19
82.35
54.90
33.99
39.15
44.90
59.19
{ (crop)
85.98
82.82
55.78
35.68
43.21
47.10
62.05
[color)
85.55
82.90
58.11
40.32
43.56
47.85
71.08
(crop, color)
85.70
82.76
58.65
41.58
45.67
48.42
72.18
augmentation with probability of 0.5. We also evaluate the effectiveness of augmentation prediction tasks for rotation and solarization. Note that we formulate the rotation prediction as 4-way classification task (i.e., waife E {0. 1, 2, 3)) and the solarization prediction as regression task (i.e., well [-1,1], As shown in Table 7, we obtain consistent gains across various downstream tasks even if stronger augmentations are applied. Furthermore, in the case of rotation, we observe that our augmentation prediction task tries to prevent the performance degradation from learning invariance to rotations. For example, in CIFAR 100 [29]. the baseline loses 4.66% accuracy (54.90%-500 24%) when using rotations, but ours does only 0.37% (58.65 5-.565 AugSelf is less sensitive to the choice of augmentations. We believe that this robustness would be useful future research on representation learning with strong augmentations.
These results show that our
Solving geometric and color-related pretext tasks. To val- idate that our AugSelf is capable of learning augmentation- aware information, we try to solve two pretext tasks requiring the information: 4-way rotation (09, 90" 180" 270") and 6-way color channel permutation (RGB, RBG, BGR) classification tasks. We note that the baseline (SimSiam) and our method SimSiamtAugSSlf) do not observe rotated or color-permuted samples in the pretraining phase. We train linear classifier on top of pretrained representation without finetuning for each task. As reported in Table 8, our AugSelf solves the pretext tasks well even without their prior knowledge in pretraining; these results validate that our method learns augmentation-avee awware information.
Table 8: Linear evaluation accuracy in augmentation-aware pretext tasks.
Method
Rotation
Color perm
SimSiam + +AugSelf
59.11
24.66
64.61
60.49
Compatibility with other methods. While we mainly focus on SimSiam [13] and MoCo [1] in the previous section, our AugSelf can be incorporated into other unsupervised learning methods, SimCLR [2], BYOL [12], and SwAV [11]. Table 9 shows the consistent and significant gains by AugSelf across all methods and downstream tasks.
5 Related work
Self-supervised pretext tasks. For visual representation learning without labels, various pretext tasks have been proposed in literature [17, 18, 19, 47, 48, 49, 50, 51] by constructing self supervsspni from an image. For example, Doersch et al. [17], Noroozi and Favaro [18] split the original image into 3 x 3 patches and then learn visual representations by predicting relations between the patch locations. Instead, Zhang et al. [19]. Larsson et al. [48] construct color prediction tasks by converting colorful images to gray ones. Zhang et al. [51] propose similar task requiring to predict one subset of channels (c.g., depth) from another (c.g.. RGB values). Mcanwhile, Gidaris et al. [47], Qi et al.
8

---

Table 7: Transfer learning accuracy (%) of ResNet- [20] pretrained by SimSiam [13] with without our AugSelf using strong augmentations. C, J. R and S denote cropping, color jittering, rotation and solarization prediction tasks, respectively. Bold entries are the best of each augmentation.
Strong Aug.
AnugBeit
STLIO
CIFARIO
CIFARI00
Food
MIT67
Pets
Flowers
85.19
82,35
54.90
33.99
39.15
44,90
59. 19
None
85.70
82.76
58.65
41.58
45.67
48.42
72.18
80.11
77.78
50.24
36.40
36.39
41.43
61.77
Rotation
[c
81.85
79.93
57.27
43.04
41.32
47.30
72.52
J,
82.67
80.71
58.28
43.28
44.48
46.65
72.94
86.32
81.08
52.50
32.59
41.29
44.76
58.79
Solarization
{c, J}
86.03
82.64
57.94
40.29
46.67
48.81
71.43
(c, J,S}
85.91
82.63
58.18
40.17
45.57
49.02
71.43
Table 9: Transfer learning accuracy (%) of various unsupervised learning frameworks with and without our AugSelf framework. Bold entries indicates the best for each baseline method.
Method
AugSelf' (ours)
STL10
CIFARIO
CIFAR100
Food
MIT67
Pets
Flowers
84.87
78.93
48.94
31.97
36.82
43.18
56.20
SimCLR [2]
84.99
80,92
53.64
36.21
40.62
46.51
64.31
86.73
82,66
55.94
37.30
42.78
50.21
66.89
BYOL [12]
86.79
83.60
59.66
42.89
46.17
52.45
74.07
82.21
81.60
52.00
29.78
36.69
37.68
53.01
SwAV [11]
82.57
82.00
55.10
33.16
39.13
40.74
61.69
[49], Zhang et al. [50] show that solving affine transformation (e.g., rotation) prediction tasks can learn high-level representations. These approaches often require specific preprocessing procedures (c.g.. 3 x 3 patches [17, 18], or specific affine transformations [49, 50]). In contrast, our AugSelf can be working with common augmentations such random cropping and color jittering. This advantage allows us to incorporate AugSelf to the recent state-of-the-art frameworks like SimCLR [2] while not increasing the computational cost. Furthermore, construct AugSelf, but also finding on the importance of learning augmentation-aware representations together with the existing augmentation-innarian approaches.
emphasize that contribution is not only to
Augmentations for unsupervised representation learning. Chen et [2]. Tian et al. [15] found that the choice of augmentations plays critical role in contrastive learning. Based this finding, many unsupervised learning methods [1. 2, 11, 12, 13] have used similar augmentations (c.g.. cropping and color jittering) and then achieved outstanding performance in ImageNet [31]. Tian et al. [15] discuss a similar observation to ours that the optimal choice of augmentations task-dependent, but they focus on finding the optimal choice for specific downstream task. However, in the pretraining phase, prior knowledge of downstream tasks could be not available. In this case, we need to preserve augmentatiion-aware information in representations much as possible for unknown downstream tasks. Recently, Xiao et al. [16] propose a contrastive method for learning augmentation-awwe representations. This method requires additional augmented samples for each augmentation; hence, the training cost increases with respect to the number of augmentations. Furthermore, the method is specialized to contrastive learning only, and is not attractive be used for non-contrastive methods like BYOL [12]. In contrast, our AugSelf does not suffer from these issues as shown in Table and 9,
Discussion and conclusion
To improve the transferability of representations, we propose AugSelf, an auxiliary augmentation- aware self-supervision method that encourages representations contain augmentation-awwe aaware infor- mation that could be useful in downstream tasks. Our idea is to learn to predict the difference between augmentation parameters of two randomly augmented views. Through extensive experiments, we demonstrate the effectiveness of AugSelf in various transfer learning scenarios. We believe that our work would guide many research directions for unsupervised representation learning and transfer learning.

---

Limitations. Even though our method provides large gains when we apply it to popular data augmentations like random cropping, it might not be applicable to some specific data augmentation. where it is non-trivial to parameterize augmentation such as GAN-based one [52]. Hence, an interesting future direction would be to develop an augmentation-agnosia way that does not require to explicitly design augmeniation-specifi self-supervision. Even for this direction, we think that our idea of learning the relation between two augmented samples could be used, c.g., constructing contrastive pairs between the relations.
Negative societal impacts. Self-supervised training typically requires huge training cost (e.g.. training MoCo v2 [14] for 1000 epochs requires 11 days on 8 V100 GPUs), large network capacity (e.g., GPT3 [5] requires 175 billion parameters), therefore it raises environmental issues such as carbon generation [53]. Hence. efficient training methods [54] or distilling knowledge to smaller network [55] would be required to ameliorate such environmental problems.
Acknowledgments and disclosure of funding
This work was mainly supported by Samsung Electronics Co., Ltd (10201211-08107-01 and partly supported by Institute Information & Communications Technology Planning & Evaluation (ITPP) grant funded by the Korea government (MSIT) (No.2019-0-00075, Artificial Intelligence Graduate School Program (KAIST)).
References
[1] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2020.
[2] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of International Conference on Machine Learning, pages 1.597-1007. PMLR, 2020.
[3] Tengda Han, Weidi Xie, and Andrew Zisserman. Self-supervised co-training for video represen- tation learning. In Advances in Neural Information Processing Systems, 2020.
[4] Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:/8/00 1810.04805. 2018.
[5] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
[6] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng. Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. CodeBERT: A pre-trained model for programming and natural languages. In Proceedings of the Conference on Empirical Methods in Natural Language Processing:
Findings, pages 1536-1547, 2020.
[7] Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Marc-Alexandre Côté, and R Devon Hjelm. Unsupervised state representation learning in atari. In Advances in Neural Information Processing Systems, pages 8769-8788 2019.
[8] Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In Proceedings of International Conference on Machine Learning, 2021.
[9] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: Contrastive unsupervised repre- sentations for reinforcement learning. In Proceedings of International Conference on Machine Learning, pages (659,, 2020.
[10] Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In Proceedings of International Conference on Machine Learning, pages 41 4116-4126, 2020.
10

---

[11] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems, 2020.
[12] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap Your Own Latent: A new approach to self-supervised learning, arXiv preprint arXiv:2006.07733, 2020.
[13] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. arXiv preprint arXiv:2011.10566 2020.
[14] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020.
[15] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In Advances Neural Information Processing Systems, pages 6827-6839, 2020.
[16] Tete Xiao, Xiaolong Wang, Alexei A Efros, and Trevor Darrell. What should not be contrastive contrastive learning. In International Conference on Learning Representations, 2021.
[17] Carl Doersch, Abhinav Gupta, and Alexei A Efros, Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference Computer Vision, pages 1422-1430, 2015.
[18] Mehdi Noroozi and Paolo Favaro, Unsupervised learning of visual representations by solving jigsaw puzzles. In Proceedings of the European Conference Computer Vision, pages 69 84. Springer, 2016.
[19] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In Proceedings of the European Conference on Computer Vision, pages 169.666. Springer, 2016.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016
[21] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing, pages 722-729. IEEF, 2008
[22] Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Can features off-the-shelf: an astounding baseline for recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 806-813, 2014.
[23] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In Proceedings of the International Conference on Artificial Intelligence and Statistics, pages 215-222 2233. JMLR Workshop and Conference Proceedings, 2011.
[24] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 - mining discriminative components with random forests. In Proceedings of the European Conference Computer Vision, 2014.
[25] Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2661-2671, 2019.
[26] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parane parameemitiiic instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages
2018.
[27] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European Conference on Computer Vision, pages 132-149, 2018.
11

---

[28] Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. In International Conference on Learning Representations. 2020.
[29] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[30] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 11X7:1887.03748 2018
[31] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-255) 2015.
[32] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv:1906.05884 2019.
[33] Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems, pages 721-731, 2018.
[34] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-201 1-001, California Institute of Technology, 2011.
[35] Sharada P Mohanty, David P Hughes, and Marcel Salathé. Using deep learning for image-based plant disease detection. Frontiers in plant science, 7:1419, 2016.
[36] Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 413-420. IEEE, 2009,
[37] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3498 3505. IEEE, 2012.
[38] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In Conference on Computer Vision and Pattern Recognition Workshop, pages 178- 178. IEEE, 2004.
[39] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13) Sydney, Australia, 2013.
[40] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- 2013.
grained visual classification of aircraft. arXiv preprint
[41] Mircea Cimpoi, Subhransu Maji, Jasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3606-3613, 3613,611222 2014.
[42] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to 70o. In IEEE Computer Society Confer- ence on Computer Vision and Pattern Recognition, pages 3485-3492. IEEE, 2010.
[43] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V L.e. RandAugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 702-703, 2020.
[44] Wei- Yu Chen, Yen- Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia- Huang. A closer look at few-shot classification. In International Conference on Learning Representations, 2019.
[45] Yunhui Guo, Noel C Codella, Leonid Karlinsky, James V Codella, John R Smith, Kate Saenko, Tajana Rosing, and Rogerio Feris. A broader study of crmssn few-shot learning. In Proceedings of the European Conference Computer Vision, 2020.
12

---

[46] Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie. Building a bird recognition upp and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In Procedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 595-604, 2015.
[47] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference Learning Representations. 2018.
[48] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for automatic colorization. In Proceedings of the European Conference on Computer Vision, pages 577-593, Springer, 2016.
[49] Guo-Jun Qi, Liheng Zhang, Chang Wen Chen, and Qi Tian. AVT: Unsupervised learning of transformation equivariant representations by autoencoding variational transformations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8130-8139, 2019.
[50] Liheng Zhang, Guo-Jun Qi, Liqiang Wang, and Jiebo Luo. AET vs. AED: Unsupervised representation learning by auto-encoding transformations rather than data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2547-2555, 2019.
[51] Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1058-1067, 2017.
[52] Veit Sandfort, Ke Yan, Perry Pickhardt, and Ronald M Summers. Data augmentation using generative adversarial networks (cyclegan) to improve generalizability in ct segmentation tasks. Scientific reports, 91): 2019.
[53] Roy Schwartz, Jesse Dodge, Noah Smith, and Oren Etzioni. Green ai. arXiv preprint 1907 2019.
[54] Guangrun Wang, Keze Wang, Guangcong Wang, Phillip HS Torr, and Liang Lin. Solving inefficiency self-supervised representation learning. arXiv preprint (XXv:2104.08760, 2021.
[55] Zhiyuan Fang, Jianfeng Wang. Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. SEED: Self-supervised distillation for visual representation. In International Conference on Learning Representations, 2021.
13


Reviews:
[{"review_id": "ksjoqTwja3c", "reviewer": "Reviewer_QC3H", "summary": "This paper proposes a simple regularization for self-supervised/supervised representation learning, which aims to benefit downstream tasks. Main idea is to make the network to be variant to different augmentation. Extensive experimental results are reported to show the effectiveness.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The idea is simple and seems to provide good performance gains. However, I am curious what's the main learning objective of the main paper. The main loss tries to learn augmentation-invariant representation while the AugSelf tries to learn augmentation-variant representation. These two losses are contradicting. What do we want the network to learn? To be invariant or variant to augmentation. If some augmentations are bad for downstream tasks, such as color jittering and Gaussian blurring, can we just remove these negative augmentation in the main loss? Maybe just removing these augmentation is sufficient to bring large performance gains. Adding the regularization will make things complex. So I suggest to add ablation studies on this: main loss without color/blurring, and more importantly provide more insights about why your regularized method could work.\n\n2. How can you determine λ for each augmentation? Can you provide the sensitivity analysis to these hyperparamters.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "jyMoPt8LrPl", "reviewer": "Reviewer_qDmw", "summary": "This paper presents AugSelf, an auxiliary self-supervised task in contrastive learning to improves the transferability of learned representations. The task predicts the difference of augmentation parameters between two randomly augmented samples, and therefore forces the representation to preserve the information which could be potentially discarded by augmentations. Extensive experiments demonstrate the effectiveness of the proposed method.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The observation that “learning representations with an augmentation-invariant objective might hurt its performance in downstream tasks” and the idea of preserving the information that is removed by the augmentations to improve transferability is already presented in [1]. However, the idea of adding an auxiliary task to preserve the information is novel and interesting.\n\nQuality: \nThe proposed method of predicting the augmentations to preserve information is novel and interesting. The method is simple and can be easily adapted to different contrastive learning framework and is applicable to a wide range of common quantifiable augmentations. The experiments and ablation studies are also extensive.\n\nMinor points: \n1.\tthe observation that “learning representations with an augmentation-invariant objective might hurt its performance in downstream tasks” should not be claimed as part of the contribution since [1] already observed it.\n2.\tThe checklist is not well formatted. The instructions are not deleted and there are still placeholders. Also, the error bars and standard deviation is missing, which could be >0.5 as shown in [1].\n\nClarity: The paper is well written and easy to follow.\n\nSignificance: the paper proposes a simple method to preserve information that could be potentially removed by the strong augmentations of contrastive learning. Although the idea of preserving the information removed by augmentations to improve transferability is not new, the method is simple and effective to preserve the information and its performance on multiple datasets are consistently good.\n\n[1] Xiao, Tete, et al. \"What should not be contrastive in contrastive learning.\" arXiv preprint arXiv:2008.05659 (2020).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "VDPjQl5TiB", "reviewer": "Reviewer_wbQH", "summary": "This paper proposes a self-supervision method with learning the difference of augmentation parameters as the pretext task to improve the transferability of learned representations. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n+ The paper proposes a new self-supervision method to improve the transferability of learned representations. The method is simple and clear. \n+ The proposed method consistently improves the unsupervised learning methods. \n\nWeakness:\n- There are no other sota representation learning methods, especially other self-supervision related methods with different design of pretext tasks for comparison. \n\n\nQuestion:\nDoes the augmentation help with the robustness of the learned representation under perturbations? ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don't see any potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours."}, "extra_text": {}}, {"review_id": "Au0LRWySR49", "reviewer": "Reviewer_fxVi", "summary": "This paper aims at improving the transferability of representations, especially self-supervised pre-trained representations. Motivated by the observation that augmentation-invariance may be harmful to downstream tasks, e.g., location- or color-sensitive ones, they proposed an auxiliary augmentation-aware self-supervision loss named AugSelf. AugSelf learns the difference of augmentation parameters (e.g., cropping positions, color adjustment intensities) between two randomly augmented samples. Experiments show that AugSelf helps for transfer learning tasks with two pre-trained models and a supervised learning model, as well as few-shot learning setups.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strongness:\n+ This paper is well-written and easy to follow, making it enjoyable to read.\n+ The proposed method is very simple, even somewhat intuitive, by introducing an auxiliary augmentation-aware self-supervision loss.\n+ Experiments on several benchmarks demonstrate the effectiveness of the proposed method.\n\nWeakness:\n- My major concern is about the claim on improving the transferability of representations. It is reasonable to achieve better performance by introducing an auxiliary augmentation-aware self-supervision loss for location- or color-sensitive tasks, but why will AugSelf work for those tasks that are not augmentation-sensitive? If not, the proposed method is not a general method to improve transferability, instead, it captures dataset-specific information.\n- Another major concern is the tradeoff between preserving augmentation-aware information in learned representations with the loss of augmentation-invariant information. If the model exhausts itself to capture augmentation-aware information, will it be harmful to augmentation-invariant information? Meanwhile, how to decide the value of the hyperparameter of $\\lambda$ between these two losses.\n- There are two protocols to evaluate the transferability of representations: linear evaluation and fine-tuning. It seems that many fine-tuning baselines focusing on improving transferability are missing, e.g. L2SP [1], DELTA [2], BSS [3], DELTA [4].\n\n[1] Li, X., Grandvalet, Y., and Davoine, F. Explicit inductive bias for transfer learning with convolutional networks. In ICML, 2018.\n[2] Li, X., Xiong, H., Wang, H., Rao, Y., Liu, L., and Huan, J. Delta: Deep learning transfer using feature map with attention for convolutional networks. In ICLR, 2019.\n[3] Chen, X., Wang, S., Fu, B., Long, M., and Wang, J. Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning. In NeurIPS, 2019.\n[4] You, K., Kou, Z., Long, M., and Wang, J. Co-tuning for transfer learning. In NeurIPS, 2020.\n\n- The question \"when is learning invariance to a given set of augmentations harmful to representation learning?\" mentioned in the Introduction section is unsolved. \n- Actually, the augmentation-aware loss is defined on the same specific augmentation, instead of the whole augmentation space. Will it be possible that the proposed method be extended into different types of augmentations? If so, which kind of different information will be learned?\n- Why the improvement of the proposed methods on some datasets such as STL10 in Table 9 is minor? \n-Typos: Line 83 \"In these methods, if h optimal, then...\"\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims to improve the transferability of representations via augmentation-aware self-supervision, by preserving both augmentation-invariant and augmentation-aware information. The main finding is that augmentation-variant information may be relevant to downstream tasks, which is interesting and inspiring for the self-supervised learning community. The rebuttal is informative and relevant, and after the discussion all reviewers unanimously recommended acceptance. Reviewers also pointed out some important directions for improvement, such as a larger set of augmentations for pre-training, more sophisticated fine-tuning methods, and a clear elaboration on the relationship between augmentation invariance and awareness. Authors shall guarantee to include the rebuttal material into their future draft.
Decision: Accept (Poster)