Residual Relaxation for Multi-view Representation Learning
Authors: Yifei Wang
Zhengyang Geng
Feng Jiang
Chuming Li
Yisen Wang
Jiansheng Yang
Zhouchen Lin
Venue: NeurIPS 2021 Poster (2021)
Keywords: Self-supervised Learning
Representation Learning
Multi-view Learning

Abstract: Multi-view methods learn representations by aligning multiple views of the same image and their performance largely depends on the choice of data augmentation. In this paper, we notice that some other useful augmentations, such as image rotation, are harmful for multi-view methods because they cause a semantic shift that is too large to be aligned well. This observation motivates us to relax the exact alignment objective to better cultivate stronger augmentations. Taking image rotation as a case study, we develop a generic approach, Pretext-aware Residual Relaxation (Prelax), that relaxes the exact alignment by allowing an adaptive residual vector between different views and encoding the semantic shift through pretext-aware learning. Extensive experiments on different backbones show that our method can not only improve multi-view methods with existing augmentations, but also benefit from stronger image augmentations like rotation.

Residual Relaxation for Multi-view Representation Learning
Yifei Wang's Zhengyang Geng² Feng Jiang"
Chuming Li
Yisen Wang2.4 Jiansheng Yang' Zhouchen Lin
School of Mathematical Sciences, Peking University, China
2 Key Lab. of Machine Perception, School Artificial Intelligence, Peking University. Beijing, China School of Engineering. The University of Sydney, Australia Institute for Artificial Intelligence, Peking University, Beijing, China Pazhou Lab, Guangzhou, China
Abstract
Multi-view methods learn representations by aligning multiple views of the same image and their performance largely depends the choice of data augmentation. In this paper, we notice that some other useful augmentations, such as image ro- tation, harmful for multi-view methods because they cause semantic shift that is too large to be aligned well. This observation motivates us to relax the ex- act alignment objective better cultivate stronger augmentations. Taking image rotation as a case study, we develop a generic approach, Pretext-aware Resid- ual Relaxation (Prelax), that relaxes the exact alignment by allowing adaptive residual vector between different views and encoding the semantic shift through pretext-aware learning. Extensive experiments our method can not only improve multi-view methods with existing augmenta- tions, but also benefit from stronger image augmentations like rotation.
different backbones show that
Introduction
Without access to labels, self-supervised learning relies on surrogate objectives to extract meaningful representations from unlabeled data, and the chosen surrogate objectives largely determine the qual- ity and property of the learned representations [24, 19]. Recently, multi-view methods have become dominant approach for self-supervised representation learning that achieves impressive down- stream performance, and many modern variants have been proposed [22, 14, 1, 23, 2, 1,, 4, 11, 5]. multi-view methods can be abstracted and summarized the following pipeline: for each input x. we apply several (typically two) random augmentations to it, and learn to align these different "views" (Xi,X2,.... x2,...) .) of by minimizing their distance in the representation space.
Nevertheless,
In multi-view methods, the pretext, i.e. image augmentation, has a large effect on the final perfor- mance. Typical choices include image re-scaling, cropping, color jitters, etc [2]. However, find that some augmentations, for example, image rotation, is seldom utilized in state-of-the-arr multi- view methods. Among these augmentations, Figure la shows that rotation causes severe accuracy drop in a standard supervised model. Actually, image rotation is a stronger augmentation that largely affects the image semantics, and result, enforcing exact alignment of two different rotation an- gles could degrade the representation ability in existing multi-view methods. Nevertheless, it does mean that strong augmentations cannot provide useful semanties for representation learning. In fact, rotation is known as an effective signal for predictive learning [10, 30, 21]. Differently. predictive methods learn representations by predicting the pretext (e.g., rotation angle) from the cor-
"Coresponding author: Yisen Wang (yisen.wang @ pku.edu.en).
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

1el A20
exact alignment
residual alignment
Rew
HortornaaFPp
Goyyican
Golo./ter
Rotston
(a) Comparison of augmentations.
(b) toy example of residual relaxation.
Figure I: Left: the effect of different augmentations of CIFAR-10 10 test images with a supervised model (trained without using any data augmentation, more details in Appendix A). Right: an illus- tration of the exact alignment objective of multi-view methods (z' - 2) and the relaxed residual alignment our Prelax (z' ー 2). As the rotation largely modifies the image semantics, our Prelax adopts aaation-wware residual vector r to bridge the representation of two different views.
responding view. In this way. the model is encouraged to encode pretext-aware image semantics, which also yields good representations.
To summarize, strong augmentations like rotation carry meaningful semantics, while being harmful for existing multi-view methods due large semantic shift. To address this dilemma, in this paper, we propose generic approach that generalizes multi-view methods to cultivating stronger augmen- tations. Drawing inspirations from the soft-margin SVM, we propose residual aligument, which relaxes the exact alignment in multi-view methods by incorporating a residual vector between two views. Besides, we develop predictive loss for the residual ensure that it encodes the seman- tic shift between views (e.g., image rotation). We name this technique as Pretext-aware REsidual ReLAXation (Prelax), and an illustration is shown in Figure lb. Prelax serves as generalized multi-view method that is adaptive to large semantic shift and combines image semantics extracted from both pretext-int -intrrttninaaiant and pretext-aware methods. We summarize our contributions as follows:
We propose generic technique, Pretext-aware Residual Relaxation (Prelax), that general- izes multi-view representation learning to benefit from stronger image augmentations.
Prelax not only extracts pretext-invariant features as in multi-view methods, but also en- codes pretext-aware features into the pretext-aware residuals. Thus, it can serve a unified approach to bridge the two existing methodologies for representation learning.
Experiments show that Prelax can bring significant improvement over both multi-view and predictive methods a wide range of benchmark datasets.
2
Related Work
Multi-view Learning. Although multi-view learning could refer a wider literature [16], here we restrict our discussions to the context of Self-Supervised Learning (SSL), where multi-view methods learn representations by aligning multiple views of the same image generated through random data augmentation [1]. There are two kinds of methods to keep the representations well separated: contrastive methods, which achieve this by maximizing the difference between different samples [2, 12], and similarity -bssed methods, which prevent representation collapse via implicit mechanisms like predictor and gradient stopping [11, 5J. Although having lots modern variants, multi-view methods share the same methodology. that is to extract features that are invariant to the predefined augmentations, i.e., pretext-invarien features [20].
Predictive Learning. Another thread of methods is to leam representations by predicting self- generated surrogate labels. Specifically, it applies a transformation (e.g., image rotation) to the input image and requires the learner to predict properties of the transformation (e.g., the rotation angle) from the transformed images. As result, the extracted image representations are encouraged to become aware of the applied pretext (e.g., image rotation). Thus, we also refer to them as pretext-

---

aware methods. The pretext tasks can be various, to name few, Rotation [10], Jigsaw [21], Relative Path Location [8], Colorization [30].
Generalized Multi-view Learning. Although there are plenty of works on each branch, how to bridge the methodologies remains under-explored. Prior to our work, there are only few works on this direction. Some directly combine AMDIM (pretext-invariant) [1] and Rotation (pretext- aware) [10] objectives [9]. However, a direct combination of the two contradictory objectives may harm the final representation. LooC [28] proposes to separate the embedding space to several parts, where each subspace learns local invariance w.Er. specific augmentation. But this is achieved the cost of limiting the representation flexibility of each pretext to the predefined subspace. Different from them, our proposed Prelax provides more general solution by allowing an adaptive residual vector to encode the semantic shift. In this way, both kinds of features are encoded in the same representation space.
3
The Proposed Pretext-awark Residual Relaxation (Prelax) Method
3.1 Preliminary
Problem Formulation. Given unlabeled data {x:}. unsupervised representation learning aims to
learn encoder network Fo that extracts meaningful low-dimensional representations Rdd from high-dimensional input images x E TRd= The learned representation is typically evaluated on downstream classification task by learning linear classifier with labeled data {x;, [x)
Multi-view Representation Learning. For an input image x E IRd we can generate different view by data augmentation, operator from the pretext set T. Then, the transformed input x' and the original input x are passed into an online network Fo and target network Fs. respectively. Optionally, the output of the online network is further processed by an MLP predictor network Go. to match the output of the target network. As two different views of the same image (i.e., positive samples), and x' should have similar representations, so we align their representations with the following similarity loss,
t(x), where t E T refers to randomly drawn augmentation
Laim (x x: 0) 1|9o (Fo(x') Fx))))
The representations, e.g., Z Fe (x), are typically projected a unit spherical ball before calculat- ing the distance (z/[|z||>) [2]) which makes the by distance equivalent to the cosine similarity [2].
Remark. Aside from the similarity loss between positive samples, contrastive methods [22, 14, 23, 2] further encourage representation uniformity with an additional regularization minimizing the similarity between input and independently drawn negative sample. Nevertheless, some recent works find that the similarity loss alone already suffices [11, 5]. In this paper, we mainly focus on improving the alignment between positive samples in the similarity loss. It can also be easily extended to contrastive methods by considering the dissimilarity regularization.
3.2 Objective Formulation
As we have noticed, the augmentation sometimes may bring a certain amount of semantic shift. Thus, enforcing exact alignment of different views may hurt the representation quality, particularly when the data augmentation is too strong for the positive pairs to be matched exactly. Therefore, we need to relax the exact alignment in Eq. (1) to account for the semantic shift brought by the data augmentation.
Residual Relaxed Similarity Loss. Although the representations may not align exactly, i.e., z' 7' z, however, the representation identity will always hold: z' (z' z) z. where z' represents the shifted semantics by augmentation. This makes this identity proper candidate for multi-view alignment under various augmentations as long the shifted semantic is taken into consideration.
Specifically, we replace the exact alignment (denoted as +(-) in the similarity loss (Bq. (1)) by the proposed identity alignment, i.e.,
Go (z(z))) + 2 7
(o(2)) (r + Z
where we include a residual vector r the representations. To further enable better tradeoff between the exact and identity alignments,
A %/g z Fo(x') Fo(x) to represent the difference on
3

---

we have the following residual alignment:
Go (G) n
where a E [0, 1] is the interpolation parameter. When a = O, we recover the exact alignment; when = 1, recover the identity alignment. We name the corresponding learning objective as the Residual Relaxed Similarity (R2S) loss, which minimizes the squared &g distance among two sides:
Cias( ,x; x; = ||9o (Fo(x')) (G) (x)
(4)
Predictive Learning (PL) Loss. To ensure the relaxation works expected, the residual r should encode the semantic shift caused by the augmentation, i.e., the pretext. Inspired by predictive learn- ing [10], we utilize the residual to predict the corresponding augmentation for its pretext-awareness. In practice, the assigned parameters for the random augmentation can be generally divided into the discrete categorical variables ta (e.g.. flipping or not, graying not), and the continuous variables to (e.g., scale, ratio, jittered brightness). Thus, we learn a PL predictor Ho to predict (t", t²) with cross entropy loss (CE) and mean square error loss (MSE), respectively:
(xI(x' t; 0) (r). t²) (r) 1
Constraint on the Similarity. Different from the exact alignment, the residual vector can be un- bounded, i.e., the difference between views can be arbitrarily large. This is not reasonable as the two views indeed share many common semantics. Therefore, we should utilize this prior knowledge to prevent the bad cases under residual similarity and add the following constraint
Siim ||Go (Fo(x') (x) S E,
(6)
where denotes the maximal degree of mismatch allowed between positive samples.
The Overall Objective of Prelax. By combining the three components above, we can reliably encode the semantic shift between augmentations while ensuring good alignment between views:
min 0
Cizz( (x', x: 0) + 7 (PL x; x),),
s.t. | (x)) F s.
For simplicity, we transform into a Lagrangian objective with fixed multiplier 2 0, and obtain the overall Pretext-aware REsidual ReLAXation (Prelax) objective.
C (x', x; 0) P x; 0) + 3L (x, (x,x; 0),
where a tradeoffs between the exact and identity alignments, y adjusts the amount of pretext- awareness of the residual, and controlls the degree of similarity between positive pairs. An il- lustrative diagram of the Prelax objective is shown Figure 1.
Discussions. In fact, there are other alternatives to relax the exact alignment. For example, we can utilize a margin loss
mmargia (x',
xx; θ) = max( (Fo(x')) Fx 77,0),
where 1) 0 is threshold for the mismatch tolerance. However, it has two main drawbacks: 1) as each image and augmentation have different semantics, it is hard to choose a universal threshold for all images; and 2) the representation keeps shifting along with the training progress, making it even harder to maintain a proper threshold dynamically. Thus, a good relaxation should be adaptive the training progress and the aligning of different views. While our Prelax adopts pretext-aware residual vector, which is learnable, flexible, and semantically meaningful.
3.3 Theoretical Analysis
As Prelax encodes both pretext-invariant and pretext-aware features, it can be semantically richer than both multi-view learning and predictive learning. Following the information theoretic frame- work developed by [25], we show that Prelax provably enjoys better downstream performance.
We denote the random variable of input as X and learn a representation Z through a deterministic encoder Fo: Z = Fo ? The representation Z is evaluated for a downstream task T by learning
We use capitals denote the random variable, egg. X. and use lower cases denote its outcome, e.g.

---

Go (x1)
Fo
20
Go
Go (xp.
Go(r)
Go
7
Vig
CpL.
Excs Cea
X
F
B
8g
2
diagram of our proposed Prelax objective. An image x firstly augmented as x'. Then the positive pair x'). is processed by the online network Fe and the target network F respectively. Output the online network further processed by the target network Go. and the gradient of F& is detached, iee, stop grad, denoted sg. Then the outputs are used compute the three objectives, Li2s (Eq. 4), LeL (Bq. 5), and Laim (Eq. 1) in the Prelax objective (Eq. 7).
Figure 2:
a classifier on top of Z. From an information-theoreticc learning perspective, a desirable algorithm should maximize the Mutual Information (MI) between Z and T, i.e. I(Z; T) [6]. Supervised learning on task T can learn representations by directly maximizing [Z: T). Without access to the labels T, unsupervised learning resorts maximizing I(Z; S), where S denotes the surrogate signal designed by each method. Specifically, multi-view learning matches with the randomly aug- mented view, denoted as S,; while predictive learning uses to predict the applied augmentation, denoted as S.- In Prelax, as we combine both semantics, we actually maximize the MI w.r. their joint distribution, i.e., [Z; S., We denote the representations learned by supervised learning, multi-view learning, predictive learning, and Prelax as Zsup: Zuv, ZPL. Prelax respectively.
Theorem 1. Assume that by maximizing the mutual information, each method can relain all infor- mation in X about the learning signal S (or T), i.e., T(X:S) maxz (ZZ S). Then we have the following inequalities on their task-relevant information I(Z; ():
I(X:T) = I(Zeupi T) 2 [ZZrelaaxi T) max([(Zuvv: T), [ZZp: T)).
(10)
Theorem 2. Further assume that T is a K-class categorical variable. In general, we have the upper bound ue on the downstream Bayes errors pe = EE [1 maxteT P (T t/z)),
pe < u == log 2 sup logg K + I(X:T(S).
(11)
where P Th(P") min mmnfmmxx (max{ "0,, - denotes the thresholded Bayes error. Accord- ingly, we have the following inequalities on the upper bounds for different unsupervised methods,
Soup S UPrelax I min(u upr..) max(uu )
(12)
Theorem shows that Prelax extracts more relk-nelevant information than multi-view and predictive methods, and Theorem 2 further shows that Prelax has tighter upper bound on the downstream Bayes error. Therefore, Prelax is indeed theoretically superior to previous unsupervised methods by utilizing both pretext-invariant and pretext-aware features. Proofs are in Appendix B.
4 Practical Implementation
In this part, we present three practical variants of Prelax generalize existing multi-view backbones: I) one with existing multi-view augmentations (Prelax-std); 2) one with a stronger augmentation, image rotation (Prelax-rot); and one with previous two strategies (Prelax-all).
4.1 Backbone
BYOL [11] and SimSiam [5] are both similarity- methods and they differ mainly in the design of the target network Fo. BYOL [1I] utilizes momentum update of the target parameters & from
5

---

the online parameters 8, i.e. Φ t TΦ + - where E [0, 1] is the target decay rate. While SimSiam [5] simply regards the sstopped-gaddies online network as the target network, Le., Φ < sg(0), We mainly take SimSiam for discussion and our analysis also applies to BYOL..
For given training image x, SimSiam draws two random augmentations (41,22 and get two views X2), respectively. Then, SimSiam maximizes the similarity of their representations with a dual objective, where the two views can both serve the input and the target each other,
Csimsiam (x: 0) = | (Fe (x1)) Fx2))) 2 +
F Fx1)))
(13)
4.2
Prelax-std
To begin with, can directly generalize the baseline method with our Prelax method under existing
multi-view augmentation strategies [2. 11]. For the same positive pair (x1, X2). we can calculate
their residual vector 112 (x) Fx2) and use it for the R2S loss (Eq. (4))
Cis x 0) ||9o (Fo(x1)) (Gg( F (x2)
(14)
We note that there is no difference in using r12 r21 the two views are dual. Then, adopt the similarity loss in the reverse direction as our similarity constraint loss
Saim (X2. x1 0) ||9o (Fo(xe)) Fx
(15)
At last, use the residual r12 for the PL loss predict the augmentation parameters of x1, i.e., t1, because r12 = Z1 Z/2 directs towards 21. Combining the three losses above, we obtain our Prelax-
std objective,
L (x; 0) = 2i x2; 0) + CPL x2, 01 + BL (x2, x1;0)
(16)
4.3 Prelax-rot
As mentioned previously, with residual relaxation we can benefit from stronger augmentations that are harmful for multi-viww methods. Here, we focus on the image rotation example and propose the Prelax-rot objective with rotation residual vector. To achieve this, we further generalize existing dual-view methods by incorporating a third rotation view.
Specifically, given two views (X1,x.) x2) generated with existing multi-view augmentations, we ad- ditionally draw a random rotation angle R {0*, ,900, 180 270") and apply it rotate x clockwise, leading to the third view X3. Note that the only difference between x3 and x is the rotation semantic a. Therefore, we substitute x, with X3 in the similarity loss, we should add a rotation-aware residual r31 23- 21 to bridge the gap. Motivated by this analysis, we propose the Rotation Residual Relaxation Similarity (R3S) loss,
(17) which replace (x)((x))) by its rotation-relaxed version (Fo(())) (Ga( in the similarity loss. Comparing the R2S loss (Eq. 14) and the R3S loss, we note that the relaxation of the R2S loss accounts for all the semantic shift between x; and x2, while that of the R3S loss only accounts for the rotation augmentation between X₁ and X3- Therefore, we could use the residual r3ı to predict the rotation angle with the following RotPL loss for its
Liss (x
(Fo ( Fo(xx)1
(xL, X3, 0) = CE(He (T31). a). Combining with the similarity constraint, we obtain the Prelax-rot objective:
(18)
P vot (x: 0) Cass (x1:: 0) + 7CPL (x1, X3, a; 0) + 3 (X2, x1: 0).
(19)
4.4 Prelax-all
We have developed Prelax-std that cultivates existing multi-view augmentations and Prelax-rot that incorporates image rotation. Here, we further utilize both existing augmentations and image rotation by combining the two objectives together, denoted as Prelax-all:
Lprelax-all all (x; 0) (C (x1, X2; 0) + Las (x13; 0)) 21 CPI(x), X2, 01:
1
(20)
12 not
+
(x, a; 0) + 3L (X2, X1; 0).
2
where a1, 022 72,72 denotes the coefficients for R2S, R3S, PL and RotPL losses, respectively.

---

4.5 Discussions
Here we design three practical versions as different implementations of our generic framework of residual relaxation. Among them, Prelax-std focuses further cultivating existing augmenta- tion strategies, Prelax-rot is to incorporate the stronger (potentially harmful) rotation augmentation, while Prelax-all combines them all. Through the three versions, demonstrate the wide applica- bility of Prelax as a generic framework. As for practical users, they could also adapt Prelax to their own application by incorporating specific domain knowledge. In this paper, images, we take rotation as a motivating example as it is harmful for natural images. Nevertheless, rotation is not necessarily harmful in other domains, e.g, medical images. Instead, random cropping could instead be very harmful for medical images as the important part could lie in the corner. In this scenario, our residual relaxation mechanism could also be used to encode the semantic shift caused by cropping and alleviate its bad effects.
focus on natural
Experiments
Datasets. Due to computational constraint, we carry out experiments on a range of medium-sized real-world image datasets, including well known benchmarks like CIFAR-10 [15], CIFAR-100 [15], and two ImageNet variants: Tiny-ImageNet-200 (200 classes with image size resized to 32x32) [27] and ImageNette (10 classes with image size 128x 128)².
Backbones. As Prelax is designed to be a generie method for generalizing existing multi-view meth- ods, we implement it on two different multi-view methods, SimSiam [5] and BYOL [11] [1]]. Specif- ically, we notice that SimSiam reported results on CIFAR-IO, while the official code of BOQL. in- cluded results on ImageNette. For a fair comparison, we evaluate SimSiam and its Prelax variant on CIFAR- 10, and evaluate BYOL and its Prelax variant on ImageNette. In addition, we evaluate Sim- Siam and its Prelax variant on two additional datasets CIFAR-100 and liny-ImageNet-200 which are more challenging because they include larger number of classes. For computational efficiency, we adopt the ResNet-18 [13] backbone (adopted in SimSiam [5] for CIFAR-10) to benchmark our experiments. For a comprehensive comparison, we also experiment with larger backbones, like ResNet-34 [13], and the results are included in Appendix C.
Setup. For Prelax-std, we use the same data augmentations as SimSiam [2, 5] (or BYOL [11]), in- cluding RandomkesizeddCnee ing the PyTorch notations. For Prelax-rot and Prelax-all, we further apply random image rotation at last of the transformation, where the angles are drawn randomly from {0,, 90" 180", 270"). J. To generate targets for the PL objective in Prelax, for each image, we collect the assigned parameters in each random augmentation, such crop centers, aspect ratios, rotation angles, etc. More details can be found in Appendix A.
RandomHHorizottllFigg ColorJitter, and RaydomGeayscale, etc us-
Training, For SimSiam and its Prelax variants, we follow the same hyperparameters in [5] on CIFAR-10. 10. Specifically, we use ResNet-18 as the backbone network, followed by a 3-layer projec- tion MLP, whose hidden and output dimension are both 2048. The predictor is 2-layer MLP whose hidden layer and output dimension are 512 and 2048 respectively. We use SGD for pre-training with batch size 512, learning rate 0.03, momentum 0.9, weight decay 5 x 10-4, and cosine decay sched- ule [18] for 800 epochs. For BYOL and its Prelax variants, we also adopt the ResNet- 18 backbone, and the projector and predictor are 2-layer MLPs whose hidden layer and output dimension are 256 and 4096 respectively. Following the default hyper-parameters on ImageNette", we use LARS opti- mizer [29] to train 1000 epochs with batch size 256, learning rate 2.0, weight decay I x 10-6 while excluding the biases and batch normalization parameters from both LARS adaptation and weight de- cay. For the target network, the exponential moving average parameter starts from Touse 0.996 and increases to 1 during training. As for the Prelax objective, we notice that sometimes, adopting reverse residual r21 in the R2S loss (Eq. (14)) can bring slightly better results, which could be due to the swapped prediction in SimSiam's dual objective (Eq. (13)). Besides, a naïve choice of Prelax coefficients already works well:
1, 1, 0.1 for Prelax-std and Prelax-rot, and a a2 = 1, 3 = 1,7) = 72- = 0.1 for Prllax-all. More discussion about the eoper-parametery of Prelax can be found in Appendix E.
Phttps://g thub. htt.bb.comafattta http//aastai/mmmmeeetttttt 'nttps: : wwthub..htttgggg com
7

---

Table 1: Linear evaluation on CIFAR-10 (a) and ImageNette (b) with ResNet- backbone, TTA:
Test-Time Augmentation.
(a) CIFAR-10.
(b) ImageNette.
Method
Acc. (%)
Method
Acc. (%)
Supervised [13] (e-produuedd
95.0
Supervised
91.0
Rotation [10] (re- (eepprouuedd)
88.3
Supervised + TTA
92.2
BYOL [11] (re-prodeey (re-produced)
91.1
BYOL [1I] (ResNet-18)
91.9
SimCLR [2]
91.1
BYOL [11] (ResNet-50)
92.3
SimSiam [5]
91.8
BYOL Prelax (ResNet-11)
92.6
SimSiam + Prelax
93.4
Evaluation. After unsupervised training, we evaluate the backbone network by fine-tuning linear classifier top of its representation with other model weights held fixed. For SimSiam and its Prelax variants, the linear classifier is trained on labeled data from scratch using SGD with batch size 256, learning rate 30.0, momentum 0.9 for 100 epochs. The learning rate decays by 0.1 the 60-th and 80-th epochs. For BYOL and its Prelax variants, we use SGD with Nesterov momentum over 80 epochs using batch size 25, learning rate 0.5 and momentum 0.9. Besides the in-domain linear evaluation, we also evaluate its transfer learning performance on the out-of-domain data by learning linear classifier on the labeled target domain data.
5.1 Performance on Benchmark Datasets
CIFAR-10. Table la, we compare Prelax against previous multi-view methods (SimCLR [2], SimSiam [5], and BYOL [11]) and predictive methods (Rotation [10]) on CIFAR-10. We can see that multi-view methods are indeed better than predictive ones. Nevertheless, predictive learning alone (e.g.. Rotation) achieves quite good performance, indicating that pretext-aware features are also very useful. By encoding both pretext-invariant and pretext-aware features, Prelax outperforms previous methods by large margin, and achieve state-of-the-an performance CIFAR-10. A comparison the learning dynamics between SimSiam and Prelax can be found in Appendix F.
ImageNette. Beside the SimSiam backbone, we further apply our Prelax loss to the BYOL frame- work [11] and evaluate the two methods on the ImageNette dataset. In Table 1b, Prelax also shows a clear advantage over BYOL. Specifically, it improves the ResNet-18 18 version of BYOL by 0.7%, and even outperforms the ResNet-50 version by 0.3%.
Here, we can see that Prelax yields significant improvement on two different datasets with two dif- ferent backbone methods. Thus, Prelax could serve as generic method for improving existing multi-view methods by encoding pretext-aware features into the residual relaxation. For complete- ness, we also evaluate Prelax on the large scale dataset, ImageNet [7], well as its transferability to other kinds of downstream tasks, such as object detection and instance segmentation on MS COCO [17]. shown in Appendix D, Prelax still consistently outperforms the baselines across all tasks.
5.2 Effectiveness of Prelax Variants
For a comprehensive comparison of the three variants of Prelax objectives (Prelax-std, Prelax-rot, and Prelax-all), conduct controlled experiments backbone. Except CIFAR-10, we also conduct experiments on CIFAR-100 and Tiny-ImageNet- 200, which are more challenging with more classes of images. For fair comparison, use the same training and evaluation protocols across all tasks.
range of datasets based on the SimSiam
In-domain Linear Evaluation. As shown in Table 2a, our Prelax objectives outperform the multi- view objective consistently on all three datasets, where Prelax-all improves SimSiam by 1.6% on CIFAR-10 10, 3.1% on CIFAR-100, and 1.5% on Tiny-ImageNet-2000 Besides, Prelax-std and Prelax- not are also better than SimSiam in most cases. Thus, the pretext-aware residuals Prelax indeed help encode more useful semantics.

---

Table A detailed comparison of SimSiam [5] and Prelax (ours) across three datasets: CIFAR-10 (C10), CIFAR-100 (C100), and Tiny-Imane mny-ImmagNeet2200 (Tiny200) with the same hyper-parameters.
(a) In-domain linear evaluation.
Method
CIFAR-10
CIFAR-100
Tmny-mmageNet-200
SimSiam [5]
91.8
66.9
47.7
SimSiam Prelax-std
92.5
67.5
47.9
SimSiam + Prelax-rot
92.4
67.3
47.1
SimSiam Prelax-all
93.4
70.0
49.2
(b) Out-of-domain linear evaluation.
Method
C100 - C10
Tiny200 - C10
Tiny200 + C100
SimSiam [5]
44.1
43.9
21.8
SimSiam + Prelax-std
45,0
45.1
21.8
SimSiam Prelax-rot
45.0
45.1
22.0
SimSiam + Prelax-all
44.9
44.6
22.1
Table 3: Linear evaluation results of possible mechanisms for generalized multi-view learning on
CIFAR-10 with SimSiam backbone.
(b) Ablation study.
(a) Comparison against alternative options.
Method
Acc. (%)
Method
Acc. (%)
Sim (i.e. SimSiam [5])
91.8
SimSiam [5]
91.8
Sim + PL
92.2
SimSiam + margin loss
91.9
Sim + R2S
91.5
R2S PL
91.7
Rotation [10]
88.3
Sim + PL. + R2S (Prelax-std)
92.5
SimSiam rotation aug-
87.9
SimSiam Rotation loss
91.7
Sim + RotPL
91.1
Sim + R3S
91.9
SimSiam Prelax (ours)
93.4
R3S RotPL
79.8
Sim + RotPL + R3S (Prelax-rot)
92.4
Out-of-domain Linear Evaluation. Besides the in-domain linear evaluation, we also transfer the representations to target domain. In the out-of-domain linear evaluation results shown in Table 2b, the Prelax objectives still have clear advantage over the multi-view objective (SimSiam), while sometimes Prelax-std and Prelax-ro enjoy better transferred accuracy than Prelax-all.
5.3 Empirical Understanding of Prelax
Comparison against Alternative Options. In Table 3a, compare Prelax against several other relaxation options. "SimSiam + margin" refers to the margin loss discussed in Fq. (9), which uses scalar 77 relax the exact alignment in multi-view methods. Here we tune the margin r 0.5 with the best performance. Nevertheless, it has no clean advantage over SimSiam. Then, we try several options for incorporating strong augmentation and image rotation: 1) Rotation the PL baseline by predicting rotation angles [10], which is inferior to multi-view methods (SimSiam). 2) "SimSiam rotation aug." directly applies random rotation augmentation to each view, and learn with the SimSiam loss. However, it leads to lower accuracy, showing that the image rotation, strong augmentation, will huri the performance of multi-view methods. 3) "SimSiam + Rotation" directly combines the SimSiam loss and the Rotation loss for training, which is still ineffective. 4) Our Prelax shows a significant improvement over SimSiam and other variants, showing that the residual alignment is an effective mechanism for utilizing strong augmentations like rotation.

---

(a) Representation visualization.
(b) Nearest image retrieval.
Figure 3: (a) Representation visualization of our Prelax CIFAR-I0 test set. Each point represents an image representation and its color denotes the class the image. (b) On CIFAR-10 10 test set, given 10 random queries (not cherry-picked), retrieve 15 nearest images in the representation space with Prelax (ours).
Ablation Study. We perform ablation study of each component of the Prelax objectives on CIFAR- 10. From Table 3b, we notice that simply adding the PL loss alone cannot improve over SimSiam consistently, for example, Sim + RotPL causes 0.7 point drop in test accuracy. While with the help of our residual relaxation, we can improve over the baselines significantly and consistently, for example, Prelax-rot (Sim RotPL R3S) brings 0.6 point improvement on accuracy. Besides, we can see that the PL loss is necessary by making the residual pretext-aware, without which the performance drops lot, and the similarity constraint (Sim loss) is also important by avoiding bad cases when augmented images drift far from the anchor image. Therefore, the ablation study shows the residual relaxation loss, similarity loss, and PI. loss all matter in our Prelax objectives.
5.4 Qualitative Analysis
Representation Visualization. To provide an intuitive understanding of the learned representations, visualize them with t-SNE [26] Figure 3a. We that general, Prelax learn well-separated clusters of representations corresponding to the ground-truth image classes.
Image Retrieval. In Figure 3b, evaluate Prelax an image retrieval task. Given random query image (not cherry-picked). the top-15 most similar images in representation space are retrieved, and the query itself is shown in the first column. We Prelax has no access to labels, the retrieved nearest images of Prelax are all correctly from the same class and semantically consistent with the query.
that although the unsupervised training with
Conclusion
In this paper, we proposed a generic method, Prelax (Pretext-aware Residual Relaxation). to ac- count for the (possibly large) semantic shift caused by image augmentations. With pretext-aware learning of the residual relaxation, method generalizes existing multi-viww learning by encod- ing both pretext-aware and pretext-invariant representations. Experiments show that our Prelax has outperformed existing multi-view methods significantly variety of benchmark datasets.
Acknowledgement
Yisen Wang is partially supported by the National Natural Science Foundation of China under Grant 62006153, and Project 2020BD006 supported by PKU-Baidu Fund. Jiansheng Yang supported by the National Science Foundation of China under Grant No. 11961 141007. Zhouchen Lin was supported by the NSF China (No.s 61625301 and 61731018), NSFC Tianyuan Fund for Mathematics (No. 12026606) and Project 2020BD006 supported by PKU-Baidu Fund.
10

---

References
[1] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by max- imizing mutual information across views. arXiv preprint arXiv:1906.00911 2019. 1, 2,
[2] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICMI., 2020. 2, 6, 7,
[3] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi- cumi-supereieed learners. arXiv preprint arXiv:2006. 10029, 2020.
[4] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint 00X2: 2020.
[5] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. arXiv preprint arXiv:2011.10566 2020. 1, 2, 3, 6, 8,9
[6] Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. CVPR, 2009. 8
[8] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learn- ing by context prediction. ICCV, 2015.
[9] Zeyu Feng. Chang Xu, and Dacheng Tao. Self-supervised representation learning from multi- domain data. ICCV, 2019. 3
[10] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. ICLR, 2018. 3,3, 4, 8,
[1I] Jean-Bastien Grill, Florian Strub, Florent Altché, C. Tallec, Pierre H. Richemond, Elena Buchatskaya, C. Doersch, Bernardo Avila Pires. Zhaohan Daniel Guo, Mohammad Ghesh- laghi Azar, B. Piot, K. Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: new approach self-supervised learning. NeurIPS. 2020. 3. 7,8
[12] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. CVPR, 2020. 1,2
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016. 7,8
[14] Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. ICLR, 2019. 1,3
[15] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.1
[16] Yingming Li, Ming Yang, and Zhongfei Zhang. A survey of multi-view representation learn- ing. IEEE transactions Knowledge and Data Engineering, 31310):1863-188,, 2018. 2
[17] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In ECCV, 2014. 8
[18] 1lya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. ICLR, 2017.7
[19] Sean Metzger, Aravind Srinivas, Trevor Darrell, and Kurt Keutzer. Evaluating self-supervised pretraining without using labels. arXiv preprint S0ti:22009007222 2020.
[20] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant repre- sentations. CVPR, 2020. 2

---

[21] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving Jigsaw puzzles. ECCV, 2016. 3,3
[22] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv: arXiv: 1000000077,, 2018. 1,3
[23] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint / 2019. 1,3
[24] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. NeurIPS, 2020. 1
[25] Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Self- supervised learning from a multi-view perspective. ICLR, 2021. 4
[26] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9(11), 2008. 10
[27] Jiay Wu, Qixiang Zhang, and Guoxi Xu. Tiny ImageNet challenge, 2017.
[28] Tete Xiao, Xiaolong Wang, Alexei A Efros, and Trevor Darrll. What should not be contrastive in contrastive learning. ICIR,, 2021. 3
[29] Yang You, Igor Gitman, and Boris Ginsburg. Scaling sgd batch size to 32k for ImageNet training. arXiv preprint
2017. 7
[30] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. ECCV, 2016.
1,3
12


Reviews:
[{"review_id": "utrEO-BJ2YX", "reviewer": "Reviewer_TbTy", "summary": "The paper focus on the representation learning using a relaxed alignment based on the fact that some strong data augmentations may hurt the performance. The authors introduced an adaptive residual vector and designed several novel objectives to make the residual vector useful and meaningful. Experiments shows a significant improvement over various backbones. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is well organized and easy to follow. The proposed prelax method is well motivated and general. Prelax can be used for various data augmentations even there is some semantic shift, which is very important for real-world applications. \n- Prelax is also a unified approach to bridge the pretext-invariant features and pretext-aware features for representation learning. \n- The authors also provided the theoretical analysis based on the information-theoretic framework, which shows that Prelax provably enjoys better downstream performance.\n\nWeakness:\n- Prelax has three versions, Prelax-std, Prelax-rot and Prelax-all, then which one should be used when facing with an application?\n- How to choose the hyperparameters alpha and gama?\n- The difference between R2S and R3S is not clear.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No significant social risk", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "cWJ-fwkN5ae", "reviewer": "Reviewer_wdr9", "summary": "The paper proposes a novel Pretext-aware Residual Relaxation (Prelax) approach to self-supervised learning that does not require exact alignment of the two different views, which can be harmful for views created from some types of augmentations, but introduces a residual vector that encodes the semantic shift caused by the augmentation. To ensure pretext-awareness, a loss is introduced to predict the pretext task from the residual vector. A theoretical analysis is provided to show Prelax's advantage over unsupervised approaches that either only consider multi-view learning or only predictive learning. Experiments indicate that adding Prelax to common self-supervised methods such as SimSiam and BYOL improves performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \nThe proposed method bridges the gap between the contrastive/similarity-based methods, which learn pretext-invariant features, and the predictive learning frameworks that learn pretext-aware features. While previous work have attempted to do this in an ad-hoc manner by directly combining multiple losses or by using separate feature spaces, the authors propose a well-motivated approach of encoding the pretext-awareness into a residual alignment vector. Building on the theoretical framework developed in [1], they further provide an information theoretic perspective, illustrating the benefits of the proposed approach. \n\nQuality: \nThe paper is technically sound and claims are well-motivated. However, an analysis of the interdependence and effects of the multiple losses would have been interesting. For instance by having a closer look at the gradients and monitor the alignment/mismatch of the various terms (or comparing them to the gradient of a supervised loss). Further, do the authors observe negligible improvements or performance degradation when increasing the number of views further beyond three?\n\nClarity: \nThe paper is overall easy to follow and the material is presented in a manner that should allow reproducibility.\n- One statement, that could have been expanded a bit is the discussion of the results in the out-of-domain linear evaluation (2b). Do the authors have a hypothesis on why the Prelax-std/Prelax-rot perform slightly better than Prelax-all?\n- The authors definition of multi-view seems a bit limited. Although multi-view representation learning includes the self-supervised techniques where additional views are created through augmentations, it commonly also includes settings where data is actually obtained from multiple views (different camera angles, or even different modalities).\n- The claim that rotation causes severe accuracy drops is a bit strong and depends on the application domain. While it is a valid assumption for natural images, this is not the case in other domains. The authors might want to more explicitly include that this claim/work is addressing the case of natural images (although the method in itself is more general). \n\nSignificance:\nThe proposed approach is a relative generic framework that can be combined with most existing methods and should motivate additional work on trying to develop methods that exploit both pretext-invariant and pretext-aware features explicitly. Results illustrate a sufficient boost over the baseline approaches that only focus on similarity-based methods or predictive learning.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not discuss the limitations of their proposed approach. While the reviewer agrees that the negative social impact should be limited due to its general methodology focus, there are certain limitations, such as the reliance on manually choosing the augmentations, which will be domain and dataset dependent.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "PXzIra9uhv", "reviewer": "Reviewer_6qif", "summary": "In this work, the authors propose a new approach, named Pretext-aware Residual Relaxation (Prelex) to learning the representations from multiple views. The method mainly employs the residual relaxed similarity loss to improve the alignment between positive samples. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Majors.\n1 Motivation. Fig. 1 (a) shows the main motivation that rotation augmentation causes severe accuracy drop in a standard supervised model. In line 34, the authors claim the rotation augmentation carries meaningful semantics while being harmful to existing multi-view methods due to a large semantic shift. Thus, they propose a generic approach to address this dilemma. I have two questions. (1) I am curious whether the setting in Fig. 1 (a) is right or suitable to conclude the rotation is harmful. The setting in Fig. 1 (a) is that they train the model using two data augmentations (random crop and random horizontal flip in line 69 of the supple), then evaluate the model on images under four data augmentations, including horizontal flip, grayscale, color jitter and rotation. The drop of accuracy on rotated images seems reasonable since the trained model doesn’t ‘see’ such data? Maybe, these experimental results cannot support their motivation. (2) If the setting in Fig.1 (a) is right, the authors should try to indicate the effectiveness of their proposed method to deal with the dilemma on the same setting in Fig. 1 (a). I don’t find any results about it. \n\n2 Fig. 2 shows the network. But it needs more correlation with the equations in the manuscript and indications, the meaning of the symbols. \n\n3 Comparison with existing work [5] and [10]. The authors need to provide more comparisons to list the similarities and differences in a table format, including the key loss items, such as the residual relaxed similarity loss, predictive learning loss, and constraint on the similarity. The comparison is used to highlight the novel part of Prelax. After reading the whole manuscript, at the first time, I am confused that the proposed Prelax is an individual loss combination (in Section 3) or some additional parts used in SOTA [5][10] (such as the items, ‘SimSiam + Prelax’ or ‘BYOL+Prelax’ in Table 1) \n\n4 In section 4, they design three variants of Prelax. I want to know whether they investigate the dual order in Prelax-std, rot, all. For example, in eq (14), the order is $r_{12}$. How about the $r_{21}$. Also for eq (15,15,17,18,19,20).  \n\n5 Experiment. The datasets used in this work include CIFAR-10, CIFAR-100, Tiny-ImageNet-200 and ImageNette, which might not support a wide range of benchmark datasets. The authors need to report the results on more downstream tasks as the table 5 in [5], such as detection and segmentation.  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hrs"}, "extra_text": {}}, {"review_id": "EfkEgiMSk5U", "reviewer": "Reviewer_hQVQ", "summary": "The paper proposes a residual relaxation method to encode the semantic shift for the multi-view self-supervised learning. The method relaxes the exact alignment for the latent vector representation from different views. The experimental results show the method can improve the performance of the data augmentation.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Originality: The work is a novel combination of the multi-view representation learning and the predictive learning, and proposes residual relaxation to improve the performance.\n\n2. Quality: The work is technically sound. But, from the ablation study, the predictive learning term contributes more than the proposed residual relaxation technique.\n\n3. Clarity: The paper is well written and organized. \n\n4. Significance: The result has referential significance. The other researchers could build on the idea of residual relaxation for other applications. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper relaxes the alignment objective in multiview self-supervised learning when the data augmentation causes semantic shifts in different views. The proposed pretext-aware residual relaxation method allows an adaptive residual vector between different views. The proposed method can benefit from stronger image augmentations like rotation and outperform the existing methods. Based on a recent theoretical framework on self-supervised learning, the authors provide theoretical guarantees of the proposed method. The paper is well motivated, well written and has practical impact on downstream image classification tasks. Reviewers all agreed that the proposed solution is interesting and the improvement is significant. I am recommending acceptance of this paper.  The authors need to make sure they add the new results in the rebuttal to the final version.

Decision: Accept (Poster)