Residual Relaxation for Multi-view Representation Learning
Year: 2022

Abstract:
Multi-view methods learn representations by aligning multiple views of the same image and their performance largely depends on the choice of data augmentation. In this paper, we notice that some other useful augmentations, such as image rotation, are harmful for multi-view methods because they cause a semantic shift that is too large to be aligned well. This observation motivates us to relax the exact alignment objective to better cultivate stronger augmentations. Taking image rotation as a case study, we develop a generic approach, Pretext-aware Residual Relaxation (Prelax), that relaxes the exact alignment by allowing an adaptive residual vector between different views and encoding the semantic shift through pretext-aware learning. Extensive experiments on different backbones show that our method can not only improve multi-view methods with existing augmentations, but also benefit from stronger image augmentations like rotation.

heading: 1 Introduction
text: Without access to labels, self-supervised learning relies on surrogate objectives to extract meaningful representations from unlabeled data, and the chosen surrogate objectives largely determine the quality and property of the learned representations [24, 19]. Recently, multi-view methods have become a dominant approach for self-supervised representation learning that achieves impressive downstream performance, and many modern variants have been proposed [22, 14, 1, 23, 2, 12, 3, 4, 11, 5]. Nevertheless, most multi-view methods can be abstracted and summarized as the following pipeline: for each input x, we apply several (typically two) random augmentations to it, and learn to align these different “views” (x1,x2, . . . ) of x by minimizing their distance in the representation space.
In multi-view methods, the pretext, i.e., image augmentation, has a large effect on the final performance. Typical choices include image re-scaling, cropping, color jitters, etc [2]. However, we find that some augmentations, for example, image rotation, is seldom utilized in state-of-the-art multiview methods. Among these augmentations, Figure 1a shows that rotation causes severe accuracy drop in a standard supervised model. Actually, image rotation is a stronger augmentation that largely affects the image semantics, and as a result, enforcing exact alignment of two different rotation angles could degrade the representation ability in existing multi-view methods. Nevertheless, it does not mean that strong augmentations cannot provide useful semantics for representation learning. In fact, rotation is known as an effective signal for predictive learning [10, 30, 21]. Differently, predictive methods learn representations by predicting the pretext (e.g., rotation angle) from the cor-
∗Corresponding author: Yisen Wang (yisen.wang@pku.edu.cn).
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
responding view. In this way, the model is encouraged to encode pretext-aware image semantics, which also yields good representations.
To summarize, strong augmentations like rotation carry meaningful semantics, while being harmful for existing multi-view methods due to large semantic shift. To address this dilemma, in this paper, we propose a generic approach that generalizes multi-view methods to cultivating stronger augmentations. Drawing inspirations from the soft-margin SVM, we propose residual alignment, which relaxes the exact alignment in multi-view methods by incorporating a residual vector between two views. Besides, we develop a predictive loss for the residual to ensure that it encodes the semantic shift between views (e.g., image rotation). We name this technique as Pretext-aware REsidual ReLAXation (Prelax), and an illustration is shown in Figure 1b. Prelax serves as a generalized multi-view method that is adaptive to large semantic shift and combines image semantics extracted from both pretext-invariant and pretext-aware methods. We summarize our contributions as follows:
• We propose a generic technique, Pretext-aware Residual Relaxation (Prelax), that generalizes multi-view representation learning to benefit from stronger image augmentations.
• Prelax not only extracts pretext-invariant features as in multi-view methods, but also encodes pretext-aware features into the pretext-aware residuals. Thus, it can serve as a unified approach to bridge the two existing methodologies for representation learning.
• Experiments show that Prelax can bring significant improvement over both multi-view and predictive methods on a wide range of benchmark datasets.

heading: 2 Related Work
text: Multi-view Learning. Although multi-view learning could refer to a wider literature [16], here we restrict our discussions to the context of Self-Supervised Learning (SSL), where multi-view methods learn representations by aligning multiple views of the same image generated through random data augmentation [1]. There are two kinds of methods to keep the representations well separated: contrastive methods, which achieve this by maximizing the difference between different samples [2, 12], and similarity-based methods, which prevent representation collapse via implicit mechanisms like predictor and gradient stopping [11, 5]. Although having lots of modern variants, multi-view methods share the same methodology, that is to extract features that are invariant to the predefined augmentations, i.e., pretext-invariant features [20].
Predictive Learning. Another thread of methods is to learn representations by predicting selfgenerated surrogate labels. Specifically, it applies a transformation (e.g., image rotation) to the input image and requires the learner to predict properties of the transformation (e.g., the rotation angle) from the transformed images. As a result, the extracted image representations are encouraged to become aware of the applied pretext (e.g., image rotation). Thus, we also refer to them as pretext-
aware methods. The pretext tasks can be various, to name a few, Rotation [10], Jigsaw [21], Relative Path Location [8], Colorization [30].
Generalized Multi-view Learning. Although there are plenty of works on each branch, how to bridge the two methodologies remains under-explored. Prior to our work, there are only a few works on this direction. Some directly combine AMDIM (pretext-invariant) [1] and Rotation (pretextaware) [10] objectives [9]. However, a direct combination of the two contradictory objectives may harm the final representation. LooC [28] proposes to separate the embedding space to several parts, where each subspace learns local invariance w.r.t. a specific augmentation. But this is achieved at the cost of limiting the representation flexibility of each pretext to the predefined subspace. Different from them, our proposed Prelax provides a more general solution by allowing an adaptive residual vector to encode the semantic shift. In this way, both kinds of features are encoded in the same representation space.

heading: 3 The Proposed Pretext-aware Residual Relaxation (Prelax) Method

heading: 3.1 Preliminary
text: Problem Formulation. Given unlabeled data {xi}, unsupervised representation learning aims to learn an encoder network Fθ that extracts meaningful low-dimensional representations z ∈ Rdz from high-dimensional input images x ∈ Rdx . The learned representation is typically evaluated on a downstream classification task by learning a linear classifier with labeled data {xi, yi}. Multi-view Representation Learning. For an input image x ∈ Rdx , we can generate a different view by data augmentation, x′ = t(x), where t ∈ T refers to a randomly drawn augmentation operator from the pretext set T . Then, the transformed input x′ and the original input x are passed into an online network Fθ and a target network Fφ, respectively. Optionally, the output of the online network is further processed by an MLP predictor network Gθ, to match the output of the target network. As two different views of the same image (i.e., positive samples), x and x′ should have similar representations, so we align their representations with the following similarity loss,
Lsim(x′,x;θ) = ‖Gθ (Fθ(x′))−Fφ(x)‖22. (1) The representations, e.g., z = Fθ(x), are typically projected to a unit spherical ball before calculating the distance (z/‖z‖2), which makes the `2 distance equivalent to the cosine similarity [2]. Remark. Aside from the similarity loss between positive samples, contrastive methods [22, 14, 23, 2] further encourage representation uniformity with an additional regularization minimizing the similarity between input and an independently drawn negative sample. Nevertheless, some recent works find that the similarity loss alone already suffices [11, 5]. In this paper, we mainly focus on improving the alignment between positive samples in the similarity loss. It can also be easily extended to contrastive methods by considering the dissimilarity regularization.

heading: 3.2 Objective Formulation
text: As we have noticed, the augmentation sometimes may bring a certain amount of semantic shift. Thus, enforcing exact alignment of different views may hurt the representation quality, particularly when the data augmentation is too strong for the positive pairs to be matched exactly. Therefore, we need to relax the exact alignment in Eq. (1) to account for the semantic shift brought by the data augmentation.
Residual Relaxed Similarity Loss. Although the representations may not align exactly, i.e., z′ 6= z, however, the representation identity will always hold: z′ − (z′ − z) = z, where z′ − z represents the shifted semantics by augmentation. This makes this identity a proper candidate for multi-view alignment under various augmentations as long as the shifted semantic is taken into consideration.
Specifically, we replace the exact alignment (denoted as→←) in the similarity loss (Eq. (1)) by the proposed identity alignment, i.e.,
Gθ(z′θ)→← zφ ⇒ Gθ(z′θ)− Gθ(r)→← zφ, (2)
where we include a residual vector r ∆= z′θ − zθ = Fθ(x′) − Fθ(x) to represent the difference on the representations. To further enable a better tradeoff between the exact and identity alignments,
we have the following residual alignment:
Gθ(z′θ)− αGθ(r)→← zφ, (3) where α ∈ [0, 1] is the interpolation parameter. When α = 0, we recover the exact alignment; when α = 1, we recover the identity alignment. We name the corresponding learning objective as the Residual Relaxed Similarity (R2S) loss, which minimizes the squared `2 distance among two sides:
LαR2S(x′,x;θ) = ‖Gθ (Fθ(x′))− αGθ(r)−Fφ(x)‖22. (4)
Predictive Learning (PL) Loss. To ensure the relaxation works as expected, the residual r should encode the semantic shift caused by the augmentation, i.e., the pretext. Inspired by predictive learning [10], we utilize the residual to predict the corresponding augmentation for its pretext-awareness. In practice, the assigned parameters for the random augmentation t can be generally divided into the discrete categorical variables td (e.g., flipping or not, graying or not), and the continuous variables tc (e.g., scale, ratio, jittered brightness). Thus, we learn a PL predictor Hθ to predict (td, tc) with cross entropy loss (CE) and mean square error loss (MSE), respectively:
LPL(x′,x, t;θ) = CE(Hdθ(r), td) + ‖Hcθ(r)− tc‖22. (5)
Constraint on the Similarity. Different from the exact alignment, the residual vector can be unbounded, i.e., the difference between views can be arbitrarily large. This is not reasonable as the two views indeed share many common semantics. Therefore, we should utilize this prior knowledge to prevent the bad cases under residual similarity and add the following constraint
Lsim = ‖Gθ (Fθ(x′))−Fφ(x)‖22 ≤ ε, (6) where ε denotes the maximal degree of mismatch allowed between positive samples.
The Overall Objective of Prelax. By combining the three components above, we can reliably encode the semantic shift between augmentations while ensuring a good alignment between views:
min θ LαR2S(x′,x;θ) + γLPL(x′,x;θ),
s.t. ‖Gθ (Fθ(x′))−Fφ(x)‖22 ≤ ε. (7)
For simplicity, we transform it into a Lagrangian objective with a fixed multiplier β ≥ 0, and obtain the overall Pretext-aware REsidual ReLAXation (Prelax) objective,
LαR2S(x′,x;θ) + γLPL(x′,x;θ) + βLsim(x′,x;θ), (8) where α tradeoffs between the exact and identity alignments, γ adjusts the amount of pretextawareness of the residual, and β controlls the degree of similarity between positive pairs. An illustrative diagram of the Prelax objective is shown in Figure 1.
Discussions. In fact, there are other alternatives to relax the exact alignment. For example, we can utilize a margin loss
Lmargin(x′,x;θ) = max(‖Gθ (Fθ(x′))−Fφ(x)‖22 − η, 0), (9) where η > 0 is a threshold for the mismatch tolerance. However, it has two main drawbacks: 1) as each image and augmentation have different semantics, it is hard to choose a universal threshold for all images; and 2) the representation keeps shifting along with the training progress, making it even harder to maintain a proper threshold dynamically. Thus, a good relaxation should be adaptive to the training progress and the aligning of different views. While our Prelax adopts pretext-aware residual vector, which is learnable, flexible, and semantically meaningful.

heading: 3.3 Theoretical Analysis
text: As Prelax encodes both pretext-invariant and pretext-aware features, it can be semantically richer than both multi-view learning and predictive learning. Following the information-theoretic framework developed by [25], we show that Prelax provably enjoys better downstream performance.
We denote the random variable of input as X and learn a representation Z through a deterministic encoder Fθ: Z = Fθ(X)2. The representation Z is evaluated for a downstream task T by learning
2We use capitals to denote the random variable, e.g., X, and use lower cases to denote its outcome, e.g., x.
a classifier on top of Z. From an information-theoretic learning perspective, a desirable algorithm should maximize the Mutual Information (MI) between Z and T, i.e., I(Z;T) [6]. Supervised learning on task T can learn representations by directly maximizing I(Z;T). Without access to the labels T, unsupervised learning resorts to maximizing I(Z;S), where S denotes the surrogate signal S designed by each method. Specifically, multi-view learning matches Z with the randomly augmented view, denoted as Sv; while predictive learning uses Z to predict the applied augmentation, denoted as Sa. In Prelax, as we combine both semantics, we actually maximize the MI w.r.t. their joint distribution, i.e., I(Z;Sv,Sa). We denote the representations learned by supervised learning, multi-view learning, predictive learning, and Prelax as Zsup,Zmv,ZPL,ZPrelax, respectively. Theorem 1. Assume that by maximizing the mutual information, each method can retain all information in X about the learning signal S (or T), i.e., I(X;S) = maxZ I(Z;S). Then we have the following inequalities on their task-relevant information I(Z;T):
I(X;T) = I(Zsup;T) ≥ I(ZPrelax;T) ≥ max(I(Zmv;T), I(ZPL;T)). (10)
Theorem 2. Further assume that T is a K-class categorical variable. In general, we have the upper bound ue on the downstream Bayes errors P e := Ez [1−maxt∈T P (T = t|z)],
P̄ e ≤ ue := log 2 + P esup · logK + I(X;T|S). (11) where P̄ e = Th(P e) = min{max{P e, 0}, 1− 1/K} denotes the thresholded Bayes error. Accordingly, we have the following inequalities on the upper bounds for different unsupervised methods,
uesup ≤ uePrelax ≤ min(uemv, uePL) ≤ max(uemv, uePL). (12)
Theorem 1 shows that Prelax extracts more task-relevant information than multi-view and predictive methods, and Theorem 2 further shows that Prelax has a tighter upper bound on the downstream Bayes error. Therefore, Prelax is indeed theoretically superior to previous unsupervised methods by utilizing both pretext-invariant and pretext-aware features. Proofs are in Appendix B.

heading: 4 Practical Implementation
text: In this part, we present three practical variants of Prelax to generalize existing multi-view backbones: 1) one with existing multi-view augmentations (Prelax-std); 2) one with a stronger augmentation, image rotation (Prelax-rot); and 3) one with previous two strategies (Prelax-all).

heading: 4.1 Backbone
text: BYOL [11] and SimSiam [5] are both similarity-based methods and they differ mainly in the design of the target network Fφ. BYOL [11] utilizes momentum update of the target parameters φ from
the online parameters θ, i.e., φ ← τφ + (1 − τ)θ, where τ ∈ [0, 1] is the target decay rate. While SimSiam [5] simply regards the (stopped-gradient) online network as the target network, i.e., φ← sg(θ). We mainly take SimSiam for discussion and our analysis also applies to BYOL. For a given training image x, SimSiam draws two random augmentations (t1, t2) and get two views (x1,x2), respectively. Then, SimSiam maximizes the similarity of their representations with a dual objective, where the two views can both serve as the input and the target to each other,
LSimsiam(x;θ) = ‖Gθ(Fθ(x1))−Fφ(x2)‖22 + ‖Gθ(Fθ(x2))−Fφ(x1)‖22. (13)

heading: 4.2 Prelax-std
text: To begin with, we can directly generalize the baseline method with our Prelax method under existing multi-view augmentation strategies [2, 11]. For the same positive pair (x1,x2), we can calculate their residual vector r12 = Fθ(x1)−Fθ(x2) and use it for the R2S loss (Eq. (4))
LαR2S(x1,x2;θ) = ‖Gθ (Fθ(x1))− αGθ(r12)−Fφ(x2)‖22. (14) We note that there is no difference in using r12 or r21 as the two views are dual. Then, we can adopt the similarity loss in the reverse direction as our similarity constraint loss,
Lsim(x2,x1;θ) = ‖Gθ (Fθ(x2))−Fφ(x1)‖22. (15) At last, we use the residual r12 for the PL loss to predict the augmentation parameters of x1, i.e., t1, because r12 = z1 − z2 directs towards z1. Combining the three losses above, we obtain our Prelaxstd objective,
LPrelax−std(x;θ) = LαR2S(x1,x2;θ) + γLPL(x1,x2, t1;θ) + βLsim(x2,x1;θ). (16)

heading: 4.3 Prelax-rot
text: As mentioned previously, with our residual relaxation we can benefit from stronger augmentations that are harmful for multi-view methods. Here, we focus on the image rotation example and propose the Prelax-rot objective with rotation-aware residual vector. To achieve this, we further generalize existing dual-view methods by incorporating a third rotation view.
Specifically, given two views (x1,x2) generated with existing multi-view augmentations, we additionally draw a random rotation angle a ∈ R = {0◦, 90◦, 180◦, 270◦} and apply it to rotate x1 clockwise, leading to the third view x3. Note that the only difference between x3 and x1 is the rotation semantic a. Therefore, if we substitute x1 with x3 in the similarity loss, we should add a rotation-aware residual r31 = z3 − z1 to bridge the gap. Motivated by this analysis, we propose the Rotation Residual Relaxation Similarity (R3S) loss,
LαR3S(x1:3;θ) = ‖Gθ(Fθ(x3))− αGθ(r31)−Fφ(x2)‖22. (17) which replace Gθ(Fθ(x1)) by its rotation-relaxed version Gθ(Fθ(x3))−αGθ(r31) in the similarity loss. Comparing the R2S loss (Eq. 14) and the R3S loss, we note that the relaxation of the R2S loss accounts for all the semantic shift between x1 and x2, while that of the R3S loss only accounts for the rotation augmentation between x1 and x3. Therefore, we could use the residual r31 to predict the rotation angle a with the following RotPL loss for its rotation-awareness:
LrotPL(x1,x3,a;θ) = CE(Hθ(r31), a). (18) Combining with the similarity constraint, we obtain the Prelax-rot objective:
LPrelax−rot(x;θ) = LαR3S(x1:3;θ) + γLrotPL(x1,x3, a;θ) + βLsim(x2,x1;θ). (19)

heading: 4.4 Prelax-all
text: We have developed Prelax-std that cultivates existing multi-view augmentations and Prelax-rot that incorporates image rotation. Here, we further utilize both existing augmentations and image rotation by combining the two objectives together, denoted as Prelax-all:
LPrelax−all(x;θ) = 1
2 (Lα1R2S(x1,x2;θ) + Lα2R3S(x1:3;θ)) + γ1 2 LPL(x1,x2, t1;θ) + γ2 2 LrotPL(x1,x3, a;θ) + βLsim(x2,x1;θ),
(20)
where α1, α2, γ1, γ2 denotes the coefficients for R2S, R3S, PL and RotPL losses, respectively.

heading: 4.5 Discussions
text: Here we design three practical versions as different implementations of our generic framework of residual relaxation. Among them, Prelax-std focuses on further cultivating existing augmentation strategies, Prelax-rot is to incorporate the stronger (potentially harmful) rotation augmentation, while Prelax-all combines them all. Through the three versions, we demonstrate the wide applicability of Prelax as a generic framework. As for practical users, they could also adapt Prelax to their own application by incorporating specific domain knowledge. In this paper, as we focus on natural images, we take rotation as a motivating example as it is harmful for natural images. Nevertheless, rotation is not necessarily harmful in other domains, e.g., medical images. Instead, random cropping could instead be very harmful for medical images as the important part could lie in the corner. In this scenario, our residual relaxation mechanism could also be used to encode the semantic shift caused by cropping and alleviate its bad effects.

heading: 5 Experiments
text: Datasets. Due to computational constraint, we carry out experiments on a range of medium-sized real-world image datasets, including well known benchmarks like CIFAR-10 [15], CIFAR-100 [15], and two ImageNet variants: Tiny-ImageNet-200 (200 classes with image size resized to 32×32) [27] and ImageNette (10 classes with image size 128×128)3. Backbones. As Prelax is designed to be a generic method for generalizing existing multi-view methods, we implement it on two different multi-view methods, SimSiam [5] and BYOL [11]. Specifically, we notice that SimSiam reported results on CIFAR-10, while the official code of BYOL included results on ImageNette. For a fair comparison, we evaluate SimSiam and its Prelax variant on CIFAR-10, and evaluate BYOL and its Prelax variant on ImageNette. In addition, we evaluate SimSiam and its Prelax variant on two additional datasets CIFAR-100 and Tiny-ImageNet-200, which are more challenging because they include a larger number of classes. For computational efficiency, we adopt the ResNet-18 [13] backbone (adopted in SimSiam [5] for CIFAR-10) to benchmark our experiments. For a comprehensive comparison, we also experiment with larger backbones, like ResNet-34 [13], and the results are included in Appendix C.
Setup. For Prelax-std, we use the same data augmentations as SimSiam [2, 5] (or BYOL [11]), including RandomResizedCrop, RandomHorizontalFlip, ColorJitter, and RandomGrayscale, etc using the PyTorch notations. For Prelax-rot and Prelax-all, we further apply a random image rotation at last of the transformation, where the angles are drawn randomly from {0◦, 90◦, 180◦, 270◦}. To generate targets for the PL objective in Prelax, for each image, we collect the assigned parameters in each random augmentation, such as crop centers, aspect ratios, rotation angles, etc. More details can be found in Appendix A.
Training. For SimSiam and its Prelax variants, we follow the same hyperparameters in [5] on CIFAR-10. Specifically, we use ResNet-18 as the backbone network, followed by a 3-layer projection MLP, whose hidden and output dimension are both 2048. The predictor is a 2-layer MLP whose hidden layer and output dimension are 512 and 2048 respectively. We use SGD for pre-training with batch size 512, learning rate 0.03, momentum 0.9, weight decay 5× 10−4, and cosine decay schedule [18] for 800 epochs. For BYOL and its Prelax variants, we also adopt the ResNet-18 backbone, and the projector and predictor are 2-layer MLPs whose hidden layer and output dimension are 256 and 4096 respectively. Following the default hyper-parameters on ImageNette4, we use LARS optimizer [29] to train 1000 epochs with batch size 256, learning rate 2.0, weight decay 1× 10−6 while excluding the biases and batch normalization parameters from both LARS adaptation and weight decay. For the target network, the exponential moving average parameter τ starts from τbase = 0.996 and increases to 1 during training. As for the Prelax objective, we notice that sometimes, adopting a reverse residual r21 in the R2S loss (Eq. (14)) can bring slightly better results, which could be due to the swapped prediction in SimSiam’s dual objective (Eq. (13)). Besides, a naı̈ve choice of Prelax coefficients already works well: α = 1, β = 1, γ = 0.1 for Prelax-std and Prelax-rot, and α1 = α2 = 1, β = 1, γ1 = γ2 = 0.1 for Prelax-all. More discussion about the hyper-parameters of Prelax can be found in Appendix E.
3https://github.com/fastai/imagenette 4https://github.com/deepmind/deepmind-research/tree/master/byol
Evaluation. After unsupervised training, we evaluate the backbone network by fine-tuning a linear classifier on top of its representation with other model weights held fixed. For SimSiam and its Prelax variants, the linear classifier is trained on labeled data from scratch using SGD with batch size 256, learning rate 30.0, momentum 0.9 for 100 epochs. The learning rate decays by 0.1 at the 60-th and 80-th epochs. For BYOL and its Prelax variants, we use SGD with Nesterov momentum over 80 epochs using batch size 25, learning rate 0.5 and momentum 0.9. Besides the in-domain linear evaluation, we also evaluate its transfer learning performance on the out-of-domain data by learning a linear classifier on the labeled target domain data.

heading: 5.1 Performance on Benchmark Datasets
text: CIFAR-10. In Table 1a, we compare Prelax against previous multi-view methods (SimCLR [2], SimSiam [5], and BYOL [11]) and predictive methods (Rotation [10]) on CIFAR-10. We can see that multi-view methods are indeed better than predictive ones. Nevertheless, predictive learning alone (e.g., Rotation) achieves quite good performance, indicating that pretext-aware features are also very useful. By encoding both pretext-invariant and pretext-aware features, Prelax outperforms previous methods by a large margin, and achieve state-of-the-art performance on CIFAR-10. A comparison of the learning dynamics between SimSiam and Prelax can be found in Appendix F.
ImageNette. Beside the SimSiam backbone, we further apply our Prelax loss to the BYOL framework [11] and evaluate the two methods on the ImageNette dataset. In Table 1b, Prelax also shows a clear advantage over BYOL. Specifically, it improves the ResNet-18 version of BYOL by 0.7%, and even outperforms the ResNet-50 version by 0.3%.
Here, we can see that Prelax yields significant improvement on two different datasets with two different backbone methods. Thus, Prelax could serve as a generic method for improving existing multi-view methods by encoding pretext-aware features into the residual relaxation. For completeness, we also evaluate Prelax on the large scale dataset, ImageNet [7], as well as its transferability to other kinds of downstream tasks, such as object detection and instance segmentation on MS COCO [17]. As shown in Appendix D, Prelax still consistently outperforms the baselines across all tasks.

heading: 5.2 Effectiveness of Prelax Variants
text: For a comprehensive comparison of the three variants of Prelax objectives (Prelax-std, Prelax-rot, and Prelax-all), we conduct controlled experiments on a range of datasets based on the SimSiam backbone. Except CIFAR-10, we also conduct experiments on CIFAR-100 and Tiny-ImageNet200, which are more challenging with more classes of images. For a fair comparison, we use the same training and evaluation protocols across all tasks.
In-domain Linear Evaluation. As shown in Table 2a, our Prelax objectives outperform the multiview objective consistently on all three datasets, where Prelax-all improves SimSiam by 1.6% on CIFAR-10, 3.1% on CIFAR-100, and 1.5% on Tiny-ImageNet-200. Besides, Prelax-std and Prelaxrot are also better than SimSiam in most cases. Thus, the pretext-aware residuals in Prelax indeed help encode more useful semantics.
Out-of-domain Linear Evaluation. Besides the in-domain linear evaluation, we also transfer the representations to a target domain. In the out-of-domain linear evaluation results shown in Table 2b, the Prelax objectives still have a clear advantage over the multi-view objective (SimSiam), while sometimes Prelax-std and Prelax-rot enjoy better transferred accuracy than Prelax-all.

heading: 5.3 Empirical Understandings of Prelax
text: Comparison against Alternative Options. In Table 3a, we compare Prelax against several other relaxation options. “SimSiam + margin” refers to the margin loss discussed in Eq. (9), which uses a scalar η to relax the exact alignment in multi-view methods. Here we tune the margin η = 0.5 with the best performance. Nevertheless, it has no clean advantage over SimSiam. Then, we try several options for incorporating a strong augmentation and image rotation: 1) Rotation is the PL baseline by predicting rotation angles [10], which is inferior to multi-view methods (SimSiam). 2) “SimSiam + rotation aug.” directly applies a random rotation augmentation to each view, and learn with the SimSiam loss. However, it leads to lower accuracy, showing that the image rotation, as a strong augmentation, will hurt the performance of multi-view methods. 3) “SimSiam + Rotation” directly combines the SimSiam loss and the Rotation loss for training, which is still ineffective. 4) Our Prelax shows a significant improvement over SimSiam and other variants, showing that the residual alignment is an effective mechanism for utilizing strong augmentations like rotation.
Ablation Study. We perform ablation study of each component of the Prelax objectives on CIFAR10. From Table 3b, we notice that simply adding the PL loss alone cannot improve over SimSiam consistently, for example, Sim + RotPL causes 0.7 point drop in test accuracy. While with the help of our residual relaxation, we can improve over the baselines significantly and consistently, for example, Prelax-rot (Sim + RotPL + R3S) brings 0.6 point improvement on test accuracy. Besides, we can see that the PL loss is necessary by making the residual pretext-aware, without which the performance drops a lot, and the similarity constraint (Sim loss) is also important by avoiding bad cases when augmented images drift far from the anchor image. Therefore, the ablation study shows the residual relaxation loss, similarity loss, and PL loss all matter in our Prelax objectives.

heading: 5.4 Qualitative Analysis
text: Representation Visualization. To provide an intuitive understanding of the learned representations, we visualize them with t-SNE [26] on Figure 3a. We can see that in general, our Prelax can learn well-separated clusters of representations corresponding to the ground-truth image classes.
Image Retrieval. In Figure 3b, we evaluate Prelax on an image retrieval task. Given a random query image (not cherry-picked), the top-15 most similar images in representation space are retrieved, and the query itself is shown in the first column. We can see that although the unsupervised training with Prelax has no access to labels, the retrieved nearest images of Prelax are all correctly from the same class and semantically consistent with the query.

heading: 6 Conclusion
text: In this paper, we proposed a generic method, Prelax (Pretext-aware Residual Relaxation), to account for the (possibly large) semantic shift caused by image augmentations. With pretext-aware learning of the residual relaxation, our method generalizes existing multi-view learning by encoding both pretext-aware and pretext-invariant representations. Experiments show that our Prelax has outperformed existing multi-view methods significantly on a variety of benchmark datasets.

heading: Acknowledgement
text: Yisen Wang is partially supported by the National Natural Science Foundation of China under Grant 62006153, and Project 2020BD006 supported by PKU-Baidu Fund. Jiansheng Yang is supported by the National Science Foundation of China under Grant No. 11961141007. Zhouchen Lin was supported by the NSF China (No.s 61625301 and 61731018), NSFC Tianyuan Fund for Mathematics (No. 12026606) and Project 2020BD006 supported by PKU-Baidu Fund.

Review summaries:
The paper proposes a residual relaxation method to encode the semantic shift for the multi-view self-supervised learning. The method relaxes the exact alignment for the latent vector representation from different views. The experimental results show the method can improve the performance of the data augmentation.

The paper focus on the representation learning using a relaxed alignment based on the fact that some strong data augmentations may hurt the performance. The authors introduced an adaptive residual vector and designed several novel objectives to make the residual vector useful and meaningful. Experiments shows a significant improvement over various backbones. 

The paper proposes a novel Pretext-aware Residual Relaxation (Prelax) approach to self-supervised learning that does not require exact alignment of the two different views, which can be harmful for views created from some types of augmentations, but introduces a residual vector that encodes the semantic shift caused by the augmentation. To ensure pretext-awareness, a loss is introduced to predict the pretext task from the residual vector. A theoretical analysis is provided to show Prelax's advantage over unsupervised approaches that either only consider multi-view learning or only predictive learning. Experiments indicate that adding Prelax to common self-supervised methods such as SimSiam and BYOL improves performance.