Residual Relaxation for Multi-view Representation Learning

1 Introduction . Without access to labels , self-supervised learning relies on surrogate objectives to extract meaningful representations from unlabeled data , and the chosen surrogate objectives largely determine the quality and property of the learned representations [ 24 , 19 ] . Recently , multi-view methods have become a dominant approach for self-supervised representation learning that achieves impressive downstream performance , and many modern variants have been proposed [ 22 , 14 , 1 , 23 , 2 , 12 , 3 , 4 , 11 , 5 ] . Nevertheless , most multi-view methods can be abstracted and summarized as the following pipeline : for each input x , we apply several ( typically two ) random augmentations to it , and learn to align these different “ views ” ( x1 , x2 , . . . ) of x by minimizing their distance in the representation space . In multi-view methods , the pretext , i.e. , image augmentation , has a large effect on the final performance . Typical choices include image re-scaling , cropping , color jitters , etc [ 2 ] . However , we find that some augmentations , for example , image rotation , is seldom utilized in state-of-the-art multiview methods . Among these augmentations , Figure 1a shows that rotation causes severe accuracy drop in a standard supervised model . Actually , image rotation is a stronger augmentation that largely affects the image semantics , and as a result , enforcing exact alignment of two different rotation angles could degrade the representation ability in existing multi-view methods . Nevertheless , it does not mean that strong augmentations can not provide useful semantics for representation learning . In fact , rotation is known as an effective signal for predictive learning [ 10 , 30 , 21 ] . Differently , predictive methods learn representations by predicting the pretext ( e.g. , rotation angle ) from the cor- ∗Corresponding author : Yisen Wang ( yisen.wang @ pku.edu.cn ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . responding view . In this way , the model is encouraged to encode pretext-aware image semantics , which also yields good representations . To summarize , strong augmentations like rotation carry meaningful semantics , while being harmful for existing multi-view methods due to large semantic shift . To address this dilemma , in this paper , we propose a generic approach that generalizes multi-view methods to cultivating stronger augmentations . Drawing inspirations from the soft-margin SVM , we propose residual alignment , which relaxes the exact alignment in multi-view methods by incorporating a residual vector between two views . Besides , we develop a predictive loss for the residual to ensure that it encodes the semantic shift between views ( e.g. , image rotation ) . We name this technique as Pretext-aware REsidual ReLAXation ( Prelax ) , and an illustration is shown in Figure 1b . Prelax serves as a generalized multi-view method that is adaptive to large semantic shift and combines image semantics extracted from both pretext-invariant and pretext-aware methods . We summarize our contributions as follows : • We propose a generic technique , Pretext-aware Residual Relaxation ( Prelax ) , that generalizes multi-view representation learning to benefit from stronger image augmentations . • Prelax not only extracts pretext-invariant features as in multi-view methods , but also encodes pretext-aware features into the pretext-aware residuals . Thus , it can serve as a unified approach to bridge the two existing methodologies for representation learning . • Experiments show that Prelax can bring significant improvement over both multi-view and predictive methods on a wide range of benchmark datasets . 2 Related Work . Multi-view Learning . Although multi-view learning could refer to a wider literature [ 16 ] , here we restrict our discussions to the context of Self-Supervised Learning ( SSL ) , where multi-view methods learn representations by aligning multiple views of the same image generated through random data augmentation [ 1 ] . There are two kinds of methods to keep the representations well separated : contrastive methods , which achieve this by maximizing the difference between different samples [ 2 , 12 ] , and similarity-based methods , which prevent representation collapse via implicit mechanisms like predictor and gradient stopping [ 11 , 5 ] . Although having lots of modern variants , multi-view methods share the same methodology , that is to extract features that are invariant to the predefined augmentations , i.e. , pretext-invariant features [ 20 ] . Predictive Learning . Another thread of methods is to learn representations by predicting selfgenerated surrogate labels . Specifically , it applies a transformation ( e.g. , image rotation ) to the input image and requires the learner to predict properties of the transformation ( e.g. , the rotation angle ) from the transformed images . As a result , the extracted image representations are encouraged to become aware of the applied pretext ( e.g. , image rotation ) . Thus , we also refer to them as pretext- aware methods . The pretext tasks can be various , to name a few , Rotation [ 10 ] , Jigsaw [ 21 ] , Relative Path Location [ 8 ] , Colorization [ 30 ] . Generalized Multi-view Learning . Although there are plenty of works on each branch , how to bridge the two methodologies remains under-explored . Prior to our work , there are only a few works on this direction . Some directly combine AMDIM ( pretext-invariant ) [ 1 ] and Rotation ( pretextaware ) [ 10 ] objectives [ 9 ] . However , a direct combination of the two contradictory objectives may harm the final representation . LooC [ 28 ] proposes to separate the embedding space to several parts , where each subspace learns local invariance w.r.t . a specific augmentation . But this is achieved at the cost of limiting the representation flexibility of each pretext to the predefined subspace . Different from them , our proposed Prelax provides a more general solution by allowing an adaptive residual vector to encode the semantic shift . In this way , both kinds of features are encoded in the same representation space . 3 The Proposed Pretext-aware Residual Relaxation ( Prelax ) Method . 3.1 Preliminary . Problem Formulation . Given unlabeled data { xi } , unsupervised representation learning aims to learn an encoder network Fθ that extracts meaningful low-dimensional representations z ∈ Rdz from high-dimensional input images x ∈ Rdx . The learned representation is typically evaluated on a downstream classification task by learning a linear classifier with labeled data { xi , yi } . Multi-view Representation Learning . For an input image x ∈ Rdx , we can generate a different view by data augmentation , x′ = t ( x ) , where t ∈ T refers to a randomly drawn augmentation operator from the pretext set T . Then , the transformed input x′ and the original input x are passed into an online network Fθ and a target network Fφ , respectively . Optionally , the output of the online network is further processed by an MLP predictor network Gθ , to match the output of the target network . As two different views of the same image ( i.e. , positive samples ) , x and x′ should have similar representations , so we align their representations with the following similarity loss , Lsim ( x′ , x ; θ ) = ‖Gθ ( Fθ ( x′ ) ) −Fφ ( x ) ‖22 . ( 1 ) The representations , e.g. , z = Fθ ( x ) , are typically projected to a unit spherical ball before calculating the distance ( z/‖z‖2 ) , which makes the ` 2 distance equivalent to the cosine similarity [ 2 ] . Remark . Aside from the similarity loss between positive samples , contrastive methods [ 22 , 14 , 23 , 2 ] further encourage representation uniformity with an additional regularization minimizing the similarity between input and an independently drawn negative sample . Nevertheless , some recent works find that the similarity loss alone already suffices [ 11 , 5 ] . In this paper , we mainly focus on improving the alignment between positive samples in the similarity loss . It can also be easily extended to contrastive methods by considering the dissimilarity regularization . 3.2 Objective Formulation . As we have noticed , the augmentation sometimes may bring a certain amount of semantic shift . Thus , enforcing exact alignment of different views may hurt the representation quality , particularly when the data augmentation is too strong for the positive pairs to be matched exactly . Therefore , we need to relax the exact alignment in Eq . ( 1 ) to account for the semantic shift brought by the data augmentation . Residual Relaxed Similarity Loss . Although the representations may not align exactly , i.e. , z′ 6= z , however , the representation identity will always hold : z′ − ( z′ − z ) = z , where z′ − z represents the shifted semantics by augmentation . This makes this identity a proper candidate for multi-view alignment under various augmentations as long as the shifted semantic is taken into consideration . Specifically , we replace the exact alignment ( denoted as→← ) in the similarity loss ( Eq . ( 1 ) ) by the proposed identity alignment , i.e. , Gθ ( z′θ ) →← zφ ⇒ Gθ ( z′θ ) − Gθ ( r ) →← zφ , ( 2 ) where we include a residual vector r ∆= z′θ − zθ = Fθ ( x′ ) − Fθ ( x ) to represent the difference on the representations . To further enable a better tradeoff between the exact and identity alignments , we have the following residual alignment : Gθ ( z′θ ) − αGθ ( r ) →← zφ , ( 3 ) where α ∈ [ 0 , 1 ] is the interpolation parameter . When α = 0 , we recover the exact alignment ; when α = 1 , we recover the identity alignment . We name the corresponding learning objective as the Residual Relaxed Similarity ( R2S ) loss , which minimizes the squared ` 2 distance among two sides : LαR2S ( x′ , x ; θ ) = ‖Gθ ( Fθ ( x′ ) ) − αGθ ( r ) −Fφ ( x ) ‖22 . ( 4 ) Predictive Learning ( PL ) Loss . To ensure the relaxation works as expected , the residual r should encode the semantic shift caused by the augmentation , i.e. , the pretext . Inspired by predictive learning [ 10 ] , we utilize the residual to predict the corresponding augmentation for its pretext-awareness . In practice , the assigned parameters for the random augmentation t can be generally divided into the discrete categorical variables td ( e.g. , flipping or not , graying or not ) , and the continuous variables tc ( e.g. , scale , ratio , jittered brightness ) . Thus , we learn a PL predictor Hθ to predict ( td , tc ) with cross entropy loss ( CE ) and mean square error loss ( MSE ) , respectively : LPL ( x′ , x , t ; θ ) = CE ( Hdθ ( r ) , td ) + ‖Hcθ ( r ) − tc‖22 . ( 5 ) Constraint on the Similarity . Different from the exact alignment , the residual vector can be unbounded , i.e. , the difference between views can be arbitrarily large . This is not reasonable as the two views indeed share many common semantics . Therefore , we should utilize this prior knowledge to prevent the bad cases under residual similarity and add the following constraint Lsim = ‖Gθ ( Fθ ( x′ ) ) −Fφ ( x ) ‖22 ≤ ε , ( 6 ) where ε denotes the maximal degree of mismatch allowed between positive samples . The Overall Objective of Prelax . By combining the three components above , we can reliably encode the semantic shift between augmentations while ensuring a good alignment between views : min θ LαR2S ( x′ , x ; θ ) + γLPL ( x′ , x ; θ ) , s.t . ‖Gθ ( Fθ ( x′ ) ) −Fφ ( x ) ‖22 ≤ ε . ( 7 ) For simplicity , we transform it into a Lagrangian objective with a fixed multiplier β ≥ 0 , and obtain the overall Pretext-aware REsidual ReLAXation ( Prelax ) objective , LαR2S ( x′ , x ; θ ) + γLPL ( x′ , x ; θ ) + βLsim ( x′ , x ; θ ) , ( 8 ) where α tradeoffs between the exact and identity alignments , γ adjusts the amount of pretextawareness of the residual , and β controlls the degree of similarity between positive pairs . An illustrative diagram of the Prelax objective is shown in Figure 1 . Discussions . In fact , there are other alternatives to relax the exact alignment . For example , we can utilize a margin loss Lmargin ( x′ , x ; θ ) = max ( ‖Gθ ( Fθ ( x′ ) ) −Fφ ( x ) ‖22 − η , 0 ) , ( 9 ) where η > 0 is a threshold for the mismatch tolerance . However , it has two main drawbacks : 1 ) as each image and augmentation have different semantics , it is hard to choose a universal threshold for all images ; and 2 ) the representation keeps shifting along with the training progress , making it even harder to maintain a proper threshold dynamically . Thus , a good relaxation should be adaptive to the training progress and the aligning of different views . While our Prelax adopts pretext-aware residual vector , which is learnable , flexible , and semantically meaningful .

Summary: The paper proposes a novel Pretext-aware Residual Relaxation (Prelax) approach to self-supervised learning that does not require exact alignment of the two different views, which can be harmful for views created from some types of augmentations, but introduces a residual vector that encodes the semantic shift caused by the augmentation. To ensure pretext-awareness, a loss is introduced to predict the pretext task from the residual vector. A theoretical analysis is provided to show Prelax's advantage over unsupervised approaches that either only consider multi-view learning or only predictive learning. Experiments indicate that adding Prelax to common self-supervised methods such as SimSiam and BYOL improves performance.