Residual Relaxation for Multi-view Representation Learning

1 Introduction . Without access to labels , self-supervised learning relies on surrogate objectives to extract meaningful representations from unlabeled data , and the chosen surrogate objectives largely determine the quality and property of the learned representations [ 24 , 19 ] . Recently , multi-view methods have become a dominant approach for self-supervised representation learning that achieves impressive downstream performance , and many modern variants have been proposed [ 22 , 14 , 1 , 23 , 2 , 12 , 3 , 4 , 11 , 5 ] . Nevertheless , most multi-view methods can be abstracted and summarized as the following pipeline : for each input x , we apply several ( typically two ) random augmentations to it , and learn to align these different “ views ” ( x1 , x2 , . . . ) of x by minimizing their distance in the representation space . In multi-view methods , the pretext , i.e. , image augmentation , has a large effect on the final performance . Typical choices include image re-scaling , cropping , color jitters , etc [ 2 ] . However , we find that some augmentations , for example , image rotation , is seldom utilized in state-of-the-art multiview methods . Among these augmentations , Figure 1a shows that rotation causes severe accuracy drop in a standard supervised model . Actually , image rotation is a stronger augmentation that largely affects the image semantics , and as a result , enforcing exact alignment of two different rotation angles could degrade the representation ability in existing multi-view methods . Nevertheless , it does not mean that strong augmentations can not provide useful semantics for representation learning . In fact , rotation is known as an effective signal for predictive learning [ 10 , 30 , 21 ] . Differently , predictive methods learn representations by predicting the pretext ( e.g. , rotation angle ) from the cor- ∗Corresponding author : Yisen Wang ( yisen.wang @ pku.edu.cn ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . responding view . In this way , the model is encouraged to encode pretext-aware image semantics , which also yields good representations . To summarize , strong augmentations like rotation carry meaningful semantics , while being harmful for existing multi-view methods due to large semantic shift . To address this dilemma , in this paper , we propose a generic approach that generalizes multi-view methods to cultivating stronger augmentations . Drawing inspirations from the soft-margin SVM , we propose residual alignment , which relaxes the exact alignment in multi-view methods by incorporating a residual vector between two views . Besides , we develop a predictive loss for the residual to ensure that it encodes the semantic shift between views ( e.g. , image rotation ) . We name this technique as Pretext-aware REsidual ReLAXation ( Prelax ) , and an illustration is shown in Figure 1b . Prelax serves as a generalized multi-view method that is adaptive to large semantic shift and combines image semantics extracted from both pretext-invariant and pretext-aware methods . We summarize our contributions as follows : • We propose a generic technique , Pretext-aware Residual Relaxation ( Prelax ) , that generalizes multi-view representation learning to benefit from stronger image augmentations . • Prelax not only extracts pretext-invariant features as in multi-view methods , but also encodes pretext-aware features into the pretext-aware residuals . Thus , it can serve as a unified approach to bridge the two existing methodologies for representation learning . • Experiments show that Prelax can bring significant improvement over both multi-view and predictive methods on a wide range of benchmark datasets . 2 Related Work . Multi-view Learning . Although multi-view learning could refer to a wider literature [ 16 ] , here we restrict our discussions to the context of Self-Supervised Learning ( SSL ) , where multi-view methods learn representations by aligning multiple views of the same image generated through random data augmentation [ 1 ] . There are two kinds of methods to keep the representations well separated : contrastive methods , which achieve this by maximizing the difference between different samples [ 2 , 12 ] , and similarity-based methods , which prevent representation collapse via implicit mechanisms like predictor and gradient stopping [ 11 , 5 ] . Although having lots of modern variants , multi-view methods share the same methodology , that is to extract features that are invariant to the predefined augmentations , i.e. , pretext-invariant features [ 20 ] . Predictive Learning . Another thread of methods is to learn representations by predicting selfgenerated surrogate labels . Specifically , it applies a transformation ( e.g. , image rotation ) to the input image and requires the learner to predict properties of the transformation ( e.g. , the rotation angle ) from the transformed images . As a result , the extracted image representations are encouraged to become aware of the applied pretext ( e.g. , image rotation ) . Thus , we also refer to them as pretext- aware methods . The pretext tasks can be various , to name a few , Rotation [ 10 ] , Jigsaw [ 21 ] , Relative Path Location [ 8 ] , Colorization [ 30 ] . Generalized Multi-view Learning . Although there are plenty of works on each branch , how to bridge the two methodologies remains under-explored . Prior to our work , there are only a few works on this direction . Some directly combine AMDIM ( pretext-invariant ) [ 1 ] and Rotation ( pretextaware ) [ 10 ] objectives [ 9 ] . However , a direct combination of the two contradictory objectives may harm the final representation . LooC [ 28 ] proposes to separate the embedding space to several parts , where each subspace learns local invariance w.r.t . a specific augmentation . But this is achieved at the cost of limiting the representation flexibility of each pretext to the predefined subspace . Different from them , our proposed Prelax provides a more general solution by allowing an adaptive residual vector to encode the semantic shift . In this way , both kinds of features are encoded in the same representation space . 3 The Proposed Pretext-aware Residual Relaxation ( Prelax ) Method . 3.1 Preliminary . Problem Formulation . Given unlabeled data { xi } , unsupervised representation learning aims to learn an encoder network Fθ that extracts meaningful low-dimensional representations z ∈ Rdz from high-dimensional input images x ∈ Rdx . The learned representation is typically evaluated on a downstream classification task by learning a linear classifier with labeled data { xi , yi } . Multi-view Representation Learning . For an input image x ∈ Rdx , we can generate a different view by data augmentation , x′ = t ( x ) , where t ∈ T refers to a randomly drawn augmentation operator from the pretext set T . Then , the transformed input x′ and the original input x are passed into an online network Fθ and a target network Fφ , respectively . Optionally , the output of the online network is further processed by an MLP predictor network Gθ , to match the output of the target network . As two different views of the same image ( i.e. , positive samples ) , x and x′ should have similar representations , so we align their representations with the following similarity loss , Lsim ( x′ , x ; θ ) = ‖Gθ ( Fθ ( x′ ) ) −Fφ ( x ) ‖22 . ( 1 ) The representations , e.g. , z = Fθ ( x ) , are typically projected to a unit spherical ball before calculating the distance ( z/‖z‖2 ) , which makes the ` 2 distance equivalent to the cosine similarity [ 2 ] . Remark . Aside from the similarity loss between positive samples , contrastive methods [ 22 , 14 , 23 , 2 ] further encourage representation uniformity with an additional regularization minimizing the similarity between input and an independently drawn negative sample . Nevertheless , some recent works find that the similarity loss alone already suffices [ 11 , 5 ] . In this paper , we mainly focus on improving the alignment between positive samples in the similarity loss . It can also be easily extended to contrastive methods by considering the dissimilarity regularization . 3.2 Objective Formulation . As we have noticed , the augmentation sometimes may bring a certain amount of semantic shift . Thus , enforcing exact alignment of different views may hurt the representation quality , particularly when the data augmentation is too strong for the positive pairs to be matched exactly . Therefore , we need to relax the exact alignment in Eq . ( 1 ) to account for the semantic shift brought by the data augmentation . Residual Relaxed Similarity Loss . Although the representations may not align exactly , i.e. , z′ 6= z , however , the representation identity will always hold : z′ − ( z′ − z ) = z , where z′ − z represents the shifted semantics by augmentation . This makes this identity a proper candidate for multi-view alignment under various augmentations as long as the shifted semantic is taken into consideration . Specifically , we replace the exact alignment ( denoted as→← ) in the similarity loss ( Eq . ( 1 ) ) by the proposed identity alignment , i.e. , Gθ ( z′θ ) →← zφ ⇒ Gθ ( z′θ ) − Gθ ( r ) →← zφ , ( 2 ) where we include a residual vector r ∆= z′θ − zθ = Fθ ( x′ ) − Fθ ( x ) to represent the difference on the representations . To further enable a better tradeoff between the exact and identity alignments , we have the following residual alignment : Gθ ( z′θ ) − αGθ ( r ) →← zφ , ( 3 ) where α ∈ [ 0 , 1 ] is the interpolation parameter . When α = 0 , we recover the exact alignment ; when α = 1 , we recover the identity alignment . We name the corresponding learning objective as the Residual Relaxed Similarity ( R2S ) loss , which minimizes the squared ` 2 distance among two sides : LαR2S ( x′ , x ; θ ) = ‖Gθ ( Fθ ( x′ ) ) − αGθ ( r ) −Fφ ( x ) ‖22 . ( 4 ) Predictive Learning ( PL ) Loss . To ensure the relaxation works as expected , the residual r should encode the semantic shift caused by the augmentation , i.e. , the pretext . Inspired by predictive learning [ 10 ] , we utilize the residual to predict the corresponding augmentation for its pretext-awareness . In practice , the assigned parameters for the random augmentation t can be generally divided into the discrete categorical variables td ( e.g. , flipping or not , graying or not ) , and the continuous variables tc ( e.g. , scale , ratio , jittered brightness ) . Thus , we learn a PL predictor Hθ to predict ( td , tc ) with cross entropy loss ( CE ) and mean square error loss ( MSE ) , respectively : LPL ( x′ , x , t ; θ ) = CE ( Hdθ ( r ) , td ) + ‖Hcθ ( r ) − tc‖22 . ( 5 ) Constraint on the Similarity . Different from the exact alignment , the residual vector can be unbounded , i.e. , the difference between views can be arbitrarily large . This is not reasonable as the two views indeed share many common semantics . Therefore , we should utilize this prior knowledge to prevent the bad cases under residual similarity and add the following constraint Lsim = ‖Gθ ( Fθ ( x′ ) ) −Fφ ( x ) ‖22 ≤ ε , ( 6 ) where ε denotes the maximal degree of mismatch allowed between positive samples . The Overall Objective of Prelax . By combining the three components above , we can reliably encode the semantic shift between augmentations while ensuring a good alignment between views : min θ LαR2S ( x′ , x ; θ ) + γLPL ( x′ , x ; θ ) , s.t . ‖Gθ ( Fθ ( x′ ) ) −Fφ ( x ) ‖22 ≤ ε . ( 7 ) For simplicity , we transform it into a Lagrangian objective with a fixed multiplier β ≥ 0 , and obtain the overall Pretext-aware REsidual ReLAXation ( Prelax ) objective , LαR2S ( x′ , x ; θ ) + γLPL ( x′ , x ; θ ) + βLsim ( x′ , x ; θ ) , ( 8 ) where α tradeoffs between the exact and identity alignments , γ adjusts the amount of pretextawareness of the residual , and β controlls the degree of similarity between positive pairs . An illustrative diagram of the Prelax objective is shown in Figure 1 . Discussions . In fact , there are other alternatives to relax the exact alignment . For example , we can utilize a margin loss Lmargin ( x′ , x ; θ ) = max ( ‖Gθ ( Fθ ( x′ ) ) −Fφ ( x ) ‖22 − η , 0 ) , ( 9 ) where η > 0 is a threshold for the mismatch tolerance . However , it has two main drawbacks : 1 ) as each image and augmentation have different semantics , it is hard to choose a universal threshold for all images ; and 2 ) the representation keeps shifting along with the training progress , making it even harder to maintain a proper threshold dynamically . Thus , a good relaxation should be adaptive to the training progress and the aligning of different views . While our Prelax adopts pretext-aware residual vector , which is learnable , flexible , and semantically meaningful .

Summary: The paper focus on the representation learning using a relaxed alignment based on the fact that some strong data augmentations may hurt the performance. The authors introduced an adaptive residual vector and designed several novel objectives to make the residual vector useful and meaningful. Experiments shows a significant improvement over various backbones.