Self-Contrastive Learning

1 INTRODUCTION . Recent studies have studied the success of deep neural networks by investigating how neural networks can encode representations with rich information ( Tishby & Zaslavsky , 2015 ; Shwartz-Ziv & Tishby , 2017 ; Hjelm et al. , 2018 ; Saxe et al. , 2019a ) . Among the various approaches suggested , contrastive loss functions , designed to maximize the lower bound of mutual information ( MI ) between the target and context , have achieved considerable success in self-supervised representation learning first ( Gutmann & Hyvärinen , 2010 ; Oord et al. , 2018 ) and supervised learning recently ( Khosla et al. , 2020 ; Gunel et al. , 2020 ; Wang et al. , 2021 ) . The main objective of the contrastive loss functions in supervision is to make representations from the same class closer and representations from different classes farther . To this end , they define positive samples , i.e. , augmented samples from the same image or ( augmented ) images sharing the same class label , and negative samples , i.e. , all other samples , for every data batch . Contrasting two random augmented samples , often referred to as a multi-viewed batch , has shown impressive results in representation learning ( Chen et al. , 2020 ; He et al. , 2020 ; Grill et al. , 2020 ; Caron et al. , 2020 ; Chen & He , 2020 ; Khosla et al. , 2020 ) , yet a multi-viewed batch causes the following issues . ( 1 ) A multi-viewed batch doubles the batch size , which is a huge burden on memory and computation . ( 2 ) Oracle needs to carefully choose the augmentation policies ( Chen et al. , 2020 ; Tian et al. , 2020 ; Caron et al. , 2020 ; Kim et al. , 2020 ) . ( 3 ) A multi-viewed contrastive task is domain-specific , because data-level augmentation requires specific domain knowledge such as image cropping and flipping ( Verma et al. , 2021 ) . In spite of these concerns , a supervised contrastive learning framework ( Khosla et al. , 2020 ) still relies on multi-views , although they are not necessarily needed with the help of usable label information . In this work , we propose a novel contrastive learning framework in supervision , called SelfContrastive ( SelfCon ) learning , that does not require additional augmented samples . Instead , SelfCon uses a multi-exit framework ( Teerapittayanon et al. , 2016 ; Zhang et al. , 2019a ; b ) where sub-networks produce multiple outputs for the same input and self-contrasts within multiple outputs from the different levels of a single network . For training , SelfCon learning can use the multi-viewed batch as well . However , the multi-exit framework already generates positive pairs from a single image , which can replace the augmentation based multi-view . In Figure 1 , we compare SelfCon learning and Supervised Contrastive ( SupCon , Khosla et al . ( 2020 ) ) learning . SelfCon learning improves the classification performance of the encoder network , owing to ( 1 ) the increase of the lower bound of label-conditional MI between the intermediate and the last features , and ( 2 ) the regularization effect from the single-view and sub-network . We theoretically demonstrate that SelfCon loss is the lower bound of label-conditional MI . Furthermore , unlike SupCon loss , it encourages the encoder to learn the label information that only intermediate features can explain . Second , we empirically show that SelfCon learning reduces the generalization error with the single-view and sub-network . The former prevents the encoder from overfitting to each instance , and the latter regularizes the intermediate feature to be similar to the last feature . The contributions of our paper can be summarized as follows : [ S3 ] We propose Self-Contrastive learning , a novel supervised contrastive framework between the multiple features from the different levels of a single network . [ S4 ] SelfCon loss guarantees the lower bound of label-conditional MI between the intermediate and the last features , and leads to increasing interaction information between the features and the label . [ S5.1 , S5.2 ] SelfCon learning achieved higher classification accuracy for various benchmarks and architectures than cross-entropy and SupCon loss . Our empirical study of MI estimation provides evidence for superior performance . [ S5.3 , S5.4 ] We investigate the advantages of single-viewed batch in terms of the generalization error and computation resources . We also identify that SelfCon learning benefits from the sub-network owing to the regularization effect , vanishing gradient , and ensemble prediction . 2 RELATED WORKS . 2.1 CONTRASTIVE LEARNING . As the cost of data labeling increases exponentially , there is an increased need for acquiring representation under unsupervised scenarios . After Oord et al . ( 2018 ) proposed an InfoNCE loss ( also called a contrastive loss ) , contrastive learning-based algorithms showed a remarkable improvement in performance ( Chen et al. , 2020 ; He et al. , 2020 ; Grill et al. , 2020 ; Caron et al. , 2020 ; Chen & He , 2020 ) . Most previous works use the structure of a Siamese network , which is a weight-sharing network applied on two or more inputs , with negative pairs ( SimCLR ( Chen et al. , 2020 ) , MoCo ( He et al. , 2020 ) ) , momentum encoders ( MoCo ( He et al. , 2020 ) , BYOL ( Grill et al. , 2020 ) ) , online clustering ( SwAV ( Caron et al. , 2020 ) ) , or a stop-gradient operation ( SimSiam ( Chen & He , 2020 ) ) . While the softmax form is frequently used for the contrastive loss ( Chen et al. , 2020 ; He et al. , 2020 ) , recent state-of-the-art algorithms utilize an MSE loss ( Grill et al. , 2020 ; Chen & He , 2020 ) , or a cross-entropy loss ( Caron et al. , 2020 ; 2021 ) . Khosla et al . ( 2020 ) , inspired by the success in self-supervised learning , propose a label-based contrastive loss in supervision , named SupCon loss . Supervised contrastive learning has also been extended to semantic segmentation ( Wang et al. , 2021 ) and language tasks ( Gunel et al. , 2020 ) . While the SupCon loss utilizes the output features from two randomly augmented images , we also contrast the features from different network paths by introducing the multi-exit framework ( Teerapittayanon et al. , 2016 ; Zhang et al. , 2019a ; b ) . We further propose a novel loss function that we can apply on the single-viewed batch . 2.2 MUTUAL INFORMATION . Mutual information is a measure to quantify the information held in a random variable about the other variable , and it has been used as a powerful tool to open the black box of deep neural networks ( Tishby & Zaslavsky , 2015 ; Shwartz-Ziv & Tishby , 2017 ; Saxe et al. , 2019a ) . As it is difficult to compute the MI exactly ( Paninski , 2003 ) , several works have proposed variational MI estimators based on neural networks , e.g. , InfoNCE ( Oord et al. , 2018 ) , MINE ( Belghazi et al. , 2018 ) , NWJ ( Nguyen et al. , 2010 ) , ML-CPC ( Song & Ermon , 2020 ) , and SMILE ( Song & Ermon , 2019 ) . Recently , MI estimator-based objectives have been proposed to improve the performance of contrastive learning ( Hjelm et al. , 2018 ; Bachman et al. , 2019 ; Song & Ermon , 2020 ; Wu et al. , 2020b ) and knowledge distillation ( Tian et al. , 2019b ; Ahn et al. , 2019 ) . Among previous approaches , we are highly motivated by those that aim to increase the MI between the intermediate and the last features . DIM ( Hjelm et al. , 2018 ) and AMDIM ( Bachman et al. , 2019 ) propose novel contrastive losses between the global features and local features ( i.e. , all pixels or patches of the intermediate features ) , while we contrast the refined local features via sub-networks . VID ( Ahn et al. , 2019 ) makes the student learn the distribution of the activations in the auxiliary teacher ’ s intermediate features . Our method differs from VID in that we self-distill within the network itself . Zhang et al . ( 2019a ; b ) share a similar idea with the aforementioned works , but they implicitly increase MI using Kullback–Leibler ( KL ) divergence loss . 3 SELF-CONTRASTIVE LEARNING . We propose a new supervised contrastive loss that maximizes the similarity of the outputs from different network paths by introducing the multi-exit framework . We define the encoder structure with F as a backbone network and G as a sub-network that shares the backbone ’ s parameters up to some intermediate layer . T denotes those sharing layers that produce the intermediate feature . The sub-network has additional non-sharing parameters attached after T . Note that F and G include the projection head on the encoder . We highlight the positive and negative pairs with respect to an anchor sample , following Figure 1 . SupCon loss To mitigate the weaknesses of cross-entropy , such as the reduced generalization performance and the possibility of poor margins , Khosla et al . ( 2020 ) proposed a supervised version of the contrastive loss that defines the positive pairs as every sample with the same ground-truth label . We reformulate the SupCon loss function as follows : Lsup = − ∑ i , p1 log exp ( F ( xi ) • F ( xp1 ) /τ ) ∑ p2 exp ( F ( xi ) • F ( xp2 ) /τ ) + ∑ n exp ( F ( xi ) • F ( xn ) /τ ) ( 1 ) i ∈ I ≡ { 1 , . . . , 2B } p∗ ∈ Pi∗ ≡ { p ∈ I \ { i } |yp = yi } n ∈ Ni ≡ { n ∈ I|yn 6= yi } where • denotes the inner product , B is the batch size , and τ is the temperature to soften or harden the softmax value . I denotes a set of indices of the multi-viewed batch that concatenates the original B images and the augmented ones , i.e. , xB+i is an augmented pair of xi . Pi∗ and Ni are a set of positive and negative pair indices with respect to an anchor i , respectively . Note that Eq . 1 is the same as NT-Xent loss ( Chen et al. , 2020 ) when Pi∗ ≡ { ( i+B ) mod 2B } . We dropped the dividing term of sums for brevity . SelfCon loss Besides minimizing the above loss , we aim to maximize the similarity between the outputs from the backbone and the sub-network . To this end , we define SelfCon loss , which forms a self-contrastive task for every output including the features from the sub-network . In Section 4 , we clarify the connection of this loss with the label-conditional MI between the intermediate and the last features . Lself = − ∑ ω , ω1 ∑ i , p1 log exp ( ω ( xi ) • ω1 ( xp1 ) /τ ) ∑ ω2 ( ∑ p2 exp ( ω ( xi ) • ω2 ( xp2 ) /τ ) + ∑ n exp ( ω ( xi ) • ω2 ( xn ) /τ ) ) ( 2 ) i ∈ I ≡ { { 1 , . . . , B } ( SelfCon-S ) { 1 , . . . , 2B } ( SelfCon-M ) p∗ ∈ Pi∗ ≡ { p ∈ I|yp = yi } n ∈ Ni ≡ { n ∈ I|yn 6= yi } ω , ω1 , ω2 ∈ Ω = { F , G } , a function set of the backbone network and the sub-network . We exclude an anchor sample from the positive set to avoid contrasting the same feature , i.e. , Pi∗ ← Pi∗ \ { i } when ω = ω∗ . While prevalent contrastive approaches ( Khosla et al. , 2020 ; Chen et al. , 2020 ; He et al. , 2020 ; Grill et al. , 2020 ; Caron et al. , 2020 ) force a multi-viewed batch generated by data augmentation , in SelfCon learning , the sub-network plays a role as the augmentation and provides an alternative view on the feature space . Therefore , we formulate our SelfCon loss function with singleviewed batch ( SelfCon-S ) ; I ≡ { 1 , . . . , B } without the additional augmented samples . Besides , we also define SelfCon with multi-viewed batch ( SelfCon-M ) loss , i.e. , Lself with I ≡ { 1 , . . . , 2B } . In the development of our SelfCon loss formulation , we can further use multiple sub-networks , i.e. , Ω = { F , G1 , G2 , . . . } . Appendix C provides the classification performance of the expanded network , which was comparable to that of the single sub-network in our experiments . Thus , we simply use a single sub-network throughout our paper .

Summary: Recent contrastive learning-based algorithms proposed various ways to make representations from semantically similar (positive) samples closer and those from semantically different (negative) samples farther. While they have shown remarkable performance improvement, they usually require a multi-viewed batch for defining the positive samples. In this paper, the authors proposed Self-Contrastive (SelfCon) Learning which self-contrasts within multiple outputs from the different levels of a multi-exit model. The authors theoretically demonstrated that SelfCon loss guarantees the lower bound of label-conditional mutual information (MI) between the intermediate and the last features. Then, they empirically showed the proposed method significantly outperforms the cross-entropy and Supervised Contrastive Learning (SupCon) baselines.