Self-Contrastive Learning

1 INTRODUCTION . Recent studies have studied the success of deep neural networks by investigating how neural networks can encode representations with rich information ( Tishby & Zaslavsky , 2015 ; Shwartz-Ziv & Tishby , 2017 ; Hjelm et al. , 2018 ; Saxe et al. , 2019a ) . Among the various approaches suggested , contrastive loss functions , designed to maximize the lower bound of mutual information ( MI ) between the target and context , have achieved considerable success in self-supervised representation learning first ( Gutmann & Hyvärinen , 2010 ; Oord et al. , 2018 ) and supervised learning recently ( Khosla et al. , 2020 ; Gunel et al. , 2020 ; Wang et al. , 2021 ) . The main objective of the contrastive loss functions in supervision is to make representations from the same class closer and representations from different classes farther . To this end , they define positive samples , i.e. , augmented samples from the same image or ( augmented ) images sharing the same class label , and negative samples , i.e. , all other samples , for every data batch . Contrasting two random augmented samples , often referred to as a multi-viewed batch , has shown impressive results in representation learning ( Chen et al. , 2020 ; He et al. , 2020 ; Grill et al. , 2020 ; Caron et al. , 2020 ; Chen & He , 2020 ; Khosla et al. , 2020 ) , yet a multi-viewed batch causes the following issues . ( 1 ) A multi-viewed batch doubles the batch size , which is a huge burden on memory and computation . ( 2 ) Oracle needs to carefully choose the augmentation policies ( Chen et al. , 2020 ; Tian et al. , 2020 ; Caron et al. , 2020 ; Kim et al. , 2020 ) . ( 3 ) A multi-viewed contrastive task is domain-specific , because data-level augmentation requires specific domain knowledge such as image cropping and flipping ( Verma et al. , 2021 ) . In spite of these concerns , a supervised contrastive learning framework ( Khosla et al. , 2020 ) still relies on multi-views , although they are not necessarily needed with the help of usable label information . In this work , we propose a novel contrastive learning framework in supervision , called SelfContrastive ( SelfCon ) learning , that does not require additional augmented samples . Instead , SelfCon uses a multi-exit framework ( Teerapittayanon et al. , 2016 ; Zhang et al. , 2019a ; b ) where sub-networks produce multiple outputs for the same input and self-contrasts within multiple outputs from the different levels of a single network . For training , SelfCon learning can use the multi-viewed batch as well . However , the multi-exit framework already generates positive pairs from a single image , which can replace the augmentation based multi-view . In Figure 1 , we compare SelfCon learning and Supervised Contrastive ( SupCon , Khosla et al . ( 2020 ) ) learning . SelfCon learning improves the classification performance of the encoder network , owing to ( 1 ) the increase of the lower bound of label-conditional MI between the intermediate and the last features , and ( 2 ) the regularization effect from the single-view and sub-network . We theoretically demonstrate that SelfCon loss is the lower bound of label-conditional MI . Furthermore , unlike SupCon loss , it encourages the encoder to learn the label information that only intermediate features can explain . Second , we empirically show that SelfCon learning reduces the generalization error with the single-view and sub-network . The former prevents the encoder from overfitting to each instance , and the latter regularizes the intermediate feature to be similar to the last feature . The contributions of our paper can be summarized as follows : [ S3 ] We propose Self-Contrastive learning , a novel supervised contrastive framework between the multiple features from the different levels of a single network . [ S4 ] SelfCon loss guarantees the lower bound of label-conditional MI between the intermediate and the last features , and leads to increasing interaction information between the features and the label . [ S5.1 , S5.2 ] SelfCon learning achieved higher classification accuracy for various benchmarks and architectures than cross-entropy and SupCon loss . Our empirical study of MI estimation provides evidence for superior performance . [ S5.3 , S5.4 ] We investigate the advantages of single-viewed batch in terms of the generalization error and computation resources . We also identify that SelfCon learning benefits from the sub-network owing to the regularization effect , vanishing gradient , and ensemble prediction . 2 RELATED WORKS . 2.1 CONTRASTIVE LEARNING . As the cost of data labeling increases exponentially , there is an increased need for acquiring representation under unsupervised scenarios . After Oord et al . ( 2018 ) proposed an InfoNCE loss ( also called a contrastive loss ) , contrastive learning-based algorithms showed a remarkable improvement in performance ( Chen et al. , 2020 ; He et al. , 2020 ; Grill et al. , 2020 ; Caron et al. , 2020 ; Chen & He , 2020 ) . Most previous works use the structure of a Siamese network , which is a weight-sharing network applied on two or more inputs , with negative pairs ( SimCLR ( Chen et al. , 2020 ) , MoCo ( He et al. , 2020 ) ) , momentum encoders ( MoCo ( He et al. , 2020 ) , BYOL ( Grill et al. , 2020 ) ) , online clustering ( SwAV ( Caron et al. , 2020 ) ) , or a stop-gradient operation ( SimSiam ( Chen & He , 2020 ) ) . While the softmax form is frequently used for the contrastive loss ( Chen et al. , 2020 ; He et al. , 2020 ) , recent state-of-the-art algorithms utilize an MSE loss ( Grill et al. , 2020 ; Chen & He , 2020 ) , or a cross-entropy loss ( Caron et al. , 2020 ; 2021 ) . Khosla et al . ( 2020 ) , inspired by the success in self-supervised learning , propose a label-based contrastive loss in supervision , named SupCon loss . Supervised contrastive learning has also been extended to semantic segmentation ( Wang et al. , 2021 ) and language tasks ( Gunel et al. , 2020 ) . While the SupCon loss utilizes the output features from two randomly augmented images , we also contrast the features from different network paths by introducing the multi-exit framework ( Teerapittayanon et al. , 2016 ; Zhang et al. , 2019a ; b ) . We further propose a novel loss function that we can apply on the single-viewed batch . 2.2 MUTUAL INFORMATION . Mutual information is a measure to quantify the information held in a random variable about the other variable , and it has been used as a powerful tool to open the black box of deep neural networks ( Tishby & Zaslavsky , 2015 ; Shwartz-Ziv & Tishby , 2017 ; Saxe et al. , 2019a ) . As it is difficult to compute the MI exactly ( Paninski , 2003 ) , several works have proposed variational MI estimators based on neural networks , e.g. , InfoNCE ( Oord et al. , 2018 ) , MINE ( Belghazi et al. , 2018 ) , NWJ ( Nguyen et al. , 2010 ) , ML-CPC ( Song & Ermon , 2020 ) , and SMILE ( Song & Ermon , 2019 ) . Recently , MI estimator-based objectives have been proposed to improve the performance of contrastive learning ( Hjelm et al. , 2018 ; Bachman et al. , 2019 ; Song & Ermon , 2020 ; Wu et al. , 2020b ) and knowledge distillation ( Tian et al. , 2019b ; Ahn et al. , 2019 ) . Among previous approaches , we are highly motivated by those that aim to increase the MI between the intermediate and the last features . DIM ( Hjelm et al. , 2018 ) and AMDIM ( Bachman et al. , 2019 ) propose novel contrastive losses between the global features and local features ( i.e. , all pixels or patches of the intermediate features ) , while we contrast the refined local features via sub-networks . VID ( Ahn et al. , 2019 ) makes the student learn the distribution of the activations in the auxiliary teacher ’ s intermediate features . Our method differs from VID in that we self-distill within the network itself . Zhang et al . ( 2019a ; b ) share a similar idea with the aforementioned works , but they implicitly increase MI using Kullback–Leibler ( KL ) divergence loss . 3 SELF-CONTRASTIVE LEARNING . We propose a new supervised contrastive loss that maximizes the similarity of the outputs from different network paths by introducing the multi-exit framework . We define the encoder structure with F as a backbone network and G as a sub-network that shares the backbone ’ s parameters up to some intermediate layer . T denotes those sharing layers that produce the intermediate feature . The sub-network has additional non-sharing parameters attached after T . Note that F and G include the projection head on the encoder . We highlight the positive and negative pairs with respect to an anchor sample , following Figure 1 . SupCon loss To mitigate the weaknesses of cross-entropy , such as the reduced generalization performance and the possibility of poor margins , Khosla et al . ( 2020 ) proposed a supervised version of the contrastive loss that defines the positive pairs as every sample with the same ground-truth label . We reformulate the SupCon loss function as follows : Lsup = − ∑ i , p1 log exp ( F ( xi ) • F ( xp1 ) /τ ) ∑ p2 exp ( F ( xi ) • F ( xp2 ) /τ ) + ∑ n exp ( F ( xi ) • F ( xn ) /τ ) ( 1 ) i ∈ I ≡ { 1 , . . . , 2B } p∗ ∈ Pi∗ ≡ { p ∈ I \ { i } |yp = yi } n ∈ Ni ≡ { n ∈ I|yn 6= yi } where • denotes the inner product , B is the batch size , and τ is the temperature to soften or harden the softmax value . I denotes a set of indices of the multi-viewed batch that concatenates the original B images and the augmented ones , i.e. , xB+i is an augmented pair of xi . Pi∗ and Ni are a set of positive and negative pair indices with respect to an anchor i , respectively . Note that Eq . 1 is the same as NT-Xent loss ( Chen et al. , 2020 ) when Pi∗ ≡ { ( i+B ) mod 2B } . We dropped the dividing term of sums for brevity . SelfCon loss Besides minimizing the above loss , we aim to maximize the similarity between the outputs from the backbone and the sub-network . To this end , we define SelfCon loss , which forms a self-contrastive task for every output including the features from the sub-network . In Section 4 , we clarify the connection of this loss with the label-conditional MI between the intermediate and the last features . Lself = − ∑ ω , ω1 ∑ i , p1 log exp ( ω ( xi ) • ω1 ( xp1 ) /τ ) ∑ ω2 ( ∑ p2 exp ( ω ( xi ) • ω2 ( xp2 ) /τ ) + ∑ n exp ( ω ( xi ) • ω2 ( xn ) /τ ) ) ( 2 ) i ∈ I ≡ { { 1 , . . . , B } ( SelfCon-S ) { 1 , . . . , 2B } ( SelfCon-M ) p∗ ∈ Pi∗ ≡ { p ∈ I|yp = yi } n ∈ Ni ≡ { n ∈ I|yn 6= yi } ω , ω1 , ω2 ∈ Ω = { F , G } , a function set of the backbone network and the sub-network . We exclude an anchor sample from the positive set to avoid contrasting the same feature , i.e. , Pi∗ ← Pi∗ \ { i } when ω = ω∗ . While prevalent contrastive approaches ( Khosla et al. , 2020 ; Chen et al. , 2020 ; He et al. , 2020 ; Grill et al. , 2020 ; Caron et al. , 2020 ) force a multi-viewed batch generated by data augmentation , in SelfCon learning , the sub-network plays a role as the augmentation and provides an alternative view on the feature space . Therefore , we formulate our SelfCon loss function with singleviewed batch ( SelfCon-S ) ; I ≡ { 1 , . . . , B } without the additional augmented samples . Besides , we also define SelfCon with multi-viewed batch ( SelfCon-M ) loss , i.e. , Lself with I ≡ { 1 , . . . , 2B } . In the development of our SelfCon loss formulation , we can further use multiple sub-networks , i.e. , Ω = { F , G1 , G2 , . . . } . Appendix C provides the classification performance of the expanded network , which was comparable to that of the single sub-network in our experiments . Thus , we simply use a single sub-network throughout our paper .

Summary: The authors propose an improved algorithm for supervised contrastive learning, which they term “self-contrastive learning”. Instead of only contrasting representations at one point in the network, self-contrasting uses readouts at multiple stages in the network (remapped into an appropriate feature space by an additional readout network). This makes it possible to propose two variants: A “multi-batch” variant that first augments the images, and then contrasts views between samples of the same class, both augmented and non-augmented, and a “single-batch” variant that only contrasts between samples of the same class. Improvements are shown for CIFAR-10,100, TinyImageNet, as well as a smaller version of ImageNet with 100 classes, referenced as ImageNet-100.