Year: 2021

Abstract:
This paper proposes a novel supervised contrastive learning framework, called Self-Contrastive (SelfCon) Learning, that self-contrasts within multiple outputs from the different levels of a multi-exit network. SelfCon learning does not require additional augmented samples, which resolves the concerns of multi-viewed batch (e.g., high computational cost and generalization error). Furthermore, we prove that SelfCon loss guarantees the lower bound of label-conditional mutual information between the intermediate and the last feature. In our experiments including ImageNet-100, SelfCon surpasses cross-entropy and Supervised Contrastive (SupCon) learning without the need for a multi-viewed batch. We demonstrate that the success of SelfCon learning is related to the regularization effect associated with the single-view and sub-network.

heading: 1 INTRODUCTION
text: Recent studies have studied the success of deep neural networks by investigating how neural networks can encode representations with rich information (Tishby & Zaslavsky, 2015; Shwartz-Ziv & Tishby, 2017; Hjelm et al., 2018; Saxe et al., 2019a). Among the various approaches suggested, contrastive loss functions, designed to maximize the lower bound of mutual information (MI) between the target and context, have achieved considerable success in self-supervised representation learning first (Gutmann & Hyvärinen, 2010; Oord et al., 2018) and supervised learning recently (Khosla et al., 2020; Gunel et al., 2020; Wang et al., 2021). The main objective of the contrastive loss functions in supervision is to make representations from the same class closer and representations from different classes farther. To this end, they define positive samples, i.e., augmented samples from the same image or (augmented) images sharing the same class label, and negative samples, i.e., all other samples, for every data batch.
Contrasting two random augmented samples, often referred to as a multi-viewed batch, has shown impressive results in representation learning (Chen et al., 2020; He et al., 2020; Grill et al., 2020; Caron et al., 2020; Chen & He, 2020; Khosla et al., 2020), yet a multi-viewed batch causes the following issues. (1) A multi-viewed batch doubles the batch size, which is a huge burden on memory and computation. (2) Oracle needs to carefully choose the augmentation policies (Chen et al., 2020; Tian et al., 2020; Caron et al., 2020; Kim et al., 2020). (3) A multi-viewed contrastive task is domain-specific, because data-level augmentation requires specific domain knowledge such as image cropping and flipping (Verma et al., 2021). In spite of these concerns, a supervised contrastive learning framework (Khosla et al., 2020) still relies on multi-views, although they are not necessarily needed with the help of usable label information.
In this work, we propose a novel contrastive learning framework in supervision, called SelfContrastive (SelfCon) learning, that does not require additional augmented samples. Instead, SelfCon uses a multi-exit framework (Teerapittayanon et al., 2016; Zhang et al., 2019a;b) where sub-networks produce multiple outputs for the same input and self-contrasts within multiple outputs from the different levels of a single network. For training, SelfCon learning can use the multi-viewed batch as well. However, the multi-exit framework already generates positive pairs from a single image, which can replace the augmentation based multi-view. In Figure 1, we compare SelfCon learning and Supervised Contrastive (SupCon, Khosla et al. (2020)) learning.
SelfCon learning improves the classification performance of the encoder network, owing to (1) the increase of the lower bound of label-conditional MI between the intermediate and the last features, and (2) the regularization effect from the single-view and sub-network. We theoretically demonstrate that SelfCon loss is the lower bound of label-conditional MI. Furthermore, unlike
SupCon loss, it encourages the encoder to learn the label information that only intermediate features can explain. Second, we empirically show that SelfCon learning reduces the generalization error with the single-view and sub-network. The former prevents the encoder from overfitting to each instance, and the latter regularizes the intermediate feature to be similar to the last feature.
The contributions of our paper can be summarized as follows:
[S3] We propose Self-Contrastive learning, a novel supervised contrastive framework between the multiple features from the different levels of a single network.
[S4] SelfCon loss guarantees the lower bound of label-conditional MI between the intermediate and the last features, and leads to increasing interaction information between the features and the label.
[S5.1, S5.2] SelfCon learning achieved higher classification accuracy for various benchmarks and architectures than cross-entropy and SupCon loss. Our empirical study of MI estimation provides evidence for superior performance.
[S5.3, S5.4] We investigate the advantages of single-viewed batch in terms of the generalization error and computation resources. We also identify that SelfCon learning benefits from the sub-network owing to the regularization effect, vanishing gradient, and ensemble prediction.

heading: 2 RELATED WORKS

heading: 2.1 CONTRASTIVE LEARNING
text: As the cost of data labeling increases exponentially, there is an increased need for acquiring representation under unsupervised scenarios. After Oord et al. (2018) proposed an InfoNCE loss (also called a contrastive loss), contrastive learning-based algorithms showed a remarkable improvement in performance (Chen et al., 2020; He et al., 2020; Grill et al., 2020; Caron et al., 2020; Chen & He, 2020). Most previous works use the structure of a Siamese network, which is a weight-sharing network applied on two or more inputs, with negative pairs (SimCLR (Chen et al., 2020), MoCo (He et al., 2020)), momentum encoders (MoCo (He et al., 2020), BYOL (Grill et al., 2020)), online clustering (SwAV (Caron et al., 2020)), or a stop-gradient operation (SimSiam (Chen & He, 2020)). While the softmax form is frequently used for the contrastive loss (Chen et al., 2020; He et al., 2020), recent state-of-the-art algorithms utilize an MSE loss (Grill et al., 2020; Chen & He, 2020), or a cross-entropy loss (Caron et al., 2020; 2021). Khosla et al. (2020), inspired by the success in self-supervised learning, propose a label-based contrastive loss in supervision, named SupCon loss. Supervised contrastive learning has also been extended to semantic segmentation (Wang et al.,
2021) and language tasks (Gunel et al., 2020). While the SupCon loss utilizes the output features from two randomly augmented images, we also contrast the features from different network paths by introducing the multi-exit framework (Teerapittayanon et al., 2016; Zhang et al., 2019a;b). We further propose a novel loss function that we can apply on the single-viewed batch.

heading: 2.2 MUTUAL INFORMATION
text: Mutual information is a measure to quantify the information held in a random variable about the other variable, and it has been used as a powerful tool to open the black box of deep neural networks (Tishby & Zaslavsky, 2015; Shwartz-Ziv & Tishby, 2017; Saxe et al., 2019a). As it is difficult to compute the MI exactly (Paninski, 2003), several works have proposed variational MI estimators based on neural networks, e.g., InfoNCE (Oord et al., 2018), MINE (Belghazi et al., 2018), NWJ (Nguyen et al., 2010), ML-CPC (Song & Ermon, 2020), and SMILE (Song & Ermon, 2019). Recently, MI estimator-based objectives have been proposed to improve the performance of contrastive learning (Hjelm et al., 2018; Bachman et al., 2019; Song & Ermon, 2020; Wu et al., 2020b) and knowledge distillation (Tian et al., 2019b; Ahn et al., 2019). Among previous approaches, we are highly motivated by those that aim to increase the MI between the intermediate and the last features. DIM (Hjelm et al., 2018) and AMDIM (Bachman et al., 2019) propose novel contrastive losses between the global features and local features (i.e., all pixels or patches of the intermediate features), while we contrast the refined local features via sub-networks. VID (Ahn et al., 2019) makes the student learn the distribution of the activations in the auxiliary teacher’s intermediate features. Our method differs from VID in that we self-distill within the network itself. Zhang et al. (2019a;b) share a similar idea with the aforementioned works, but they implicitly increase MI using Kullback–Leibler (KL) divergence loss.

heading: 3 SELF-CONTRASTIVE LEARNING
text: We propose a new supervised contrastive loss that maximizes the similarity of the outputs from different network paths by introducing the multi-exit framework. We define the encoder structure with F as a backbone network and G as a sub-network that shares the backbone’s parameters up to some intermediate layer. T denotes those sharing layers that produce the intermediate feature. The sub-network has additional non-sharing parameters attached after T . Note that F and G include the projection head on the encoder. We highlight the positive and negative pairs with respect to an anchor sample, following Figure 1.
SupCon loss To mitigate the weaknesses of cross-entropy, such as the reduced generalization performance and the possibility of poor margins, Khosla et al. (2020) proposed a supervised version of the contrastive loss that defines the positive pairs as every sample with the same ground-truth label. We reformulate the SupCon loss function as follows:
Lsup = − ∑ i,p1 log exp(F (xi) • F (xp1)/τ)∑
p2
exp(F (xi) • F (xp2)/τ) + ∑ n exp(F (xi) • F (xn)/τ)
(1)
i ∈ I ≡ {1, . . . , 2B} p∗ ∈ Pi∗ ≡ {p ∈ I \ {i}|yp = yi} n ∈ Ni ≡ {n ∈ I|yn 6= yi}
where • denotes the inner product, B is the batch size, and τ is the temperature to soften or harden the softmax value. I denotes a set of indices of the multi-viewed batch that concatenates the original B images and the augmented ones, i.e., xB+i is an augmented pair of xi. Pi∗ and Ni are a set of positive and negative pair indices with respect to an anchor i, respectively. Note that Eq. 1 is the same as NT-Xent loss (Chen et al., 2020) when Pi∗ ≡ {(i+B) mod 2B}. We dropped the dividing term of sums for brevity.
SelfCon loss Besides minimizing the above loss, we aim to maximize the similarity between the outputs from the backbone and the sub-network. To this end, we define SelfCon loss, which forms a self-contrastive task for every output including the features from the sub-network. In Section 4, we clarify the connection of this loss with the label-conditional MI between the intermediate and the last features.
Lself = − ∑ ω,ω1 ∑ i,p1 log exp(ω(xi) • ω1(xp1)/τ)∑
ω2 (∑ p2 exp(ω(xi) • ω2(xp2)/τ) + ∑ n exp(ω(xi) • ω2(xn)/τ) ) (2) i ∈ I ≡ { {1, . . . , B} (SelfCon-S) {1, . . . , 2B} (SelfCon-M) p∗ ∈ Pi∗ ≡ {p ∈ I|yp = yi} n ∈ Ni ≡ {n ∈ I|yn 6= yi}
ω,ω1,ω2 ∈ Ω = {F ,G}, a function set of the backbone network and the sub-network. We exclude an anchor sample from the positive set to avoid contrasting the same feature, i.e., Pi∗ ← Pi∗ \ {i} when ω = ω∗. While prevalent contrastive approaches (Khosla et al., 2020; Chen et al., 2020; He et al., 2020; Grill et al., 2020; Caron et al., 2020) force a multi-viewed batch generated by data augmentation, in SelfCon learning, the sub-network plays a role as the augmentation and provides an alternative view on the feature space. Therefore, we formulate our SelfCon loss function with singleviewed batch (SelfCon-S); I ≡ {1, . . . , B} without the additional augmented samples. Besides, we also define SelfCon with multi-viewed batch (SelfCon-M) loss, i.e., Lself with I ≡ {1, . . . , 2B}. In the development of our SelfCon loss formulation, we can further use multiple sub-networks, i.e., Ω = {F ,G1,G2, . . . }. Appendix C provides the classification performance of the expanded network, which was comparable to that of the single sub-network in our experiments. Thus, we simply use a single sub-network throughout our paper.

heading: 4 DISCUSSIONS
text: Question 1: How does SelfCon loss maximize the lower bound of label-conditional MI between the intermediate and the last features?
We explain in the following propositions to answer Question 1. All the proofs can be found in Appendix A. Proposition 4.1. Let x and z be different samples that share the same class label c. Then, with some discriminator function modeled by a neural network F and 2(K − 1) negative sample size, SupCon loss maximizes the lower bound of conditional MI between the output features of a positive pair,
log(2K − 1)− Lsup(x, z;F ,K) ≤ I(F (x);F (z)|c). (3)
Proposition 4.2. Denote Lself -s as SelfCon loss with single-viewed batch. SelfCon-S loss maximizes the lower bound of MI between the output features from the backbone and the sub-network,
log(2K − 1)− Lself -s(x; {F ,G},K) ≤ I(F (x);G(x)|c). (4)
Proposition 4.3. Let T (x) be the intermediate feature of the backbone network, which is also an input to the auxiliary network path. Then the r.h.s. of Eq. 4 is upper-bounded by
I(F (x);T (x)|c) = I(F (x); c|T (x))− I(F (x); c)︸ ︷︷ ︸ ( ) + I(F (x);T (x))︸ ︷︷ ︸ ( ) . (5)
Proposition 4.1 and 4.2 can be derived from the exact bound of InfoNCE (Poole et al., 2019). SupCon and SelfCon-S loss have 2(K − 1) negative sample size because of the augmented negative pairs for SupCon and the sub-network features for SelfCon-S. Note that SelfCon-M loss has a similar upper bound as Eq. 3 and Eq. 4 (refer to Appendix A.2).
In Proposition 4.3, ( ) is an interaction information (Yeung, 1991) that measures the influence of T (x) on the amount of shared information between F (x) and c. SelfCon-S loss is related to increasing this interaction information, so that the intermediate feature enhances the correlation between the last feature and the label, which may result in improving the downstream classification tasks. In addition, ( ) implies that SelfCon loss increases the MI between the intermediate and the last features in the backbone. T (x) is distilled from F (x) that has richer class-related information and consequently, earlier layers are trained to produce better representation (Zhang et al., 2019a; Ahn et al., 2019). We believe this is theoretical evidence for the improved performance of SelfCon. Also, refer to Appendix A.4 for the connection to the classification performance of the SelfCon loss.
aWe have re-implemented SupCon method (Khosla et al., 2020) and also run their official code for credibility, but the accuracy was slightly lower than their reported numbers. bSupCon-S sets I as {1, ..., B} in Eq. 1. Although Khosla et al. (2020) did not propose the version of the single-view, we implemented SupCon-S since it is worth investigating the effect of multi-viewed batch.
Question 2: Is it applicable to unsupervised representation learning?
Good representations should get rid of redundant input information and extract meaningful features (Tishby & Zaslavsky, 2015). However, in Eq. 2, the backbone network can be an anchored function where the contrastive features are those from the backbone as well as the sub-network. Unfortunately, this might allow the last feature to follow the intermediate feature, learning redundant information about the input. This could be why SelfCon learning might not work in an unsupervised environment where there is no label information (refer to Appendix D.1). However, in Appendix D.2, we suggest that blocking the backbone from following the sub-network, i.e., removing the term in Eq. 2 where ω = F ,ω∗ = G, can mitigate the problem.
Supervised contrastive framework, which is the main interest of our paper, is away from the above problem owing to the label information. Note that the unsupervised version of SelfCon-S loss guarantees only the lower bound of ( ) in Proposition 4.3. Meanwhile, ( ), induced by the condition on label, encourages the backbone network to be in accordance with label information. Thus, we believe that jointly maximizing the lower bound of ( + ) offers an evidence for the success of supervised SelfCon learning.

heading: 5 EXPERIMENT
text: We presented the image classification accuracy for standard benchmarks, such as CIFAR-10, CIFAR100 (Krizhevsky et al., 2009), Tiny-ImageNet (Le & Yang, 2015), ImageNet-100 (Tian et al., 2019a), and ImageNet (Deng et al., 2009), and extensively analyzed the results. We summarized the mean and standard deviation of top-1 accuracy over three random seeds for the reliability of the experimental results.
We used the optimal structure and position of the sub-network for all the network architectures (see Appendix C). We compared three types of the sub-network: the same structure with non-sharing layers of the backbone network (same), a structure with the number of non-sharing layers reduced in half (small), and a simple fully-connected layer (fc). The overall performance was comparable or better than the baselines. We observed a trend: in shallow networks the same structure was better, while fc was better in deeper networks. Besides, the performance was consistently good when the exit path is attached after the midpoint of the encoder (e.g., 2nd block in ResNet or 3rd block in VGG architecture). Complete implementation details are in Appendix B.

heading: 5.1 REPRESENTATION LEARNING
text: We measured the classification accuracy on the representation learning protocol (Chen et al., 2020), which consists of 2-stage training, (1) pretraining an encoder network and (2) fine-tuning on a linear classifier with the frozen encoder (called a linear evaluation).
The classification accuracy is summarized in Table 1. Interestingly, the loss functions in the singleviewed batch, such as SupCon-S and SelfCon-S, outperform their multi-view counterparts in all settings. Furthermore, our SelfCon learning, which trains with the sub-network, shows higher classification accuracy than CE and SupCon. The effects of the sub-network are analyzed in Section 5.4. We also experimented on the ImageNet-100 benchmark, of which 100 classes were randomly sampled (Tian et al., 2019a), to verify that SelfCon learning has the same effect on large scale datasets. Table 2 presents the consistent performance improvement of SelfCon learning. Refer to Appendix G for the ImageNet results.

heading: 5.2 MUTUAL INFORMATION ESTIMATION
text: Minimizing the SelfCon loss is highly related to maximizing the MI between the intermediate and the last features (Proposition 4.3). To empirically confirm this claim, we estimated MI using various estimators: InfoNCE (Oord et al., 2018), MINE (Belghazi et al., 2018), and NWJ (Nguyen et al., 2010). Specifically, we extracted the features of the CIFAR-100 dataset from the pretrained ResNet-50 encoders and computed the estimators. Then we optimized a simple 3-layer Conv-ReLU network with the MI estimator objectives. We consider the encoder output without the projection head, which differs from Section 3.
In Figure 2, SelfCon-M and SelfCon-S both show high I(F (x);T (x)), which supports our claim, while the values of CE and SupCon are lower. Recall that T (x) is the intermediate feature of the backbone network, which is an input to the auxiliary network path. Although I(F (x);T (x)) does not explicitly guarantee an increase in accuracy, our results imply that it has a positive correlation with the test accuracy of the backbone network. We believe that the richer information in earlier features makes the encoder output better representation because the intermediate feature is also the input for the subsequent layers.
We also found that SelfCon learning increases the information between T (x) and the label, implying that the intermediate feature is imbued with class-related knowledge, while the information between T (x) and the input x is decreased (refer to Table 12 in Appendix E). This suggests that SelfCon learning is in agreement with IB principle of training a deep neural network, i.e., fitting-compression phase (Saxe et al., 2019a). Furthermore, refer to Appendix K for the additional empirical evidence of correlation between MI and improved performance.

heading: 5.3 MULTI-VIEW VS. SINGLE-VIEW
text: We compare the multi-view and single-view in terms of generalization error, efficiency, and batch size.
Single-view reduces generalization error. In Figure 3, SupCon shows higher train accuracy but lower test accuracy than SupCon-S, and the same trend is observed with SelfCon-M and SelfCon-S. Compared to single-view, multi-view from the augmented image makes the encoder amplify the memorization of data and result in overfitting to each instance. Figure 4 shows that SelfCon-S gradually enhanced the generalization ability, while SelfCon-M or SupCon achieved little gain in test accuracy despite the fast convergence.
Single-view is efficient in terms of memory usage and computational cost. In Table 3, we compared SupCon and SelfCon-S to observe the efficiency of the single-viewed batch. SelfCon-S requires a larger number of parameters owing to the extra sub-network but is more efficient in memory and computation. In both SupCon and SelfCon-S, the same batch size implies the same number of anchor features; however, they differ in memory consumption due to the data augmentation of the multi-viewed batch.
Multi-view is advantageous for small batch size. In supervised learning, large batch size reduces the generalization ability, which results in decreasing the performance (You et al., 2017; Luo et al., 2018; Wu et al., 2020a). We examined whether the performance in a supervised contrastive framework is also dependent on the batch size. Table 4 summarizes the results. The multi-viewed method, e.g., SupCon, which doubles the effective number of training data, outperformed the single-viewed counterpart in 64- batch experiments; the opposite was observed in all other batch sizes. For small batch size such as
64, doubling the effective batch size can make the learning stable, but as batch size gets larger the stabilizing effect from multi-views decreases and the necessity of regularization appears to be critical. Hence, the optimal batch size of single-viewed methods was higher (SupCon-S vs. SupCon and SelfCon-S vs. SelfCon-M) because single-view itself helps regularization (i.e., small batch with single-view might lead to under-fitting). Also, note that SelfCon learning with sub-network still surpasses the SupCon counterpart. Refer to Appendix I for the sensitivity study of the learning rates.

heading: 5.4 WHAT DOES THE SUB-NETWORK ACHIEVE?
text: Regularization effect SelfCon loss regularizes the sub-network to output the similar features to the backbone network. It prevents the encoder from overfitting to data, and it is effective in multi-viewed as well as single-viewed batch. In Figure 3, we confirmed the regularization effect (i.e., lower train accuracy, but higher test accuracy) by comparing each bar of the same color. The strong regularization of the sub-network helped SelfCon (-M, -S) outperform the SupCon counterparts. This trend is also observed in Table 4 and Figure 4.
Vanishing gradient SelfCon learning can send more abundant information to the earlier layers through the gradients flowing from the sub-networks. Previous works (Lee et al., 2015; Teerapittayanon et al., 2016; Zhang et al., 2019a) point out that the success of the multi-exit framework is owing to solving the vanishing gradient. We showed that the same argument applies to our SelfCon learning. Note that the sub-network is positioned after the 2nd block of the backbone. In Figure 5, a larger gradient flows up to the earlier layer in the SelfCon-M, while a large amount of the SupCon loss gradient vanishes. In particular, there is a significant difference in the gradient norm in the 2nd block.
Ensemble with sub-network The sub-network in SelfCon learning can also be used on downstream tasks such as image classification. In our previous experiments, we followed the linear evaluation protocol, fine-tuning a classifier with the frozen backbone network. The network pretrained on SelfCon learning has an exit path, which is similarly allowed to be frozen and used as linear evaluation. We thus demonstrated two additional types of linear evaluation scenarios: (1) fine-tuning a classifier after the sub-network output and (2) fine-tuning two classifiers after the backbone and the sub-network and ensembling two classifiers’ predictions. Table 5 indicates that the sub-network could make appropriate, although not the best, predictions as the backbone did, while ensembling their predictions is found to be the most powerful trick we have come up with.
Feature-level multi-view One of the advantages of SelfCon learning is to relax the dependency on multi-viewed batch. This is accomplished by the multi-views on the representation space made by the parameters of the sub-network. In Figure 6, we visualized via Grad-CAM (Selvaraju et al., 2017) the gradient of SelfCon-S loss with respect to the intermediate layer of the backbone network, right before the exit path. Both networks focus on similar, but clearly different pixels of the same input image, implying that the sub-network learns another view in the feature space. As multi-view in contrastive learning requires domain-specific augmentation, recent studies have explored the domain-agnostic way of augmentation (Lee et al., 2020; Verma et al., 2021). SelfCon might be an intriguing future work in that auxiliary networks could be an efficient substitute for data augmentation.

heading: 5.5 ABLATION STUDY
text: Different encoder architectures We experimented with other architectures: VGG-16 (Simonyan & Zisserman, 2014) with Batch Normalization (BN) (Ioffe & Szegedy, 2015) and WRN-16-8 (Zagoruyko & Komodakis, 2016), and the results are presented in Table 6. The classification accuracy
for WRN-16-8 showed a similar trend as that of ResNet architectures. However, for VGG-16 with BN architecture, SupCon had lower performance than CE on every dataset. Although the contrastive learning approach does not seem to result in significant changes for the VGG-16 with BN encoders, SelfCon-S was better than or comparable to CE.
1-stage training 1-stage training framework, i.e., not decoupling the encoder pretraining and linear evaluation, on the single-viewed batch, is standard for supervised learning. With a multi-exit framework, Zhang et al. (2019a) propose Self-Distillation (SD), which distills logit information within the network itself, i.e., LSD = αLCE + (1 − α)LKL. We replaced the KL divergence term with Lself -s in Eq. 2, and distilled the features from the projection head, instead of the logits. From Table 7, we observed that adding cross-entropy loss to the sub-network (CE w/ Sub) improved the backbone network’s classification performance. However, the results of SD suggested that there is a saturation of
increase even when the classifier of sub-network better converged. Meanwhile, we hypothesize that feature distillation of SelfCon-S makes the encoder learn better representation than logit distillation of classifiers. We would like to highlight that SelfCon loss in 2-stage training still demonstrated the best classification accuracy, as Khosla et al. (2020) argued that representation learning mitigates the poor generalization performance of CE-based 1-stage training.

heading: 6 CONCLUSION
text: We proposed SelfCon learning, which self-contrasts the features from the multiple levels of a network. SelfCon learning is free from the issues that multi-viewed batch triggers. We found that SelfCon loss maximizes the lower bound of label-conditional MI between the intermediate and the last features, and it is related to improving the classification performance. In addition, we analyzed why SelfCon is better by exploring the effect of single-view and sub-network. We verify by extensive experiments, including ImageNet-100 and ImageNet, that SelfCon-S loss outperforms CE and SupCon.

heading: A PROOFS
text: A.1 PROOF OF PROPOSITION 4.1
Proof. We simply extend the exact bound of InfoNCE (Poole et al., 2019; Sordoni et al., 2021). Here, we consider the supervised setting where there are C training classes. Without loss of generality, choose a class c out of C classes, and let x and z be different samples that share the same class label c. The derivation for the multi-view (z being an augmented sample of x) is similar. For conciseness of the proof, we consider that no other image in a batch shares the same class. We prove that minimizing the SupCon loss (Khosla et al., 2020) maximizes the lower bound of conditional MI between two samples x and z given the label c:
I(x; z|c) ≥ log(2K − 1)− Lsup(x, z;F ,K) (6)
for some function F and hyperparameter K.
We start from Barber and Agakov’s variational lower bound on MI (Barber & Agakov, 2004).
I(x; z|c) = Ep(x,z|c) log p(z|x, c) p(z|c) ≥ Ep(x,z|c) log q(z|x, c) p(z|c)
(7)
where q is a variational distribution. Since q is arbitrary, we can set the sampling strategy as follows. First, sample z1 from the proposal distribution π(z|c) where c is a class label of x. Then, sample (K − 1) negative samples {z2, · · · , zK} from the distribution ∑ c′ 6=c π(z, c
′), so that these negative samples do not share the class label with x. We augment each negative sample by random augmentation and concatenate with the original samples, i.e., {z2, · · · , zK , zK+1, · · · , z2K−1}, where zK+i−1 is the augmented sample from zi for 2 ≤ i ≤ K. We define the unnormalized density of z1 given a specific set {z2, · · · , z2K−1} and x of label c is
q(z1|x, z2:(2K−1), c) = π(z1|c) · (2K − 1) · eψF (x,z1) eψF (x,z1) + ∑2K−1 k=2 e ψF (x,zk) (8)
where ψ is often called a discriminator function (Hjelm et al., 2018), defined as ψF (u,v) = F (u) · F (v) for some vectors u,v. By setting the proposal distribution as π(z|c) = p(z|c), we obtain the MI bound:
I(x; z|c) ≥ Ep(x,z1|c) log q(z1|x, c) p(z1|c)
(9)
= Ep(x,z1|c) log Ep(z2:(2K−1)|c)q(z1|x, z2:(2K−1), c)
p(z1|c) (10)
≥ Ep(x,z1|c) [ Ep(z2:(2K−1)|c) log p(z1|c) · (2K−1)·e ψF (x,z1) eψF (x,z1)+ ∑2K−1 k=2 e ψF (x,zk)
p(z1|c)
] (11)
= Ep(x,z1|c)p(z2:(2K−1)|c) log eψF (x,z1)
1 2K−1 ∑2K−1 k=1 e ψF (x,zk) (12)
= log(2K − 1)− Lsup(x, z;F ,K). (13)
where the second inequality is derived from Jensen’s inequality. Because Eq. 12 is an expectation with respect to the sampled x and z1, the case where the anchor is swapped to z1 is also being considered.
A neural network F (backbone in our framework) with L layers are formulated as F = fL ◦ fL−1 ◦ · · · ◦ f1. Then, ψF (u,v) = F (u) · F (v) = f1:L(u) · f1:L(v). We define another discriminator function as ψ†F (u,v) = f(`+1):L(u) · f(`+1):L(v). Obviously, the following equivalence holds:
ψ†F (f1:`(u), f1:`(v)) = ψF (u,v). (14)
Note that f1:`(u) is the `-th intermediate feature of input u. Following the same procedure as in Eq. 9-13,
I(f1:`(x); f1:`(z)|c) ≥ Ep(x,z1|c)p(z2:(2K−1)|c) log eψ † F (f1:`(x),f1:`(z1))
1 2K−1 ∑2K−1 k=1 e ψ†F (f1:`(x),f1:`(zk)) (15)
= Ep(x,z1|c)p(z2:(2K−1)|c) log eψF (x,z1)
1 2K−1 ∑2K−1 k=1 e ψF (x,zk) (16)
= log(2K − 1)− Lsup(x, z;F ,K) (17)
From above, as the intermediate feature is arbitrary to the position, we can obtain a similar inequality:
I(f(`+1):L(f1:`(x)); f(`+1):L(f1:`(z))|c) = I(F (x);F (z)|c) (18) = I(f1:L(x); f1:L(z)|c) (19) ≥ log(2K − 1)− Lsup(x, z;F ,K). (20)
A.2 PROOF OF PROPOSITION 4.2
Proof. In Section 4.1, we proved that SupCon loss maximizes the lower bound of conditional MI between the output features of a positive pair. We can think of another scenario where the network F now has a sub-network G. Assume that the sub-network hasM > ` layers: G = gM ◦gM−1◦· · ·◦g1. As we discussed in the paper, the exit path is placed after the `-th layer, so regarding our definition of the sub-network, G shares the same parameters with F up to `-th layer, i.e., g1 = f1, g2 = f2, · · · , g` = f`. Define ψG(u,v) = G(u) ·G(v) We introduce a discriminator function that measures the similarity between the outputs from the backbone and the sub-network, ψFG(u,v) = F (u) ·G(v). Similarly, ψGF (u,v) = G(u) · F (v). Considering that the SelfCon-S loss has the anchored function of F and G, we obtain an upper bound of two symmetric mutual information. Here, z1 = x because SelfCon-S loss is defined on the single-viewed batch and we assume that other images in a batch (i.e., z2, · · · , zK ) are sampled from the different class label with x.
I(F (x); G(x)|c) + I(G(x);F (x)|c) (21)
≥ E log e ψFG(x,x)
1 2K−1
( eψFG(x,x) + ∑K k=2 e ψFG(x,zk) + ∑K k=2 e ψF (x,zk) )
+ E log eψGF (x,x)
1 2K−1
( eψGF (x,x) + ∑K k=2 e ψGF (x,zk) + ∑K k=2 e ψG(x,zk) ) (22)
= 2 log(2K − 1)− 2Lself -s(x; {F ,G},K) (23)
Due to the symmetry of mutual information,
I(F (x);G(x)|c) ≥ log(2K − 1)− Lself -s(x; {F ,G},K) (24)
In addition, we can similarly bound the SelfCon-M loss. As the derivation of SupCon loss bound, only consider the anchor x and its positive pair z1. When the anchored feature is F (x), the contrastive features are: G(x), G(z), and F (z). By symmetry, when the anchored feature is G(x), the contrastive features are: F (x), F (z), and G(z). As the derivation of the SupCon loss bound, we assume the augmented negative samples, i.e., {z2, · · · , zK , zK+1, · · · , z2K−1}.
I(F (x); G(x)|c) + I(F (x);G(z)|c) + I(F (x);F (z)|c) +I(G(x); F (x)|c) + I(G(x);F (z)|c) + I(G(x);G(z)|c) (25)
≥ 1 3 E log eψFG(x,x) · eψFG(x,z1) · eψF (x,z1) 1 4K−1 ( eψFG(x,x) + ∑2K−1 k=1 e ψFG(x,zk) + ∑2K−1 k=1 e ψF (x,zk) )
+ 1
3 E log eψGF (x,x) · eψGF (x,z1) · eψG(x,z1) 1 4K−1 ( eψGF (x,x) + ∑2K−1 k=1 e ψGF (x,zk) + ∑2K−1 k=1 e ψG(x,zk) ) (26)
= 2
3 log(4K − 1)− 2Lself -m(x, z; {F ,G},K) (27)
A.3 PROOF OF PROPOSITION 4.3
F (x) and G(x) are the output features from the backbone network and the sub-network, respectively. Recall that T denotes the sharing layers between F and G. T (x) is the intermediate feature of the backbone which is also an input to the auxiliary network path.
Before proving the following Lemma, we would like to note that the usefulness of mutual information should be carefully discussed on the stochastic mapping of a neural network. If a mapping T (x) 7→ F (x) is a deterministic mapping, then the MI between T (x) and F (x) is degnerate because I(T (x);F (x)) is either infinite for continuous T (x) (conditional differential entropy is −∞) or a constant for discrete T (x) which is independent on the network’s parameters (equal to H(T (x))). However, for studying the usefulness of mutual information in a deep neural network, the map T (x) 7→ F (x) is considered as a stochastic parameterized channel. In many recent works about information theory with DNN, they view the training via SGD is a stochastic process, and the stochasticity in the training procedure lets us define the MI with stochastically trained representations (Shwartz-Ziv & Tishby, 2017; Goldfeld et al., 2019; Saxe et al., 2019b; Goldfeld et al., 2019). Our theoretical claim focuses on the SelfCon loss as a training loss optimized by SGD algorithm. Therefore, analyzing the MI between the hidden representations while training with the SelfCon loss is based on the information theory to understand DNN (Tishby & Zaslavsky, 2015).
Also, information theory in deep learning, especially in contrastive learning, is based on the InfoMax principle (Linsker, 1989) which is about learning a neural network that maps a set of input to a set of output to maximize the average mutual information between the input and output of a neural network, subject to stochastic processes. This InfoMax principle is nowadays widely used for analyzing and optimizing DNNs. Most works for contrastive learning based on maximizing mutual information grounds on the InfoMax principle, and they are grounding on the stochastic mapping of an encoder. Moreover, Poole et al. (2019) rigorously discussed the mutual information with respect to a stochastic encoder. This is common practice in representation learning context where x is data and z is a learned stochastic representation.
Lemma A.1.
I(F (x);G(x)|c) ≤ I(F (x);T (x)|c) (28)
Proof. As F (x) and G(x) are conditionally independent given the intermediate representation T (x), they formulate a Markov chain as follows: G ↔ T ↔ F . Under this relation, the following is
satisfied:
I(F (x);G(x)|c) = H(F (x)|c)−H(F (x)|G(x), c) (29) ≤ H(F (x)|c)−H(F (x)|T (x),G(x), c) (30)
= H(F (x)|c)− ∫ t,f ,g p(t, f ,g|c) log p(f |t,g, c)dtdfdg (31)
= H(F (x)|c)− ∫ t,f p(t, f |c) log p(f |t, c)dtdf (32)
= H(F (x)|c)−H(F (x)|T (x), c) (33) = I(F (x);T (x)|c) (34)
Eq. 30 is from the property of conditional entropy, and Eq. 32 is due to the conditional independence and marginalization of g.
From Lemma A.1 and Eq. 24, we can prove the Proposition 4.3.
Proof.
log(2K − 1)− Lself -s(x; {F ,G},K) (35) ≤ I(F (x);G(x)|c) (36) ≤ I(F (x);T (x)|c) (37) = I(F (x);T (x), c)− I(F (x); c) (38) = I(F (x); c|T (x))− I(F (x); c)︸ ︷︷ ︸
( )
+ I(F (x);T (x))︸ ︷︷ ︸ ( )
(39)
Strictly speaking, SelfCon-S loss does not guarantee the lower bound of either ( ) or ( ) in Eq. 39. However, SelfCon-S loss guarantees the label-conditional MI between the intermediate and the last feature, which is ( + ).
A.4 CONNECTION BETWEEN SELFCON LOSS AND CLASSIFICATION PERFORMANCE
We used the variational inference to prove that SelfCon loss is the lower bound of label-conditional MI between the intermediate and the last features; thus, we assumed a probabilistic model as Eq. 8. When the anchor feature is similar to the negative pairs (i.e., different class representations, z2:(2K−1)), this model becomes a distribution with random mapping, and SelfCon loss cannot be optimized. Therefore, representations of other classes should be farther to decrease the gap between SelfCon loss and MI. After all, SelfCon loss has improved performance because it aims to increase the lower bound of the label-conditional MI between F and G while increasing the distinction between different class representations.
B IMPLEMENTATION DETAILS
Network architectures We modified the architecture of networks according to the benchmarks. For smaller scale of benchmarks (e.g., CIFAR-10, CIFAR-100, and Tiny-ImageNet) and the residual networks (e.g., ResNet-18, ResNet-50, and WRN-16-8), we changed the kernel size and stride of a convolution head to 3 and 1, respectively. We also excluded Max-Pooling on the top of the ResNet architecture for the CIFAR datasets. Moreover, for VGG-16 with BN, the dimension of the fully-connected layer was changed from 4096 to 512 for CIFAR and Tiny-ImageNet. MLP projection head for contrastive learning consisted of two convolution layers with 128 dimensions and one ReLU activation. For the architectures of sub-networks, refer to Appendix C.
Representation learning We refer to the technical improvements used in SupCon, i.e., a cosine learning rate scheduler (Loshchilov & Hutter, 2016), an MLP projection head (Chen et al., 2020), and the augmentation strategies (Cubuk et al., 2019): {ResizedCrop, HorizontalFlip, ColorJitter, GrayScale}. ColorJitter and GrayScale are only used in the pretraining stage. We used 8 RTX 2080 Ti GPUs and set the batch size to 1024 for the pretraining and 512 for the linear evaluation. We used the batch size of 512 for ResNet-18 and 256 for ResNet-50 when pretraining on the ImageNet-100 dataset. We trained the encoder and the linear classifier for 400 epochs and 40 epochs, respectively, in ImageNet-100 benchmark. Meanwhile, we trained 1000 epochs and 100 epochs, respectively, in all other benchmarks.
Every experiment used SGD with 0.9 momentum and weight decay of 1e-4 without Nesterov momentum. All contrastive loss functions used temperature τ of 0.1. For a fair comparison to (Khosla et al., 2020), we set the same learning rate of the encoder network as 0.5. Then, we linearly scaled the learning rate according to the batch size (Goyal et al., 2017). For ImageNet-100 dataset, we used the small learning rate of 0.0625. We used 5.0 as a learning rate of the linear classifier for the residual architecture, but it was robust to any value and converged in nearly 20 epochs. Meanwhile, for VGG architecture, only a small learning rate of 0.1 converged.
1-stage training In the 1-stage training protocol, we trained the encoder network jointly with a linear classifier on the single-viewed batch. Most of the experimental settings were same as those of representation learning, but we trained the encoder for 500 epochs on CIFAR and Tiny-ImageNet dataset. We used the batch size of 512 and the learning rate of 0.8. For the cross-entropy result of ImageNet-100 dataset, we trained for 90 epochs with learning rate decay of 0.1 after 30 and 60 epochs. Here, we used the batch size 512 and the learning rate 0.2.
In the multi-exit framework, we used linear combination of loss functions for the backbone and sub-network. We used only cross-entropy loss for the backbone network, and weighted linear combinations of various loss functions (e.g., KL divergence and SelfCon-S) for the sub-network. Self-Distillation (Zhang et al., 2019a) used the interpolation coefficient α of 0.5. For the 1-stage version of SelfCon loss, we follow the coefficient form in (Tian et al., 2019b): L = LCE + βLself . We set the coefficient β = 1.0 for ResNet-18 and β = 0.8 for ResNet-50. Note that we used the outputs from the projection head instead of the logits. We used temperature τ = 3.0 for SD and τ = 0.1 for SelfCon loss.

heading: C ABLATION STUDY ON SUB-NETWORK
text: The structure, position, and the number of sub-networks are important to the performance of SelfCon learning. First, in order to find a suitable structure of the sub-network, the following three structures were attached after the 2nd block of an encoder: (1) a simple fc, fully-connected, layer, (2) small structure which reduced by half the number of layers in the non-sharing blocks, (3) same structure which is same as the backbone’s non-sharing block structure. After we found the optimal structure, we fixed the structure of sub-network and found which position is the best. For ResNet architectures, there are three positions to attach; after the 1st, 2nd, and 3rd block. For VGG-16 with BN, there are four positions and for WRN-16-8, there are two positions possible. Note that blocks are divided based on the Max-Pooling layer in VGG-16 with BN.
Table 8 presents the ablation study results for ResNet-18 and ResNet-50, and Table 9 presents the results for WRN-16-8 and VGG-16 with BN. We highlighted the selected structure and position in Table 8 and Table 9. For the ImageNet-100 dataset, we used same structure positioned after the 2nd block of ResNet-18.
Obviously, there are many combinations of placing sub-networks, and Table 8 and Table 9 presented an interesting result that some performance was the best when sub-networks are attached to all blocks. It seems that increasing the number of positive and negative pairs by the various views improves the performance. It is consistent with the argument of CMC (Tian et al., 2019a) that the more views, the better the representation, but our SelfCon learning is much more efficient in terms of the computational cost and GPU memory usage. However, for the efficiency of the experiments and better understanding of the framework, we stuck to a single sub-network in all experimental settings.

heading: D EXTENSIONS OF SELFCON LEARNING
text: D.1 SELFCON IN UNSUPERVISED LEARNING
Although we have experimented only in the supervision, our motivation of contrastive learning with a multi-exit framework can also be extended to unsupervised learning. We propose a SelfCon loss function for the unsupervised scenario and present the linear evaluation performance of ResNet-18 architecture on CIFAR-100 dataset.
Loss function Under the unsupervised setting, Chen et al. (2020) proposed a simple framework for contrastive learning of visual representations (SimCLR) with NT-Xent loss. SimCLR suggests a contrastive task that contrasts the augmented pair among other images. Therefore, the objective of SimCLR is exactly same as Eq. 1, while each sample has only one positive pair of its own augmented image, i.e., P ≡ {(i+B) mod 2B}. We denote this loss as Lsim. We fomulate SelfCon loss in unsupervised setting as SimCLR, using a positive set without label information. We formulate SelfCon loss with the multi-viewed and unlabeled batch (SelfCon-MU) as follows:
Lself -mu = − ∑ ω,ω1 ∑ i,p1 log exp(ω(xi) • ω1(xp1)/τ)∑
ω2 (∑ p2 exp(ω(xi) • ω2(xp2)/τ) + ∑ n exp(ω(xi) • ω2(xn)/τ) ) (40)
i ∈ I ≡ {1, . . . , 2B} p∗ ∈ Pi∗ ≡ {i, (i+B) mod 2B} n ∈ Ni ≡ I \ Pi2
As SelfCon loss in supervised setting, we exclude anchor sample from Pi∗ andNi to avoid contrasting the same feature, i.e., Pi∗ ← Pi∗ \ {i} when ω = ω∗ and Ni ← Ni \ {i} when ω = ω2. Here, we used τ = 0.5 for the unsupervised SelfCon loss and Lsim.
We also formulate SelfCon loss with the single-viewed and unlabeled batch (SelfCon-SU) as follows:
Lself -su = − ∑ ω,ω1 ∑ i,p1 log exp(ω(xi) • ω1(xp1)/τ)∑
ω2 (∑ p2 exp(ω(xi) • ω2(xp2)/τ) + ∑ n exp(ω(xi) • ω2(xn)/τ) ) (41)
i ∈ I ≡ {1, . . . , B} p∗ ∈ Pi∗ ≡ {i} n ∈ Ni ≡ I \ Pi2
Similarly, Ni ← Ni \ {i} when ω = ω2. For the positive set, since this loss is based on the single-viewed batch, we have an empty positive set when ω = ω∗.
Experimental results All implementation details for unsupervised representation learning are identical with those of supervised representation learning in Appendix B, except for temperature τ of 0.5 and linear evaluation learning rate of 1.0. We used a small sub-network attached after the 2nd block. Table 10 shows the linear evaluation performance of unsupervised learning on ResNet-18 in CIFAR-100 dataset. However, we empirically found that the encoder failed to converge with SelfCon-MU and SelfCon-SU loss.
D.2 SELFCON WITH ANCHORS ONLY FROM THE SUB-NETWORK
We suspect that Eq. 40 and 41 allow the backbone network to follow the sub-network, which makes the last feature learn more redundant information about the input variable, without any label information. Thus, unsupervised loss function under the SelfCon framework needs to modified.
When the anchor feature is from the backbone network, we remove the loss term which contrasts the features of sub-network. Strictly speaking, it does not perfectly prevent the backbone from following the sub-network, since there is no stop-gradient operation on the outputs of backbone network when the outputs of sub-network are the anchors. However, we hypothesize that it helps prevent the encoder from collapsing to the trivial solution by the contradiction of IB principle. We confirmed the performance of revised loss functions in both unsupervised and supervised scenarios.
Loss function Lself -mu* = Lsim − α (∑
ω1 ∑ i,p1 log exp(G(xi) • ω1(xp1)/τ)∑
ω2 (∑ p2 exp(G(xi) • ω2(xp2)/τ) + ∑ n exp(G(xi) • ω2(xn)/τ) )) (42)
i ∈ I ≡ {1, . . . , 2B} p∗ ∈ Pi∗ ≡ {i, (i+B) mod 2B} n ∈ Ni ≡ I \ Pi2
We exclude anchor sample from Pi∗ and Ni, and all notations are same as Eq. 40, except for the coefficient α where we used 1.0. For the supervised setting, simply change Pi∗ to {p ∈ I|yp = yi} and Lsim to Lsup. Note that Pi∗ ← Pi∗ \ {i} when ω∗ = G and Ni ← Ni \ {i} when ω2 = G. We get rid of the situation that the anchor F (x) contrasts the positive pair in sub-network G(xp∗). Still,
the proposed loss function includes SimCLR loss (Lsim) or SupCon loss (Lsup) in the unsupervised or supervised setting, respectively.
Lself -su* = − ∑ i,p1 log exp(G(xi) • F (xp1)/τ)∑
p2 exp(G(xi) • F (xp2)/τ) + ∑ ω2 ∑ n exp(G(xi) • ω2(xn)/τ)
(43)
i ∈ I ≡ {1, . . . , B} p∗ ∈ Pi∗ ≡ {i} n ∈ Ni ≡ I \ Pi2
For the supervised setting, we change the above equation as the equally-weighted linear combination of SupCon-S loss and Eq. 43 with Pi∗ ≡ {p ∈ I|yp = yi}. Note that we also exclude contrasting the anchor itself in SupCon-S loss term.
Experimental results In Table 10, we also reported the accuracy of SelfCon-MU and SelfCon-SU loss according to Eq. 42 and 43. Surprisingly, in this case, SelfCon-MU outperformed SimCLR loss (Chen et al., 2020), improving 1.3%p. Unfortunately, SelfCon-SU had not converged again, although it improved the result in a small amount compared to Eq. 41. While SelfCon-MU has SimCLR loss term which makes the backbone encoder still learn meaningful features, SelfCon-SU loss does not have the anchor features from the backbone, which makes the backbone hard to be trained. Table 11 summarizes SelfCon-M and SelfCon-S loss, removing the anchors from the backbone in the supervised setting, i.e., supervised version of Eq. 42 and 43. As we expected, these variants of SelfCon-M and SelfCon-S further improved the classification performance.

heading: E DETAILS OF MI ESTIMATION
text: Table 12 summarizes the full details of mutual information estimation, measured on CIFAR-100 with the pretrained ResNet-50 encoders. Note that we considered the encoder output without the projection head. We used three types of estimators: InfoNCE (Oord et al., 2018), MINE (Belghazi et al., 2018), and NWJ (Nguyen et al., 2010). As expected, SelfCon-M and SelfCon-S loss exhibited
larger MI between the intermediate and the last feature of the backbone network than CE and SupCon loss.

heading: F QUALITATIVE EXAMPLES FOR VANISHING GRADIENT
text: In Figure 5, we have already showed that the sub-network solves the vanishing gradient problem through the visualization for gradient norms of each layer. In Figure 7, we also visualized qualitative examples using Grad-CAM (Selvaraju et al., 2017). We used the gradient measured on the last layer in the 2nd block when the sub-network is attached after the 2nd block. In order to compare the absolute magnitude of the gradient, it is normalized by the maximum and minimum values of the two methods, SelfCon-M and SupCon. As in Figure 7, SelfCon learning led to a larger gradient via sub-networks, and more clearly Grad-CAM highlighted the pixels containing important information in the images.
G IMAGENET RESULTS
We experimented with the full-scale ImageNet benchmark to enhance the reliability of SelfCon learning on the large-scale dataset. Due to the limited computational resources, we used ResNet-18 architecture for both the pretraining and linear evaluation. For the pretraining stage, we experimented with various batch sizes and epochs. Besides, we tuned the learning rate as 0.125 for 1024 batch size and 0.25 for 2048 batch size. In the linear evaluation stage, we trained the linear classifier with a batch size of 256, 40 epochs, and a learning rate of 6.0. Other hyperparameters are the same as Appendix B. For the cross-entropy result, we used the batch size of 256 and the learning rate of 0.1. The settings of epochs and the learning rate rule are the same as the ImageNet-100 experiments.
We summarized the experimental results in Table 13. For the experiment with 1024 batch size and 400 epochs, it is still consistent that SelfCon-M showed better performance than SupCon, but the multi-viewed methods outperformed the single-viewed counterparts. We consider the possible reasons as follows. (1) Under-fitting issue due to the tremendous number of samples and relatively small size of architecture. (2) Insufficient number of training epochs (also refer to Figure 4). (3) Relatively small batch size with respect to the number of classes (1000) in ImageNet (also refer to Table 4). Therefore, we did two more ablation studies: (1) pretraining with 800 epochs and (2) pretraining with a larger batch size (B = 2048).
In the experiments with 800 epochs, the trend that multi-viewed methods outperformed the singleviewed counterparts did not change. However, the classification accuracy of each algorithm significantly increased as the pretraining epochs became longer. On the other hand, for the larger batch
size (i.e., B = 2048), we observed that SelfCon-S is better than SupCon. The difference seems to be marginal, but it is noteworthy that the training time for SelfCon-S is more than twice shorter than SupCon, and memory consumption is much lower. It is an important result in terms of the recent research flow for the compression in contrastive learning (Koohpayegani et al., 2020; Fang et al., 2021). From the result of SelfCon-S in the 2048 batch size, we are highly convinced that the trend in ImageNet is the same as the other benchmarks. The experiments for the larger architecture (e.g., ResNet-50, ResNet-101) or the larger batch size (e.g., 4096, 6144) are left for the future work.

heading: H MEMORY USAGE AND COMPUTATIONAL COST
text: In Table 14, we reported the computational cost for pretraining with ResNet-50. ImageNet-100 dataset result is not reported because ResNet-50 with batch size over 256 exceed the GPU limit. The overall trend is similar to that in Table 3.

heading: I SENSITIVITY STUDY FOR LEARNING RATE
text: In Table 4, we experimented with the supervised contrastive algorithms with various batch sizes and confirmed that the classification accuracy decreases in the large batch size. We supposed that this trend is induced by the regularization effect from the batch size. However, there could be a concern for using a sub-optimal learning rate on the large batch size. We further studied the sensitivity for the learning rate in a batch size of 1024 and summarized the results in Table 15.
We concluded that the performance comparison in Table 4 is consistent with hyperparameter tuning. The experimental results supported that a larger learning rate than 0.5 may be a better choice but the trend between all methods maintained in parallel with the learning rate of 0.5. Therefore, we stick to the initial learning rate of 0.5 that Khosla et al. (2020) had used.

heading: J ABLATION STUDY FOR DIFFERENT AUGMENTATION POLICIES
text: We could think of the optimal scenario where we have prior domain knowledge about a good augmentation policy. Also, there could be the opposite scenario where we do not know an optimal policy and just use the basic augmentations. Hence, we investigated three different augmentation policies as follows and compared the performances of SupCon and SelfCon-S under these scenarios.
• Optimal : We used RandAugment (Cubuk et al., 2020) for an optimal augmentation policy. RandAugment randomly samples N out of 14 transformation choices (e.g., shear, translate, autoContrast, and posterize) with M magnitude parameter. We used the optimized value of N = 2 and M = 9 in (Cubuk et al., 2020). It is also known that SupCon performs better with RandAugment policy (Khosla et al., 2020).
• Standard: For standard augmentation, we used {RandomResizedCrop, RandomHorizontalFlip, RandomColorJitter, RandomGrayscale}. This is the same basic policy we used in the paper.
• Simple: When we do not have domain knowledge, it might be difficult to choose the appropriate augmentation policies. We assumed a scenario where we might not know that the color would be important in this visual recognition task. Therefore, we removed the color-related augmentation policies from Standard policy, i.e., we only used {RandomResizedCrop, RandomHorizontalFlip} for a simple augmentation policy.
The results are presented in Table 16. Interestingly, when we apply Optimal augmentations, SupCon outperformed SelfCon-S, but for Standard and Simple augmentations, SelfCon-S outperformed SupCon. SupCon with the multi-viewed batch can benefit more from the strong and optimized augmentation policy because training each sample twice more encouraged the memorization. Meanwhile, SelfCon learning did not work well with RandAugment (Cubuk et al., 2020), as SupCon degraded with the Stacked RandAugment (Tian et al., 2020) in their experiments. There would be an optimal policy for SelfCon learning but, finding an optimal policy, such as with RandAugment or AutoAugment (Cubuk et al., 2019), is not a trivial process and needs a lot of computational cost. SelfCon learning can relieve this concern and is more robust to the simple augmentation (i.e., weak augmentation policy).

heading: K ADDITIONAL EXPERIMENT FOR CORRELATION BETWEEN SELFCON LOSS AND MUTUAL INFORMATION
text: To clearly show the correlation between the mutual information and test accuracy, we experimented with the interpolation between SupCon loss and SelfCon-M loss (SupCon loss is a special case of SelfCon-M loss). However, the current formulation of Eq. 1 and Eq. 2 cannot make the exact interpolation between SupCon and SelfCon-M because the SelfCon-M loss should have negative pairs from different levels of a network (i.e., backbone and sub-network), but the SupCon loss cannot produce those. Therefore, we should modify the SelfCon-M loss to a supervised version of Eq. 42. In the supervised version of Eq. 42, we can break down the SupCon loss term and SelfCon-like loss
term, with an interpolating parameter α: Lself -m∗ = Lsup − α (∑
ω1 ∑ i,p1 log exp(G(xi) • ω1(xp1)/τ)∑
ω2 (∑ p2 exp(G(xi) • ω2(xp2)/τ) + ∑ n exp(G(xi) • ω2(xn)/τ) )) (44)
i ∈ I ≡ {1, . . . , 2B} p∗ ∈ Pi∗ ≡ {p ∈ I|yp = yi} n ∈ Ni ≡ I \ Pi2
where Pi∗ ← Pi∗ \ {i} when ω∗ = G and Ni ← Ni \ {i} when ω2 = G. Therefore, if α = 0, Lself -m∗ is equivalent to the SupCon loss and if α = 1, Lself -m∗ is almost equivalent to SelfCon-M loss. Figure 8 describes the estimated mutual information and its relationship with classification performance via controlling the hyperparameter α, and Table 17 summarizes the detailed estimation values of the intermediate feature with respect to the input, label, and the last feature. We observed a clear increasing trend of both MI and test accuracy as the contribution of SelfCon gets larger. Also, the detailed MI estimation values in Table 17 imply the same interpretation as the IB principle and Appendix E.

Review summaries:
Recent contrastive learning-based algorithms proposed various ways to make representations from semantically similar (positive) samples closer and those from semantically different (negative) samples farther. While they have shown remarkable performance improvement, they usually require a multi-viewed batch for defining the positive samples. In this paper, the authors proposed Self-Contrastive (SelfCon) Learning which self-contrasts within multiple outputs from the different levels of a multi-exit model. The authors theoretically demonstrated that SelfCon loss guarantees the lower bound of label-conditional mutual information (MI) between the intermediate and the last features. Then, they empirically showed the proposed method significantly outperforms the cross-entropy and Supervised Contrastive Learning (SupCon) baselines.

The paper proposed a contrastive learning framework to remove the requirement for additional data augmentation techniques for creating positive pairs, by leveraging the idea of multi-exit network and conduct "self-contrasts" among multiple outputs of the model. Theoretical analysis on the designed loss shows that the proposed SelfCon loss is a lower bound of MI between the intermediate and last feature. Extensive experiments and ablation studies demonstrate that the proposed method has better performance than than the supervised contrastive baseline.

The authors propose an improved algorithm for supervised contrastive learning, which they term “self-contrastive learning”. Instead of only contrasting representations at one point in the network, self-contrasting uses readouts at multiple stages in the network (remapped into an appropriate feature space by an additional readout network). This makes it possible to propose two variants: A “multi-batch” variant that first augments the images, and then contrasts views between samples of the same class, both augmented and non-augmented, and a “single-batch” variant that only contrasts between samples of the same class. Improvements are shown for CIFAR-10,100, TinyImageNet, as well as a smaller version of ImageNet with 100 classes, referenced as ImageNet-100.