Self-Contrastive Learning
Authors: Sangmin Bae
Sungnyun Kim
Jongwoo Ko
Gihun Lee
SeungJong Noh
Se-Young Yun
Venue: ICLR 2022 Submitted (2022)
Keywords: contrastive learning
representation learning
image classification
mutual information

Abstract: This paper proposes a novel contrastive learning framework, called Self-Contrastive (SelfCon) Learning, that self-contrasts within multiple outputs from the different levels of a multi-exit network. SelfCon learning does not require additional augmented samples, which resolves the concerns of multi-viewed batch (e.g., high computational cost and generalization error). Furthermore, we prove that SelfCon loss guarantees the lower bound of label-conditional mutual information between the intermediate and the last feature. In our experiments including ImageNet-100, SelfCon surpasses cross-entropy and Supervised Contrastive (SupCon) learning without the need for a multi-viewed batch. We demonstrate that the success of SelfCon learning is related to the regularization effect associated with the single-view and sub-network.

Under review as a conference paper at ICLR 2022
SELF-CONTRASTIVE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
This paper proposes a novel supervised contrastive learning framework, called Cell-Contrastive (SelfCon) Learning. that self-contrasts within multiple outputs from the different levels of multi-exit network. SelfCon learning does not require additional augmented samples, which resolves the concerns of nnwwee batch (e.g., high computational cost and generalization error). Furthermore, we prove that SelfCon loss guarantees the lower bound of label-conditional mutual informa- tion between the intermediate and the last feature. In our experiments including ImageNet- 100. SelfCon surpasses cross-entropy and Supervised Contrastive (Sup- Con) learning without the need for a multi-viewed batch. We demonstrate that the success of SelfCon learning is related to the regularization effect associated with the single-vies view and sub-network.
INTRODUCTION
Recent studies have studied the success of deep neural networks by investigating how neural networks can encode representations with rich information (Tishby & Zaslavsky, 2015; Shwartz- Ziv & Tishby, 2017; Hjelm et al., 2018; Saxe et al., 2019a). Among the various approaches suggested, contrastive loss functions, designed to maximize the lower bound of mutual information (MI) between the target and context, have achieved considerable success self-supervised representation learning first (Gutmann & Hyvärinen, 2010; Oord et al., 2018) and supervised learning recently (Khosla et al., 2020; Gunel al., 2020; Wang et al., 2021). The main objective of the contrastive loss functions in supervision is to make representations from the same class closer and representations from different classes farther. To this end, they define positive samples, i.e., augmented samples from the same image or (augmented) images sharing the same class label, and negative samples, i.e., all other samples, for every data batch.
Contrasting two random augmented samples, often referred to multi nieti- batch, has shown impressive results in representation learning (Chen et al., 2020; He et al., 2020; Grill et al., 2020; Caron et al., 2020; Chen & He, 2020; Khosla et al., 2020), yet multi-viewed batch causes the following issues. (1) A multi-viewed batch doubles the batch size, which a huge burden on memory and computation. (2) Oracle needs to carefully choose the augmentation policies (Chen et al., 2020; Tian et al., 2020; Caron et al., 2020; Kim et al., 2020). (3) A multi-viewed contrastive task is domain-specific, because data-level augmentation requires specific domain knowledge such as image cropping and flipping (Verma et al., 2021). In spite of these concerns, a supervised contrastive learning framework (Khosla et al., 2020) still relies on multi-veews, although they are not necessarily needed with the help of usable label information.
In this work, we propose novel contrastive learning framework in supervision, called Self- Contrastive (SelfCon) learning, that does not require additional augmented samples. Instead, SelfCon uses multi-exit framework (Teerapittaannon al., 2016; Zhang et al., 2019a;b) where sub-networks produce multiple outputs for the same input and self-contrasts within multiple out- puts from the different levels of a single network. For training, SelfCon learning can use the multi-viewed batch as well. However, the multi-exit framework already generates positive pairs from a single image, which can replace the augmentation based multi-view. In Figure 1, we compare SelfCon learning and Supervised Contrastive (SupCon, Khosla al. (2020) learning.
SelfCon learning improves the classification performance of the encoder network, owing to (1) the increase of the lower bound of label-conditional MI between the intermediate and the last features, and (2) the regularization effect from the single-view and sub-network. We thcoreti- cally demonstrate that SelfCon loss is the lower bound of label-conditional MI. Furthermore, unlike

---

Under review as a conference paper at ICLR 2022
Anchor
Backhone (F)
Positive
Sub (G)
Negative
SupCon
(b) SelfCon Multi-View View
(c) SelfCon Single-View
Figure 1: Comparison of learning framework in terms of augmentation and architecture. Both SupCon (Khosla et al., 2020) and SelfCon use all the samples of the same ground-trutt label as the positive pairs. In all three methods, every output can be an anchor feature. Specifically, in (b) and (c), an anchor from the backbone network contrasts other features from the backbone, as well as the features from the sub-network. Best seen in color.
SupCon loss, it encourages the encoder learn the label information that only intermediate features can explain. Second, we empirically show that SelfCon learning reduces the generalization error with the single-view and sub-network. The former prevents the encoder from overfitting to each instance, and the latter regularizes the intermediate feature to be similar to the last feature.
The contributions of our paper can be summarized follows:
[S3] We propose Self-Contrastive learning, a novel supervised contrastive framework between the multiple features from the different levels of a single network.
[S4] SelfCon loss guarantees the lower bound of label-conditional MI between the intermediate and the last features, and leads increasing interaction information between the features and the label,
1S5.1, S5.2] SelfCon learning achieved higher classification accuracy for various benchmarks and architectures than cross-entie cross-entropy and SupCon loss. Our empirical study of MI estimation provides evidence for superior performance.
[S5.3, $5.4] We investigate the advantages single-viewed batch in terms of the generalization error and computation resources. We also identify that SelfCon learning benefits from the sub-network owing the regularization effect, vanishing gradient, and ensemble prediction.
RELATED WORKS
2.1 CONTRASTIVE LEARNING
As the cost of data labeling increases exponentially, there is an increased need for acquiring repre- sentation under unsupervised scenarios. After Oord al. (2018) proposed InfoNCE loss (also called a contrastive loss), contrastive learning-based algorithms showed a remarkable improvement in performance (Chen al.. 2020; He et al., 2020; Grill et al., 2020: Caron et al., 2020; Chen He, 2020). Most previous works use the structure of Siamese network, which is weight-sharing network applied on two or more inputs, with negative pairs (SimCLR (Chen al., 2020), MoCo (He et al., 2020)), momentum encoders (MoCo (He et al., 2020), BYOL (Grill et al., 2020)), online clustering (SwAV (Caron et al., 2020)), or a stop-gradient operation (SimSiam (Chen & He, 2020)). While the softmax form is frequently used for the contrastive loss (Chen al., 2020; He et al., 2020), recent tfate-of-the-a algorithms utilize an MSE loss (Grill et al., 2020; Chen & He, 2020), cross- rross-entropy loss (Caron et al., 2020: 2021). Khosla al. (2020), inspired by the success in self-supervised learning, propose a label-based contrastive loss in supervision, named SupCon loss. Supervised contrastive learning has also been extended to semantic segmentation (Wang et al.,
2

---

Under review as a conference paper at ICLR 2022
2021) and language tasks (Gunel et al., 2020). While the SupCon loss utilizes the output features from two randomly augmented images, we also contrast the features from different network paths by introducing the multi-exit framework (Teerapitayanon et al., 2016; Zhang et al., 2019a:b). We further propose a novel loss function that we can apply on the single-viewed hatch.
2.2 MUTUAL INFORMATION
Mutual information is measure quantify the information held in a random variable about the other variable, and it has been used as powerful tool to open the black box of deep neural networks (Tishby 7aslavsky, 201.5; Shwartz-Ziv Tishby, 2017; Saxe et al., 2019a). As is difficult to compute the MI exactly (Paninski, 2003), several works have proposed variational MI estimators based on neural networks, c.g., InfoNCE (Oord et al., 2018), MINE (Belghazi et al., 2018), NWJ (Nguyen et al., 2010), ML-CPC (Song& & Ermon, 2020), and SMILE (Song & Ermon, 2019). Recently, MI estimator-based objectives have been proposed to improve the performance of contrastive learning (Hjelm et al., 20:8; Bachman et al., 2019; Song & Ermon, 2020; Wu et al., 2020b) and knowledge distillation (Tian et al., 2019b; Ahn et al., 2019). Among previous approaches, we are highly motivated by those that aim to increase the MI between the intermediate and the last features. DIM (Hjelm al., 2018) and AMDIM (Bachman et al., 2019) propose novel contrastive losses between the global features and local features (i.c., all pixels or patches of the intermediate features), while we contrast the refined local features via sub-networks VID (Ahn et al., 2019) makes the student learn the distribution of the activations in the auxiliary teacher's intermediate features. Our method differs from VID in that we self-distill within the network itself. Zhang et al. (2019a;b) share a similar idea with the aforementioned works, but they implicitly increase MI using Kullback-Leibler (KL) divergence loss.
SELF-CONTRASTIVE LEARNING
We propose a new supervised contrastive loss that maximizes the similarity of the outputs from different network paths by introducing the multi-exit framework. We define the encoder structure with F as a backbone network and as sub-network that shares the backbone's 's parameters up to some intermediate layer. T denotes those sharing layers that produce the intermediate feature. The sub-network has additional non-sharing parameters attached after T. Note that F and G include the projection head on the encoder. We highlight the positive and negative pairs with respect to an anchor sample, following Figure 1.
To mitigate the weaknesses of cross-entropy, such as the reduced generalization performance and the possibility of poor margins, Khosla et al. (2020) proposed a supervised version of the contrastive loss that defines the positive pairs as every sample with the same ground-truth label. We reformulate the SupCon loss function as follows:
SupCon loss
cxp(F (z.) (Ip,,) )/T)
Coup- = -
Σ log
(1)
Σ exp F(x:)
F (zp))/7) + Σ exp( F(x,) F(a,)/r)
(,p1
P2
i 1 = [1, ...... 2B}
p+ E P..
(iH m
n E N = {n 1yn y
where denotes the inner product, B is the batch size, and T is the temperature to soften or harden the softmax value. I denotes set of indices of the multi-viewed batch that concatenates the original B images and the augmented ones, i.e., B4 is an augmented pair of æ;, P.. and N₂ are set of positive and negative pair indices with respect an anchor i, respectively. Note that Eq. is the same as NT-Xeet loss (Chen et al., 2020) when P.. {ii + B) mod 2B). We dropped the dividing term of sums for brevity.
SelfCon loss Besides minimizing the above loss, we aim to maximize the similarity between the outputs from the backbone and the sub-network. To this end, we define SelfCon loss, which forms a self-contrastive task for every output including the features from the sub-network. In Section 4, clarify the connection of this loss with the conditioon MI between the intermediate and the last features.
3

---

Under review as a conference paper at ICLR 2022
exp(w(axi) w (1tp
Lself =
ΣΣ
log
(2)
W,W1
Σ
Σ (x,)(w(x)) w2 (2 //T Σ exp(w((x:) w2(
W2
P2
{1.
B} (SelfCon-S)
P. E P. (p E 1 Up y/3 E, = (n ym F yi
iEl=
11
2 (SelfCon-M)
w, w1, We E Ω {F, L.F,G G), a function set of the backbone network and the sub-network. We exclude an anchor sample from the positive set to avoid contrasting the same feature, i.c., P.. + P.. {i} when w = w.. While prevalent contrastive approaches (Khosla et al., 2020; Chen et al., 2020; He et al., 2020; Grill et al., 2020; Caron et al., 2020) force multi-viewed batch generated by data augmentation, in SelfCon learning, the sub-network plays a role as the augmentation and provides an alternative view on the feature space. Therefore, viewed batch (SelfCon-S); = {l.... ......... }} without the additional augmented samples. Besides, we also define SelfCon with multi- viewed batch (SelfCon-M) loss, i.c., Leely with {L..... 2B}.
formulate our SelfCon loss function with single-
In the development of our SelfCon loss formulation, we can further use multiple sub-networks, ie.. = {F, G1, ..,... Appendix C provides the classification performance of the expanded network, which was comparable to that of the single sub-network in our experiments. Thus, we simply use a single sub-network throughout our paper.
DISCUSSIONS
Question 1: How does SelfCon loss maximize the lower bound of label-condiiional MI between the intermediate and the last features?
We explain in the following propositions answer Question All the proofs can be found in Appendix A.
Let x and z be different samples that share the same class label c. Then, with some discriminator function modeled by neural network F and 2(K -1) negative sample size, SupCon loss maximizes the lower bound of conditional MI between the output features of a positive pair,
Proposition 4.1.
log(2K - Lou 23, F, K) [()))))(()))))
Proposition 4.2. Denote fiseffs as SelfCon loss with single-viewed batch. SelfCon-S loss maximizes the lower bound of MI between the oulput features from the backbone and the sub-network,
12g(2K-
Ceelf-s(2;; G,,G), S II((()); G(æ)]c).
(4)
Proposition 4.3. Let Txx) be the intermediate feature of the backbone network, which is also an input to the auxiliary network path. Then the rh.s. of Eq. 4 is upper- bounded by
upper-bounded
I(F(z); I[F(x);; I(F(x):c IIF T(x))
(5)
Proposition 4.1 and 4.2 can be derived from the exact bound of InfoNCE (Poole et al., 2019). SupCon and SelfCon-S loss have 2(K - negative sample size because of the augmented negative pairs for SupCon and the sub-network features for SelfCon-S. Note that SelfCon-M loss has similar upper bound as Eq. 3 and Eq. 4 (refer to Appendix A.2).
In Proposition 4.3, is an interaction information (Yeung, 1991) that measures the influence of T(x) on the amount of shared information between F(x) and e. SelfCon-S loss is related to increasing this interaction information, so that the intermediate feature enhances the correlation between the last feature and the label, which may result in improving the downstream classification tasks. In addition, (Q) implies that SelfCon loss increases the MI between the intermediate and the last features in the backbone. T(x) is distilled from F(x) that has richer class-related information and consequently, earlier layers are trained to produce better representation (Zhang et al., 2019a; Ahn et al., 2019). We believe this is theoretical evidence for the improved performance SelfCon. Also, refer Appendix A.4 for the connection to the classification performance of the SelfCon loss.
4

---

Under review as a conference paper at ICLR 2022
Table 1: The results of linear evaluation on ResNet-18 and ResNet-50 for various datasets. In the supervised setting, we compare our SelfCon-M and SelfCon-S with cross-entropy (CE) loss, supervised contrastive loss with multi-view (SupCon), and without multi-view (SupCon-S), Bold type is for all the values of which the standard deviation range overlaps with that of the best accuracy.
ResNet-18
ResNet-50
Method
Single-View
Single View
CIFAR-100
Tiny-ImageNet
CIFAR-10
CIFAR-100
Tiny-ImageNet
CE
94.7....1
72.9
57.3
94.9
74.8
62.3
SupCon SupCon-S"
94. 2...
73.0
56.910.1
95.610.1
75.31n.2
61.610.a
94.9g...
73.9
58. 28.4
95.8
76.7
62.0
SelfCon-M (ours)
95.0+a.1
74.9+0.1
59.2 09.2+t..
95.6+0.1
76.9
63. 63.0+0.2
SelfCon-S (ours)
95. 35.3
75.420.1
59 59.8
95. 19.7:00
78 78.5
63. 20.2
"We have re-implemented SupCon method (Khosla et al., 2020) and also run their official code for credibility, the accuracy was slightly lower than their reported numbers.
"SupCon-S sets I as (1, B} in Eq. 1. Although Khosla et al. (2020) did not propose the version of the single-view, we implemented SupCon-S since it is worth investigating the effect of multi-viewer batch.
Question 2: Is applicable unsupervised representation learning?
Good representations should get rid of redundant input information and extract meaningful features (Tishby & Zaslavsky, 2015). However, in Eq. 2, the backbone network can be an anchored function where the contrastive features are those from the backbone as well as the sub-network. Unfortunately. this might allow the last feature follow the intermediate feature. learning redundant information about the input. This could be why SelfCon learning might not work in an unsupervised environment where there no label information (refer to Appendix D.1). However, Appendix D.2, we suggest that blocking the backbone from following the sub-network, i.e., removing the term in Eq. 2 where w = w. G, can mitigate the problem.
Supervised contrastive framework, which is the main interest of our paper, is away from the above problem owing to the label information. Note that the unsupervised version of SelfCon-S loss guarantees only the lower bound of (D) in Proposition 4.3. Meanwhile, on label, encourages the backbone network to be in accordance with label information. Thus, we believe that jointly maximizing the lower bound of (7+ offers evidence for the success of supervised SelfCon learning.
induced by the condition
EXPERIMENT
We presented the image classification accuracy for standard benchmarks, such as CIFAR-10, CIFAR- 100 (Krizhevsky et al., 2009), Tiny-ImageNet (Le & Yang, 2015), ImageNet-100 (Tian al., 2019a), and ImageNet (Deng et al., 2009), and extensively analyzed the results. We summarized the mean and standard deviation of t accuracy over three random seeds for the reliability of the experimental results.
We used the optimal structure and position of the sub-network for all the network architectures (see Appendix C). We compared three types of the sub-network: the same structure with non-sharing layers of the backbone network (same), structure with the number of non-sharing layers reduced in half (small), and a simple fully-connected layer (fc). The overall performance was comparable or better than the basclines. We observed trend: in shallow networks the same structure was better, while fc was better in deeper networks. Besides, the performance was consistently good when the exit path is attached after the midpoint of the encoder (e.g.. 2nd block in ResNet or 3rd block in VGG architecture), Complete implementation details are in Appendix B.
5.1 REPRESENTATION LEARNING
We measured the classification accuracy the representation learning protocol (Chen et al., 2020), which consists of 2-stage training, (1) pretraining an encoder network and (2) fine-tuning on a linear classifier with the frozen encoder (called linear evaluation).
5

---

Under review as a conference paper at ICLR 2022
Table 2: The classification accuracy on ResNet-18 and ResNet-50 for ImageNet- 100. Parentheses indicate the performance of ensemble prediction in the multi-exit frame- work (refer Section 5.4). We reported the results with one random seed. Note that Acc @5 shows the same trend of Acc@1. 1.
00
M Febrator
-20
(ML WWW. PUSSAAA RMON MMOO
Method
ResNet-18
RosNot-50
Supton Sellionll Al SuffCon-5
CE
81.1
82.3
SupCon
86.3
Figure 2: Test accuracy and the estimated mu- tual information of different methods. Mutual information estimators are measured between the intermediate and the last features.
SupCon-S
83.5
86.8
SelfCon
83.9(aa.u)
(6.8
SetSCon-S
84.5 (e6.
87. 87.8(88
The classification accuracy is summarized in Table Interestingly, the loss functions in the single- viewed batch, such as SupCon-S and SelfCon-S, outperform their multi-view counterparts in all settings. Furthermore, our SelfCon learning, which trains with the sub-network, shows higher classification accuracy than CE and SupCon. The effects of the sub-network are analyzed in Section 5.4. We also experimented on the ImageNet- benchmark, of which 100 classes were randomly sampled (Tian et al., 2019a) to verify that SelfCon learning has the same effect on large scale datasets. Table 2 presents the consistent performance improvement of SelfCon learning. Refer to Appendix G for the ImageNet results.
5.2 MUTUAL INFORMATION ESTIMATION
Minimizing the SelfCon loss is highly related to maximizing the MI between the intermediate and the last features (Proposition 4.3). To empirically confirm this claim, we estimated MI using various estimators: InfoNCE (Oord et al., 2018), MINE (Belghazi et al., 2018), and NWJ (Nguyen et al., 2010). Specifically, we extracted the features of the CIFAR-I00 dataset from the pretrained ResNet-50 encoders and computed the estimators. Then optimized simple 3-layer Conv-ReLU network with the MI estimator objectives. We consider the encoder output without the projection head, which differs from Section 3.
In Figure 2, SelfCon-M and SelfCon-S both show high (x)):) T(:)), which supports our claim, while the values of CE and SupCon lower. Recall that T(x:) is the intermediate feature of the backbone network, which is an input to the auxiliary network path. Although [=F(w); T(x)) does not explicitly guarantee an increase in accuracy, our results imply that it has positive correlation with the test accuracy of the backbone network. We believe that the richer information in earlier features makes the encoder output better representation because the intermediate feature also the input for the subsequent layers.
We also found that SelfCon learning increases the information between T(x) and the label, implying that the intermediate feature is imbued with class-related knowledge, while the information between T(x) and the input æ is decreased (refer to Table in Appendix E). This suggests that SelfCon learning is in agreement with IB principle of training deep neural network, i.e., fitting-compression phase (Saxe et al., 2019a). Furthermore, refer to Appendix for the additional empirical evidence of correlation between MI and improved performance.
5.3 MULTI-VIEW VS. SINGLE- VIEW
We compare the multi-view and single-view in terms of generalization error, efficiency, and batch size.
Single-view reduces generalization error. In Figure 3, SupCon shows higher train accuracy but lower test accuracy than SupCon-S, and the same trend is observed with SelfCon-M and SelfCon-S. Compared to single-view, multi-view from the augmented image makes the encoder amplify the memorization of data and result in overfitting to each instance. Figure shows that SelfCon-S gradually enhanced the generalization ability, while SelfCon-M or SupCon achieved little gain in test accuracy despite the fast convergence.

---

Under review as a conference paper at ICLR 2022
100
100
80
100
70
Trai Acounsy (50
90-
88
Kotntty pint
Train Acountoy (S)
98
Icemony (oo)
Trai Accuras (S)
30
87
94
94
74
8
96-
Thain Aco. M
Test Acc. Test Apc.
Train Apc. MM
Tost Aco. -M
Train Acc, MM Tram AOG.
Tost Aco. +4 Acc. -3
Train Aco. -5
Tram ADc.
Adc. -5
SupCon
Set/Con
SupCon
SefCCon
SupCon
SallCon
(a) CIFAR-I0
(b) CIFAR-100
(c) Tiny-ImageNet
Figure 3: Train accuracy and test accuracy of ResNet-18 for different views and loss functions. The train and test accuracy are measured with linear classifier during the linear evaluation. The axis on the left and right denotes the train accuracy and test accuracy, respectively.
Table 3: Memory (GiB / GPU) and computation time (sec step) comparison. All numbers measured with ResNet-18 18 training on RTX 2080 Ti GPUs and Intel i9- 90910X CPU. Note that FLOPS is for one sample. B stands for batch size. For the results of ResNet-50, see Appendix H.
256
=
1021
Dataset (Image size)
Method
FLOPS
Memory
Time
Memory
Time
Memory
Time
SupCon
1 M
L.IG
2.14
0.13
2.35
0.16
3.18
0.27
CIFAR-100 (32x32)
SelfCion S
11.89 M
0.56
1.83
0.13
2.03
0.14
2.54
0.18
Tiny ImageNet (64x64)
SalfCon-S
SupCon
11.89 M
0.56
L.13G
1.75
0.14
0.13
2.69
2.05
0.17
0.13
2.68
0.31
0.18
SupCon
ImageNet- 100 (224x224)
SelfCon
S
11.90 M
1.82 G
2,54
0.35
3,38
0.70
5.67
1,38
Single-view is efficient in terms of memory usage and computational cost. In Table 3, compared SupCon and SelfCon-S to observe the efficiency of the single-viewed batch. SelfCon-S requires a larger number of parameters owing the extra sub-network but is more efficient in memory and computation. In both SupCon and SelfCon-S, the same batch size implies the same number of anchor features; however, they differ in memory consumption due to the data augmentation of the multi-viewed batch.
Multi-view advantageous for small batch size. In supervised learning, large batch size reduces the generalization ability, which results in decreasing the performance (You et al., 2017; Luo et al., 201 8; Wu et al., 2020a). We examined whether the perfor- mance in supervised contrastive framework is also dependent on the batch size. Table 4 summarizes the results. The multi-viewed method, e.g., SupCon, which doubles the effective number of training data, outperformed the single-viewe viewed counterpart in 64- batch experiments; the opposite was observed in all other batch sizes. For small batch size such as 64, doubling the effective batch size can make the learning stable, but as batch size gets larger the stabilizing effect from multi-views decreases and the necessity regularization appears to be critical. Hence, the optimal batch size of single-viewed methods
Table CIFAR-100 results on ResNet-18 with various batch sizes. We omitted the stan- dard deviation due to the lack of margin.
Batch Size
Method
64
128
256
512
1024
CE
74.9
74.9
74.1
73.3
72.9
SupCon
SupCon S
73.6
75.3
75.0
74.0
73.9
SelfCon-M
75.8
75.9
SolfCon-S 74.0
76.6
77.0
75.8
75.4
higher (SupCon-S vs. SupCon and SelfCon-S vs. SelfCon-M) because single-view itself helps regularization (i.e., small batch with single-view might lead to under-fitting Also, note that SelfCon learning with sub-network still surpasses the SupCon counterpart. Refer to Appendix for the sensitivity study of the learning rates.
5.4 WHAT DOES THE SUB-NETWORK ACHIEVE?
Regularization effect SelfCon loss regularizes the sub-network to output the similar features to the backbone network. It prevents the encoder from overfitting to data, and it is effective in multi-viewed as well as single-viewer batch. In Figure 3, we confirmed the regularization effect (i.e., lower train accuracy, but higher test accuracy) by comparing each bar of the same color. The strong regularization of the sub-network helped SelfCon (-M, -S) outperform the SupCon counterparts. This trend is also observed in Table 4 and Figure 4.
7

---

Under review as a conference paper at ICLR 2022
Convt
Cond
Conut
Domu?
Donua
Donal
2 Block
g Plakk
-03
800% 8-0080
3 Book
y Rlock
BupDon
Block
SelCCnnSS
100 200 000 400 aão no0 obo slo 1003
Epocts
Figure 4: CIFAR-100 accuracy at different training epochs. The solid line for train accu- racy and the dashed line is for test accuracy. ResNet- is used.
Figure 5: Gradient norm of each ResNet-11 block and con- volutional layer. We computed gradients from the SupCon loss (Leff) and SelfCon-M loss (Right), both from the same initial- ized model. All convolution layers the block are named by order.
Table 5: Classification accuracy with the classifiers after backbone, sub-network, and the en- semble of them. The encoder is pretrained by SelfCon-S loss function.
ResNet-18 18
ResNet-50
Method
CIFAR-10
CIFAR-100
Tiny-ImageNet
CIFAR-10
CIFAR 100
Tiny-ImaeeNet
Backbone
95 3
75.4+0.1
59.8_0.4
95.7 2
78.5+0.2
63.7+0.2
Sub-network
92.640.1
69.1
53.5-0
93.6=0.2
73.320.3
58.920.6
Ensemble
95.2=0.1
77.4+0.0
62.2.0.3
95.5+0.1
80.0=0.a
65.7
Vanishing gradient SelfCon learning can send more abundant information to the earlier layers through the gradients flowing from the sub-networks. Previous works (Lee et al., 2015; Teerapit- layanon et al., 2016; Zhang et al., 2019a) point out that the success of the multi-exit framework is owing solving the vanishing gradient. We showed that the same argument applies our SelfCon learning. Note that the sub-new uut-network is positioned after the 2nd block of the backbone. In Figure 5, larger gradient flows up the earlier layer in the SelfCon-M, while a large amount of the SupCon loss gradient vanishes. In particular, there is significant difference in the gradient norm in the 2nd block.
Ensemble with sub-network The sub-network in SelfCon learning can also be used on downstream tasks such as image classification. our previous experiments, we followed the linear evaluation protocol, fine-tuning classifier with the frozen backbone network. The network pretrained on SelfCon learning has an exit path, which is similarly allowed to be frozen and used as linear evaluation. We thus demonstrated two additional types of linear evaluation scenarios: (1) fine-tuning a classifier after the sub-network output and (2) fine-tuning two classifiers after the backbone and the sub-network and ensembling could make appropriate, although not the best, predictions as the backbone did, while ensembling their predictions is found to be the most powerful trick we have come up with.
classifiers' predictions. Table 5 indicates that the sub-network
Feature-level multi-view One of the advantages of SelfCon learning is to relax the dependency on multi-viewed batch. This is accomplished by the multi-views on the representation space made by the parameters of the sub-network. In Figure gradient of SelfCon-S loss with respect to the intermediate layer the backbone network, right before the exit path. Both networks focus on similar, but clearly different pixels of the same input image, implying that the sub-network learns another view in the feature space. As multi-view in contrastive learning requires speeifc augmentation, recent studies have explored the domain -gnnostic way of augmentation (Lee et al., 2020; Verma et al., 2021). SelfCon might be an intriguing future work in that auxiliary networks could be an efficient substitute for data augmentation.
visualized via Grad-CAM (Selvaraju et al., 2017) the
5.5 ABLATION STUDY
Different encoder architectures We experimented with other architectures: VGG-16 (Simonyan Zisserman, 2014) with Batch Normalization (BN) (loffe & Szegedy, 2015) and WRN-16-8 (Zagoruyko & Komodakis, 2016), and the results are presented in Table 6. The classification accuracy
8

---

Under review as a conference paper at ICLR 2022
1.0
0.8
0.6
0.4
02
0.0
Figure 6: Visualizations for the feature-level multi-view generated by the sub-network. Along with the original image, each map visualizes the gradients from the sub-network (Left) and the backbone network (Right), respectively. We measured the gradient of the pretrained ResNet-18 with S loss.
Table 6:
The results of linear evaluation on WRN-16-8 and VGG-16 with BN for various datasets. We tuned the best structure and position the sub-network for each architecture. Appendix C summarizes the implementation details.
WRN-16-8
VGG-16 wih BN
Method
Single-Vies View
CIFAR-10
ImageemageNet
CIFAR-10
CIFAR
Imageem
CE
9-1622..1
73.6
56.520.5
93.8
71.2ga.2
60.7
SupCon
95.3+0.0
75.1
57.4+0.3
93.6 03.6+0.1
69.6+0.1
57.3+0.4
SupCon-S
95.2g0.1
76.0
57.310.5
93.8
71.110.0
58.4
SelfCon-M (ours)
95 05.410
75.610.
58. 1
93.4g0
71.710.
59.410.1
SelfCon-S (ours)
95. 95.6+0..0
76 76.6
59 59.3--.22
93.5+0.11
72.0=o.0
60. 60.7=0.11
for WRN-16-8 showed a similar trend as that of ResNet architectures. However, for VGG-16 with BN architecture, SupCon had lower performance than CE on every dataset. Although the contrastive learning approach does not seem to result in significant changes for the VGG-16 with BN encoders, SelfCon-S was better than or comparable CE.
1-stage training 1-stage training framework, i.e., decoupling the encoder pretraining and linear evalua- tion, on the single-viewed batch, standard for super- vised learning. With a multi-exit framework, Zhang (2019a) propose Self-Distillation (SD), which distills logit information within the network itself, i.e.,
Table 7: CIFAR-100 1-stage training re- sults on ResNet architectures. i describes a modification to 1-stage training with multi-exit framework. We omitted the stan- dard deviation. Parentheses indicate the sub-network's accuracy.
LsD = aLce (1 LKI, We replaced the KI. divergence term with L in Eq. 2, and distilled the features from the projection head, instead of the logits. From Table 7, observed that adding cross-entropy loss to the sub-network (CE w/ Sub) improved the back- bone network's is classification performance. However, the results of SD suggested that there is a saturation of SelCCon-S (2-stage)
Method
ResNet-18
ReaNes-50
CE
72.9
71.88
CF w/ Sub
73.5
76.2(12.3)
SD
73.5(71.8)
76. (va.s)
SellCinns!
74.5
76.8,72
75.4
78.5
increase even when the classifier of sub-network better converged. Meanwhile, we hypothesize that feature distillation of SelfCon-S makes the encoder learn better representation than logit distillation of classifiers. We would like to highlight that SelfCon loss 2-stage training still demonstrated the best classification accuracy, as Khosla et al. (2020) argued that representation learning mitigates the poor generalization performance of CE-based 1-stage training.
CONCLUSION
We proposed SelfCon learning, which self-contrasts the features from the multiple levels of network. SelfCon learning is free from the issues that multi-viewed batch triggers. We found that SelfCon loss maximizes the lower bound of label-conditional MI between the intermediate and the last features, and it is related to improving the classification performance. In addition, analyzed why SelfCon is better by exploring the effect of single-view and sub-network. We verify by extensive experiments, including ImageNet- 100 and ImageNet, that SelfCon-S loss outperforms CE and SupCon.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9163-9171, 2019.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprint (rXXi:9906000900 2019.
David Barber and Felix Agakov. The im algorithm: variational approach information maximiza- tion. Advances in neural information processing systems, 16(320):201, 2004.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair. Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. arXiv preprint UrXiv:/801.04000 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006. 098822 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. arXiv preprint arXiv:2104. 14294, 2021.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp- 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. arXiv preprint wrXiv:2011.10561 2020.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 113-123, 2019.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. leee, 2009.
Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. Seed: Self-supervised distillation for visual representation. arXiv preprint 1oXiv:2001.0433. 2021.
Ziv Goldfeld, Ewout van den Berg, Kristjan H Greenewald, Igor Melnyk, Nam Nguyen, Brian Kingsbury, and Yury Polyanskiy. Estimating information flow in deep neural networks. In ICMI., 2019.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint 17Xiv1700.02660 2017.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Berarro Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent: new approach to self-supervised learning. arXiv preprint arXiv:2006.07733. 2020.
Beliz Gunel, Jingfei Du Alexis Conneau, and Ves Stoyanov. Supervised contrastive learning for pre-trained language model fine-tuning, arXiv preprint arXiv:201 1.t::0001.01033 2020.
10

---

Under review as a conference paper at ICLR 2022
Michael Gutmann and Aapo Hyvärinen. contrastte estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297-304. JMLR Workshop and Conference Proceedings, 2010.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9729-9738, 2020.
R Devon Hjelm, Alex Fedorov, Samuel
Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 1808.06670, 2018.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint arXiv:2004. 13362, 2020.
Sungnyun Kim, Gihun Lee, Sangmin Bae, and Se- Young Yun. Mixco: Mix-up contrastive learning for visual representation. arXiv preprint arXiv:2010.06300, 2020.
Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash. Compress: Self-supervised learning by compressing representations. arXiv preprint arXiv:2010. 10X1::0000.14713, 2020.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7:7,
Chen-Yo Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply nets. In Artificial intelligence and statistics, pp. 562-570. PMLR, 2015.
Kibok Lee, Yian Zhu, Kihyuk Sohn, Chun-Liang Li, Jinwoo Shin, and Honglak Lee. I-mix: A domain-agnostic strategy for contrastive representation learning. arXiv preprint arXiv:2010.08887, 2020.
Ralph Linsker. An application of the principle of maximum information preservation linear systems. In Advances in neural information processing systems, pp. 186-194, 1989.
llya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608. 03983, 2016.
Ping Luo, Xinjiang Wang, Wenqi Shao, and Zhanglin Peng. Towards understanding regularization in batch normalization. arXiv preprint arXiv: 00846, 2018.
XuanLong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(11))5847-5888 5861, 2010.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1xXiv:1807.033 2018.
I Liam Paninski. Estimation of entropy and mutual information. Neural computation, 15(6):1191-12) 2003.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pp. 5171-5180. PMLR, 2019.
Andrew M Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D Tracey, and David D Cox. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2019(12): 124020,, 2019a.

---

Under review as a conference paper at ICLR 2022
Andrew M Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D Tracey, and David D Cox. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2019(12): 2019b.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-62 626, 2017.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1103: 703000810, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1555 2014.
Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators. arXiv preprint 190.0229/, 2019.
Jiaming Song and Stefano Ermon. Multi-label contrastive predictive coding. arXiv preprint arXiv:2007.09885. 2020.
Alessandro Sordoni, Nouha Dziri, Hannes Schulz, Geoff Gordon, Philip Bachman, and Remi Tachet Des Combes. Decomposed mutual information estimation for contrastive representation learning. In International Conference on Machine Learning, pp. 9859-9869, PMLR, 2021.
Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsung Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd International Conference on Puttern Recognition (ICPR), pp. 1464-2469. IEEE, 2016.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint ArXiv:190..05849, 2019a.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint arXiv:1910. 10699, 2019b.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. arXiv preprint htl1:20055/044.. 2020.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pp. 1-5. IEEE, 2015.
Vikas Verma, Thang Luong, Kenji Kawaguchi, Hieu Pham, and Quoc Le. Towards domain-agnostic contrastive learning. In International Conference on Machine Learning, pp. 10530-10541, PMIR, 2021.
Wenguan Wang. Tianfei Zhou, Fisher Yu, Jifeng Dai, Ender Konukoglu, and Lue Van Gool. Exploring cross-image pixel contrast for semantic segmentation. arXiv preprint arXiv:2101.1 1939, 2021.
Jingfeng Wu, Wenqing Hu, Haoyi Xiong, Jun Huan, Vladimir Braverman, and Zhanxing Zhu. On the noisy gradient descent that generalizes as sgd. In International Conference on Machine Learning. pp. 10367-10376. PMLR, 2020a.
Mike Wu, Chengxu Zhuang, Milan Mosse, Daniel Yamins, and Noah Goodman. On mutual in- formation in contrastive learning for visual representations. arXiv preprint 10X1::2005.13199 2020b.
Raymond W Yeung. new outlook on shannon's information measures. IEEE transactions on information theory, 3763):666447, 1991.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv:1708 03888, 2017.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07144 2016.
12

---

Under review as a conference paper at ICLR 2022
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3713-3722. 2019a.
Linfeng Zhang, Zhanhong Tan, Jiebo Song, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Scan: scalable neural networks framework towards compact and efficient models. arXiv preprint arXiv:1906.33951, 2019b.
13

---

Under review as a conference paper at ICLR 2022
Appendix
A PROOFS
A. PROOF OF PROPOSITION 4.1
We simply extend the exact bound InfoNCE (Poole et al., 2019; Sordoni et al., 2021). Here, consider the supervised setting where there are C' training classes. Without loss of generality, choose a class c out of C classes, and let æ and z be different samples that share the same class label c. The derivation for the multi-view (z being augmented sample of x)) is similar. For conciseness of the proof, we consider that no other image in a batch shares the same class. We prove that minimizing the SupCon loss (Khosla et al., 2020) maximizes the lower bound of conditional MI between two samples x and z given the label c
Proof.
I(x; 2/0) 2 log(2K 1) (xup(x, 2; F.
(6)
for some function and hyperparameter K.
We start from Barber and Agakov's variational lower bound on MI (Barber & Agakov, 2004).
p(z (x/ I(x; (00;((c) = Bp(a,= log p(z|c)
((1(2,))
2 Ep(m,z(c) log p(z|c)
where 9 variational distribution. Since g is arbitrary, we can the sampling strategy as follows. First, sample 21 from the proposal distribution т c) where e is a class label of æ. Then, sample (K - 1) negative samples (=2, ..., _x)
from the distribution En'ee क (z, d'), so that these negative samples do not share the class label with æ. We augment each negative sample by random augmentation and concatenate with the original samples, i.e., {22.... zx 2k+1, Z2K-1 }. where = +i-1 is the augmented sample from z; for 2 S S K. We define the unnormalized density of given specific set {z2,... - and of label cs
(2K 1) e0(a 2k -1 (tF(22))) 2k=
q x 22:(2K (21/c)- (2)
where va is often called a discriminator function (Hjelm et al., 2018), defined as v/ v v) = F(u) F(v) for some vectors u, By setting the proposal distribution (22cc) = p(z|c), obtain the MI bound:
(z)
x|;) 2 Eprars e) log
(9)
p(zl(c)
E 100 (2)(2K))).
Ep () log
(10)
p(z1 (c)
(2K - (p((a
p(z1/e)
e ( Ex=z (
2 Epla. o) Rp(essae--pll)) log
(11)
p(21(c)
Ep(a, c)p(=z log
(12)
2 Ex=1 (tm((m,))
= log(2K - Loup F. K).
(13)
where the second inequality is derived from Jensen's inequality. Because Eq. 12 is an expectation with respect to the sampled x and 21, the case where the anchor is swapped to z is also being considered.
14

---

Under review as a conference paper at ICLR 2022
A neural network F (backbone in our framework) with L layers are formulated = fL-1 .... o f1. Then, ()((, = =(u) F(v) = fu:L())- fu.(()). We define another discriminator function as (u, = f(e+1)=(u) (v)))) Obviously, the following equivalence holds:
(f = (
(14)
Note that f1:e(u) the f-th intermediate feature of input u. Following the same procedure as Eq. 9-13,
fiu()))) 2 (0 1y log
(15)
Ex 2K
pa (e log
(16)
22--1
2K Dx=1
= log(2K - -1) Laup 2; (P
(17)
From above, as the intermediate feature is arbitrary to the position, we can obtain similar inequality:
Stanrpashualcc)...oopppooccccCCe = I[F(());
(18)
(z) log(2K 1) Csup (2, FF
(19)
(20)
A.2 PROOF OF PROPOSITION 4.2
Proof. In Section 4.1, we proved that SupCon loss maximizes the lower bound of conditional MI between the output features of a positive pair. We can think of another scenario where the network F now has a sub-network G. Assume that the sub-network has M > l layers: G gM ogM- oo...og As we discussed in the paper, the exit path is placed after the l-th layer, so regarding our definition of the sub-network, shares the same parameters with F up 8-th layer, i.e., 91 = fi, 92 f2. ......
Jr. fr. Define (c v) = G(u)- G(v)
We introduce discriminator function that measures the similarity between the outputs from the backbone and the sub-network, thr uu, v) = F F(u) G(v). Similarly, ver(u, v) = G(u) FD)) Considering that the SelfCon-S loss has the anchored function of F and G. we obtain an upper bound of two symmetric mutual information. Here, single-viewed batch and we assume that other images a batch (i.e., 22, .. =x) are sampled from the different class label with æ.
æ because SelfCon-S loss is defined on the
I(F(x); G(z)lc) + F(G()):F(x))))
(21)
E log
2 (euro(a,,)) + evro(a, + 2k=2 Σk=z ((
e (cF(m
+ E log
(22)
2K-1 (cNor(a. +
E cvo((a.))) + e coo(m.z))
2 logg2K< - 1) - 2L (2; (F,
(23)
Due to the symmetry of mutual information.
(2);; G(x)lc) 2 log(2K - 1) (xelf-s(z; (F
(24)
15

---

Under review as a conference paper at ICLR 2022
In addition, similarly bound the SelfCon-M loss. As the derivation of SupCon loss bound, only consider the anchor æ and its positive pair 21. When the anchored feature is F(x), the contrastive features are: G(z), (z)), and F(z). By symmetry, when the anchored feature is (( the contrastive features are: F(x), F(z), and G(z). As the derivation of the SupCon loss bound, we assume the augmented negative samples, i.e., (22, 1* 2K-1;" -2K-
+I(G(=)) F
F(z))) + I(G G(z)lc)
(25)
(mo((a.) (Fo( g0((2)))
E log
4K (eUra(e.)) Li=1 emr((==)) -k=1 (vr(azz))
1
@ ( eVa((12))
E log
(26)
3
4K (eWor(x + +I 2K-1 @VGF(E,E)) + 22 e*G(2,201)
k=l
2
log(4K - 2L (x, 2; (F,
(27)
3
A.3 PROOF OF PROPOSITION 4.3
F(x) and G(z) are the output features from the backbone network and the sub-network, respectively, Recall that T denotes the sharing layers between F and G. T(x) is the intermediate feature of the backbone which is also an input to the auxiliary network path.
Before proving the following Lemma, we would like to note that the usefulness of mutual infor- mation should be carefully discussed on the stochastic mapping of neural network. If mapping T(x) H F(z) is deterministic mapping. then the MI between T(x) and F(x) is degnerate because (a)); F(x)) is either infinite for continuous T(x) (conditional differential entropy is -0o) constant for discrete T(x) which is independent
the network's 's parameters (equal to H(T()))). However, for studying the usefulness of mutual information in deep neural network, the map T(x) + F()) is considered as a stochastic parameterized channel. In many recent works about information theory with DNN, they view the training via SGD is a stochastic process, and the stochasticity in the training procedure lets define the MI with stochastically trained representations (Shwartz-Zir -Ziv & Tishby, 2017: Goldfeld et al., 2019; Saxe et al., 2019b; Goldfeld et al., 2019). Our theoretical claim focuses on the SelfCon loss as a training loss optimized by SGD algorithm. There- fore, analyzing the MI between the hidden representations while training with the SelfCon loss is based on the information theory to understand DNN (Tishby & Zaslavsky, 2015).
Also, information theory in deep learning, especially in contrastive learning, is based on the InfoMax principle (Linsker, 1989) which about learning a neural network that maps a set of input to a set of output maximize the average mutual information between the input and output of neural network, subject to stochastic processes. This InfoMax principle is nowadays widely used for analyzing and optimizing DNNs. Most works for contrastive learning based on maximizing mutual information grounds on the InfoMax principle, and they are grounding on the stochastic mapping of an encoder. Moreover, Poole al. (2019) rigorously discussed the mutual information with respect to stochastic encoder. This is common practice in representation learning context where æ is data and z is a learned stochastic representation.
Lemma A.
I(F(z): G( < [(F()):
(28)
Proof. As Fx)) and G(z) are conditionally independent given the intermediate representation T(x). they formulate Markov chain as follows: G T F. Under this relation, the following is
16

---

Under review as a conference paper at ICLR 2022
satisfied:
[(F();; G( = H(F()) ((F())
(29)
S {(F(x))cc (() G(z),c)
(30)
H(F()))c) - / p(t, f. g logp(flt, c)dtdfdg
(31)
t.f.g
H(F(())c) - / p(t, flc) c
(32)
J.
-H(F(m))c) - H(((())TT()..)) [[=)(
(33)
(34)
Eq. 30 is from the property of conditional entropy, and Eq. 32 is due to the conditional independence and marginalization of g.
From Lemma A.1 and Eq. 24, we can prove the Proposition 4.3.
Proof.
log(2K 1) Ladf ()
(35)
([F(()))(())
(36)
<I(F()):T()))))
(37)
- I(F(()))T(z)) - -[F())))))((((())))+)+)))))) +I(()))))+(((()))))))))
(38)
(39)
(LI
Strictly speaking, SelfCon-S loss does not guarantee the lower bound of either or (O) in Eq. 39. However, SelfCon-S loss guarantees the label-conditional MI between the intermediate and the last feature, which is
+ +D).
A.4 CONNECTION BETWEEN SELFCON LOSS AND CLASSIFICATION PERFORMANCE
We used the variational inference to prove that SelfCon loss the lower bound cabel-conditional between the intermediate and the last features; thus, assumed probabilistic model as Eq. 8. When the anchor feature is similar to the negative pairs (i.c., different class representations. (2:(2K( 1)), this model becomes distribution with random mapping, and SelfCon loss cannot be optimized. Therefore, representations other classes should be farther to decrease the gap between SclfCon loss and MI. After all, SelfCon loss has improved performance because it aims to increase the lower bound the label-conditional MI between and G while increasing the distinction between different class representations.
B IMPLEMENTATION DETAILS
Network architectures We modified the architecture of networks according to the benchmarks. For smaller scale of benchmarks (e.g., CIFAR-10, CIFAR-100, and ImageNn) and the residual networks (e.g., ResNet-18, ResNet-50, and WRN-16-8), we changed the kernel size and stride of convolution head to and 1, respectively. We also excluded Max-Pooling on the top of the ResNet architecture for the CIFAR datasets. Moreover, for VGG- 16 with BN, the dimension of the fully-connected layer changed from 4096 to 512 for CIFAR and head for contrastive learning consisted of two convolution layers with 128 dimensions and one ReLU activation. For the architectures of sub-networks, refer Appendix C.
MLP projection
17

---

Under review as a conference paper at ICLR 2022
Representation learning We refer to the technical improvements used in SupCon, ie.., cosine learning rate scheduler (Loshchilov & Hutter, 2016). an MLP projection head (Chen et al., 2020), and the augmentation strategies (Cubuk et al., 2019): [ResizedCrop, HorizontalFlip. ColorJitter, GrayScale). ColorJitter and GrayScale are only used in the pretraining stage. We used 8 RTX 2080 Ti GPUs and set the batch size 1024 for the pretraining and 512 for the linear evaluation. We used the batch size of 512 for ResNet-18 18 and 256 for ResNet-50 when pretraining on the ImageNet-100 dataset. We trained the encoder and the linear classifier for 400 epochs and 40 epochs, respectively, in ImageNet- 100 benchmark. Meanwhile, we trained 1000 epochs and 100 epochs, respectively, in all other benchmarks.
Every experiment used SGD with 0.9 momentum and weight decay of le-4 without Nesterov momentum. All contrastive loss functions used temperature T of 0. For fair comparison (Khosla et al., 2020), we set the same learning rate of the encoder network as 0.5. Then, we linearly scaled the learning rate according to the batch size (Goyal et al., 2017). For ImageNet- 100 dataset, we used the small learning rate of 0.0625. We used 5.0 as a learning rate of the linear classifier for the residual architecture, but it was robust to any value and converged in nearly 20 epochs. Meanwhile, for VGG architecture, only small learning rate of 0.1 converged.
1-stage training In the I-stage training protocol, we trained the encoder network jointly with a linear classifier on the single-viewed batch. Most of the experimental settings were same as those of representation learning, but we trained the encoder for 500 epochs on CIFAR and Tiny-ImageNer dataset. We used the batch size of 512 and the learning rate of 0.8. For the cross-entropy result of ImageNet-100 dataset, we trained for 90 epochs with learning rate decay of 0.1 after 30 and 60 epochs. Here, we used the batch size 512 and the learning rate 0.2.
In the multi-exit framework, we used linear combination of loss functions for the backbone and sub-network. We used only cross-entropy loss for the backbone network, and weighted linear combinations of various loss functions (e.g., KL divergence and SelfCon-S) for the sub-network. Self-Distillation (Zhang et al., 2019a) used the interpolation coefficient a of 0.5. For the 1-stage version of SelfCon loss, we follow the coefficient form in (Tian et al., 2019b): L = LCE + fLoett We set the coefficient 3 = 1.0 for ResNet- 18 and B 0.8 for ResNet-50. Note that we used the outputs from the projection head instead of the logits. We used temperature 3.0 for SD and T 0.1 for SelfCon loss.
C ABLATION STUDY ON SUB-NETWORK
The structure, position, and the number of sub-networks are important to the performance of SelfCon learning. First, in order to find a suitable structure of the sub-network, the following three structures were attached after the 2nd block of an encoder: (1) simple fc, fully-connected, layer, (2) small structure which reduced by half the number of layers in the non-sharing blocks, (3) same structure which is same as the backbone's non-sharing block structure. After we found the optimal structure, we fixed the structure of sub-network and found which position is the best. For ResNet architectures, there are three positions to attach; after the 1st 2nd and 3rd block. For VGG-16 with BN, there are four positions and for WRN-16-8 there are two positions possible. Note that blocks are divided based on the Max- Max-Pooling layer in VGG-16 16 with BN.
Table 8 presents the ablation study results for ResNet-18 18 and ResNet-50, and Table 9 presents the results for WRN-16-8 and VGG-16 with BN. We highlighted the selected structure and position in Table 8 and Table 9, For the ImageNet-100 100 dataset, we used same structure positioned after the 2nd block of 18.
Obviously, there are many combinations of placing sub-networks, and Table 8 and Table 9 presented an interesting result that some performance was the best when sub-networks are attached to all blocks. It seems that increasing the number of positive and negative pairs by the various views improves the performance. It is consistent with the argument of CMC (Tian et al., 2019a) that the more views, the better the representation, but our SelfCon learning is much more efficient in terms of the computational cost and GPU memory usage. However, for the efficiency of the experiments and better understanding of the framework, we stuck to single sub-network in all experimental settings.
18

---

Under review as a conference paper at ICLR 2022
Table 8: The results of SelfCon-S loss according the structure and position of sub-network. The classification accuracy is for ResNet-18 (Left) and ResNet-50 (Right) on the CIFAR-100 100 benchmark.
Position
Position
Structure
Accuracy
Accuracy
Black
2dd
3dd
ndd
3 Block
PC
75.3+
PC
78 78.5=0.3
74.7.0.2
Small
78.1
Same
74.5
Same
77.4+0.2
PC
73.22.0.2
77.0
FC
75.440..
FC
78.5=0.3
FC
75.5+0.1
FC
77.4+0.1
24.
78.7
Table 9: The results of SelfCon-S loss according the structure and position of sub-network. The classification accuracy is for WRN-16-8 (Left) and VGG-16 with BN (Right) on the CIFAR-100 benchmark.
Position
Position
Structure
1 Block
2dd Block
Accuracy
Streture
Block
Block
3dd Block
4t Block
Accuracy
FC
71.420.0
FC
74.4
Small
71.520.4
Small
76. 76.2+0..0
Same
71.5+0.3
Same
76. 0.1
70.9
Same
76. 0.1
FC
71.410.0
FC
72.0+0.0
0.2
71.5+0.1
Same
70.5+0.0
FC
72.520-1
D EXTENSIONS OF SELFCON LEARNING
D.1 SELFCON IN UNSUPERVISED LEARNING
Although we have experimented only in the supervision, our motivation of contrastive learning with multi-exit framework can also be extended to unsupervised learning. We propose SelfCon loss function for the unsupervised scenario and present the linear evaluation performance ResNet- 18 architecture on CIFAR-10W 100 dataset.
Loss function Under the unsupervised setting, Chen et al. (2020) proposed a simple framework for contrastive learning visual representations (SimCLR) with NT-Xent loss. SimCLR suggests contrastive task that contrasts the augmented pair among other images. Therefore, the objective of SimCLR exactly same as Eq. 1, while each sample has only one positive pair its own augmented image, i.e., P = {(i B) mod 2B). We denote this loss Lrim-
We fomulate SelfCon loss in unsupervised setting as SimCLR, using a positive set without label information. We formulate SelfCon loss with the multi-viewed and unlabeled batch (SelfCon-MU) as follows:
exp(w(x:) wi (1(p))/TT)
Lrelf-mu Σ Σ log
W.W, APE
Σ
Σ exp((w(x:) w2 w // + Σ exp(w((x,)-. . -o2((r
(40)
i (1, ...., 2B)
p. E.. (i, B) mod 2B}
E N = / Pr
As SelfCon loss in supervised setting, exclude anchor sample from Ptx and N, to avoid contrasting the same feature, i..., P.. +- P.. \ {i} when w = w. and N₂ + N; \ {t) when w = Wa. Here, we used - 0.5 for the unsupervised SelfCon loss and L
19

---

Under review as a conference paper at ICLR 2022
Table 10: The results under the unsupervised scenario. We compared our SelfCon-MU and SelfCon-SU loss with SimCLR in the unsupervised setting. For the comparison with supervised learning, we also added the classification accuracy of CE loss. We used ResNet- 18 encoder and CIFAR-100 dataset. Accuracy* denotes the accuracy of SelfCon learning with the anchors only from sub-network (see details Appendix D.2).
Method
CE
SimCLR
SelfCon-MU
SelfCon-SU
Multi-view
Accuracy Accuracy
72.910.1
63.3
5.0
6.4 6.4
64.6+0.1
12.8
We also formulate SelfCon loss with the single-viewed and unlabeled batch (SelfCon-SU) as follows:
exp(w(xx)) wi (2p:)/T)
Loels-on
Σ log
W,W! SP1
Σ
Σ
(appl((x))wg(xrpp + Σ exp((((x)) (2((xn/T))
(41)
i iEI= (1, f1..... B}
Pa E P..
{i}
n nE = I Pa2
Similarly, N; + N₂ [i} when w = w2 For the positive set, since this loss is based on the single-viewed batch, have empty positive when w..
Experimental results All implementation details for unsupervised representation learning are identical with those supervised representation learning in Appendix B, except for temperature r of 0.5 and linear evaluation learning rate of 1.0. We used small sub-network attached after the 2nd block. Table 10 shows the linear evaluation performance of unsupervised learning on ResNet- 18 in CIFAR-100 dataset. However, empirically found that the encoder failed to converge with SelfCon-MU and SelfCon-SU loss.
D.2 SELFCON WITH ANCHORS ONLY FROM THE SUB-NETWORK
We suspect that Eq. 40 and 41 allow the backbone network follow the sub-network, which makes the last feature learn more redundant information about the input variable, without any label information. Thus, unsupervised loss function under the SelfCon framework needs to modified.
When the anchor feature is from the backbone network, we remove the loss term which contrasts the features of sub-network. Strictly speaking, does not perfectly prevent the backbone from following the sub-network, since there is no gradient operation on the outputs of backbone network when the outputs of sub-network are the anchors. However, we hypothesize that helps prevent the encoder from collapsing to the trivial solution by the contradiction of IB principle. We confirmed the performance revised loss functions both unsupervised and supervised scenarios.
Loss function
(G(;)) w (Zpp )/T)
Loetffmau Lsim @ ΣΣ log Σ Σ exp( ((()) wa p/T + Σ eep(( w (x (wT)/TT))))))))))))
(42)
iEl= 2B}
p. E P.. = (i, (i B) mod 2B}
E N; = 1 Pi
We exclude anchor sample from P.. and N₁, and all notations are same as Eq. 40, except for the coefficient where used 1.0. For the supervised setting, simply change P.. to {p 12y = }} and Cxim to L Note that P., + P,, \ {i} when w, G and N; +- N₂ {d} when Wa G. We get rid of the situation that the anchor F(x:) contrasts the positive pair in sub-network G(r,,). Still,
20

---

Under review as a conference paper at ICLR 2022
Table 11: CIFAR-100 results with SelfCon extensions. Accuracy* denotes the accuracy of SelfCon learning with the anchors only from the sub-network.
Method
Architecture
Accuracy
Accuracy
SelfCon-M
24.9
74.9
ResNet-18
SelfCon-S
75.4=0.1
75 75.640.1
SclfCon-M
76.9
77.7
Re
SelfCon-S
g8.5
78.8
Table 12:
T(x). and F(x) respec- tively denotes the imput variable, label variable, intermediate feature, and the last feature. Recall that T(æ) is the intermediate feature of the backbone network, which is an input to the auxiliary network path. We summarized the average of estimated MI through multiple random seeds. We highlighted the MI between the intermediate and the last features, which is the main concern of the SelfCon loss. Bold type indicates the smallest values for I(æ; (x)) and the largest values for I (y; (=)) and I (x) T())), according to the IB principle.
The detailed results of mutual information estimation.
Estimator
MI
CE
SupCon
SelfCon-M
SelfCon-S
Z(:: T(w))
0.509
0.344
0.231
0.236
InfoNCE (Oord et al., 2018)
T(y: T(w))
0.214
0.235
0.469
0.479
([F(():T(())
0.288
0.303
0.530
0.553
2(æ; [[(]]
1.639
0.898
0.558
0.554
MINE (Belghazi 2018)
I(y: T(()))
[(F(m)):
0.734
0.762
1.726
2.123
T(ar; [(m))
1.485
0.930
0.601
0.589
NWJ (Nguyen 2010)
I(y: T(x))
0.390
0.546
1.450
1.389
[(F(():
0.655
0.799
1.820
1.873
the proposed loss function includes SimCLR loss (Csim) SupCon loss (Csup) in the unsupervised or supervised setting, respectively.
exp(G (ax) Fx 3/T)
f-ssffsuu
Σ log
(43)
Σ expppGr G(
F P(pp)//7 + Σ>> Σ exp G (z;)) w2 (TT
4P1
w2
i T (1, (1...., B}
Pe P.. (i)
E N,
= T P.2
For the supervised setting, we change the above equation as the equally linear combination of SupCon-S loss and Eq. with P.. = (p T|yp = =y Note that we also exclude contrasting the anchor itself in SupCon-S loss term.
Experimental results In Table 10, we also reported the accuracy of SelfCon-MU and SelfCon-SU loss according to Eq. 42 and 43. Surprisingly, in this case, SelfCon-MU outperformed SimCLR loss (Chen et al., 2020), improving 1.3%p. Unfortunately, SelfCon-SU had not converged again, although improved the result in a small amount compared to Eq. 41. While SelfCon-MU has SimCLR loss term which makes the backbone encoder still learn meaningful features, SelfCon-SU loss does not have the anchor features from the backbone, which makes the backbone hard to trained. Table 11 summarizes SclfCon-M and SelfCon-S loss, removing the anchors from the backbone in the supervised setting, i.c., supervised version of Eq. 42 and 43. As we expected, these variants of SelfCon-M and SelfCon-S further improved the classification performance.
DETAILS OF MI ESTIMATION
Table 12 summarizes the full details of mutual information estimation, measured on CIFAR-100 with the pretrained Re encoders. Note that considered the encoder output without the projection head. We used three types of estimators: InfoNCE (Oord et al., 2018), MINE (Belghazi al., 2018), and NWJ (Nguyen et al., 2010). As expected, SelfCon-M and SelfCon-S loss exhibited
21

---

Under review as a conference paper at ICLR 2022
1.0
0.8
0.6
0.4
02
0.0
Figure 7: Qualitative examples for mitigating vanishing gradient. Along with the original image, we visualized the gradient when training with SupCon (Left) and SelfCon-M loss (Right). Note that all the gradients are from the same model checkpoint of ResNet-18.
larger MI between the intermediate and the last feature of the backbone network than CE and SupCon loss.
F QUALITATIVE EXAMPLES FOR VANISHING GRADIENT
In Figure 5, we have already showed that the sub-network solves the vanishing gradient problem through the visualization for gradient norms of each layer. In Figure 7, we also visualized qualitative examples using Grad-CAM (Selvaraju al., 2017). We used the gradient measured on the last layer in the 2nd block when the sub-network is attached after the 2 block. In order to compare the absolute magnitude of the gradient, it is normalized by the maximum and minimum values of the two methods, SelfCon-M and SupCon. As in Figure 7, SelfCon learning led to a larger gradient via sub-networks, and more clearly Grad-CAM highlighted the pixels containing important information in the images.
G IMGENET RESULTS
We experimented with the full-scale ImageNet benchmark enhance the reliability of Self- Con learning on the large-scale dataset. Due to the limited computational resources, we used ResNet- 18 architecture for both the pretraining and linear evaluation. For the pretraining stage, we experimented with various batch sizes and epochs. Besides, we tuned the learning rate as 0.125 for 1024 batch size and 0.25 for 2048 batch size. In the linear evaluation stage, we trained the linear classifier with a batch size of 256, 40 epochs, and a learning rate of 6.0. Other hyperparameters
Table 13: The classification accuracy on ResNet- 18 for ImageNet.
Method
B
Epoch
Acc@) 1 Acc@5
CE
256
90
69.9 89.4
SupCon
1024
400
70.9 89.6
SupCon-S
1024
400
69.2 / 89.2
SelfCon-M
1024
400
71.2 / 90.1
SelfCon-S
1024
400
70.3 89.6
SupCon
1024
800
71.7 71.7/90.3 99.3
SupCon-S
1024
800
69.7 / 89.6
SelfCon-M
1024
800
71.9 / 90.4
the same Appendix B. For the cross-entropy result, we used the batch size of 256 and the learning rate of 0. The settings of epochs and the learning rate rule are the same as the ImageNet- 100 experiments.
SelsCon-S
1024
800
70.8 / 89.9
SupCon
2048
800
71.9 /1.9/90.3 990.
SelfCon-S
2048
800
72.0 90.4
We summarized the experimental results in Table 13. For the experiment with 1024 batch size and 400 epochs, it is still consistent that SelfCon-M showed better performance than SupCon, but the multi-viewed methods outperformed the single-viewed counterparts. We consider the possible reasons as follows.
(1) Under-fitting issue due the tremendous number samples and relatively small size of architecture. (2) Insufficient number of training epochs (also refer to Figure 4). (3)
Relatively small batch size with respect to the number of classes (1000) in ImageNet (also refer to Table 4). Therefore, we did two more ablation studies: (1) pretraining with 800 epochs and (2) pretraining with a larger batch size (B- = 2048).
In the experiments with 800 epochs, the trend that multi-viewed methods outperformed the single- viewed counterparts did not change. However, the classification accuracy of each algorithm signifi- cantly increased as the pretraining epochs became longer. On the other hand, for the larger batch
22

---

Under review as a conference paper at ICLR 2022
Table 14: Memory (GiB / GPU) and computation time (sec step) comparison, All numbers are measured with ResNet-50 training on FLOPS is for one sample. B stands for batch size.
8 RTX 2080 Ti GPUs and Intel 19-10940X CPU. Note that
Datasel (Image size)
Method
Params
FLOPS
Memory
Time
Menory
Time:
Memory
Time
CIFAR-IO1 (32x32)
SupCou
SelFCon
27.96
33.47 M
2.62
1.31 G
4,00
2.73
0.11
3,92
0.14
0.12
6.28
0.20
0.16
Thny ImageNett (611x64)
SupCon
S
27.96 M
33.47 M
G
G
4,41
2,98
0.21
0.21
4.21
6.71
0.26
0.23
11344
6.82
0.36
0.27
size (i.e., B 2048), observed that SelfCon-S is better than SupCon. The difference seems be marginal, but it noteworthy that the training time for SelfCon-S is more than twice shorter than SupCon, and memory consumption is much lower. an important result in terms of the recent research flow for the compression in contrastive learning (Koohpayegani et al., 2020; Fang et al., 2021). From the result of SelfCon-S in the 2048 batch size, are highly convinced that the trend in ImageNet is the same as the other benchmarks. The experiments for the larger architecture (c.g., ResNet-50, ResNet-101) the larger batch size (c.g., 4096, 6144) are left for the future work.
H MEMORY USAGE AND COMPUTATIONAL CoST
In Table 14, we reported the computational cost for pretraining with ResNet-50. ImageNet- 100 dataset result is not reported because ResNet-50 with batch size over 256 exceed the GPU limit. The overall trend similar that in Table
SENSITIVITY STUDY FOR LEARNING RATE
In Table 4, we experimented with the supervised con- trastive algorithms with various batch sizes and con- firmed that the classification accuracy decreases in the large batch size. We supposed that this trend is induced by the regularization effect from the batch size. How- ever, there could be concern for using sub-optimal learning rate on the large batch size. We further studied the sensitivity for the learning rate in batch size of 1024 and summarized the results in Table 15.
Table 15: CIFAR-100 results ResNet- 18 with various learning rates. Bold type for the best accuracy within each method.
Learning Rate
Method
0.1
0.5
1.0
1.5
SupCon
73.0
73.7
SupCon-S
72.1
73.9
74.6
75.0
SelfCon-M
72.9
74.9
75.4
75.8
SelfCon-S
73.7
75.4
75.7
76.2
We concluded that the performance comparison in Ta- ble 4 is consistent with hyperparameter tuning. The experimental results supported that larger learning rate than 0.5 may be a better choice but the trend between all methods maintained in parallel with the learning rate of 0.5. Therefore, we stick to the initial learning rate of 0.5 that Khosla et al. (2020) had used.
ABLATION STUDY FOR DIFFERENT AUGMENTATION POLICIES
We could think of the optimal scenario where we have prior domain knowledge about a good augmentation policy. Also, there could be the opposite scenario where we do not know optimal policy and just use the basic augmentations. Hence, we investigated three different augmentation policies as follows and compared the performances of SupCon and SelfCon-S under these scenarios.
Table 16: CIFAR-100 results ResNet- 18 with various augmentation policies.
Augmentation Policy
Method
Optimal
Standard
Simple
SupCon SeliCon-S
74.3
73.0.0.0
72.040.3
72.5_o.
75.4
74.220
Optimal We used RandAugment (Cubuk et al., 2020) for an optimal augmentation policy. RandAugment randomly samples N out 14 transformation choices (e.g., shear, translate, auto- Contrast, and posterize) with M magnitude parameter. We used the optimized value of N = 2 and M = 9 in (Cubuk et al., 2020). It is also known that SupCon performs better with Randenggent policy (Khosla et al., 2020).
23

---

Under review as a conference paper at ICLR 2022
MI Estimator
Test Acc.
1.8
76
-188
Mutual Information I (F()))
75
Test Accuracc (%)
1.2
74
1.0
86
73
72
-066
0.4
71
0
InfoNCE
MINE
NWJ
70
SupCon
SellCon-M 0.03
SellCon-MM MM 9.08
SallCon-M 0.0r
SellCon-M's a1
SellCon-M 0.18
SelfCon-M
Figure 8: Test accuracy and the estimated mutual information of different methods. SelfCon- M a denotes SelfCon-M* loss with hyperparameter a. We used ResNet-18 on CIFAR-100 dataset for the measurements.
Standard: For standard augmentation, we used (RandomResizedCrop. RandomHorizontalFFip RandomColorJitter, RandomGrayscale).. This is the same basic policy we used in the paper.
Simple: When we do not have domain knowledge, it might be difficult to choose the appropriate augmentation policies. We assumed a scenario where we might not know that the color would be important in this visual recognition task. Therefore, we removed the color-related augmentation policies from Standard policy, i.e., we only used (RandomResizedCrop. RandomHoorizontlFFiii for simple augmentation policy.
The results are presented in Table 16. Interestingly, when apply Optimal augmentations, SupCon outperformed SelfCon-S, but for Standard and Simple augmentations, SelfCon-S outperformed SupCon. SupCon with the multi-viewed batch can benefit more from the strong and optimized augmentation policy because training each sample twice more encouraged the memorization. Mean- while, SelfCon learning did not work well with RandAngment (Cubuk et al., 2020), as SupCon degraded with the Stacked RandAugment (Tian et al., 2020) in their experiments. There would be an optimal policy for SelfCon learning but, finding an optimal policy, such as with RandAugment or AutoAugment (Cubuk et al., 2019), is not trivial process and needs lot of computational cost. SelfCon learning can relieve this concern and is more robust to the simple augmentation (i.c., weak augmentation policy).
K ADDITIONAL EXPERIMENT FOR CORRELATION BETWEEN SELFCON Loss AND MUTUAL INFORMATION
To clearly show the correlation between the mutual information and test accuracy, we experimented with the interpolation between SupCon loss and SelfCon-M loss (SupCon loss is a special case of SelfCon-M loss). However, the current formulation of Eq. 1 and Eq. cannot make the exact interpolation between SupCon and SelfCon-M because the SelfCon-M M loss should have negative pairs from different levels of network (i.e., backbone and sub-network), but the SupCon loss cannot produce those. Therefore, we should modify the SelfCon-N loss to a supervised version of Eq. 42. In the supervised version of Eq. 42, break down the SupCon loss term and SelfCon-like loss
24

---

Under review as a conference paper at ICLR 2022
Table 17: The detailed results of mutual information estimation. Every notation is same as Table 12. We used ResNet-18 on CIFAR-100 dataset for the measurements.
SupCon
SelfCoo-M
SelfCon-M
SelfCon-M*
SelfCon-M*
SelfCon-M*
SelICon-M
Estimator
0.075
I(a: T(x))
0.285
0.296
0:277
0.299
0.293
0.232
0.203
InfoNCR
I(y: (()))
01.211
0.299
0.290
0.309
0.320
0.341
0.454
[()):: T()))
0.357
I(m: T(m))
0.758
0.843
0.719
0.744
0.065
0.697
0.508
MINE
I(y: [() T(m))
0.845
0.919
0.937
1.032
1.140
1,050
1.760
[mm)
NWJ
I(y: ((x)
0.467
0.594
0.641
0.508
0.799
0.843
1.287
I(F(())
term, with an interpolating parameter
exp ((x))) w (x 1/T)
Loelf-me
Lsup a ΣΣ log
{ exp((((x.)
w2 (Tp )/T) + Σ (xp(((()))-u((x))))
4P1
W2
P2
(44)
i
2B}
Pa E P.. = {p E TBU = vi}
MEN,=IV
=
where Pi. f- Pi. {i} when w. and N; +- N, \ {i} when W2 = G. Therefore, i Lontf-mm is equivalent to the SupCon loss and if 1, fm almost equivalent to SelfCon-M loss. Figure 8 describes the estimated mutual information and its relationship with classification performance via controlling the hyperparameter a, and Table 17 summarizes the detailed estimation values of the intermediate feature with respect to the input, label, and the last feature. We observed a clear increasing trend of both MI and test accuracy as the contribution of SelfCon gets larger. Also, the detailed MI estimation values in Table 17 imply the same interpretation as the IB principle and Appendix E.
25


Reviews:
[{"review_id": "wRxPrFj-FhF", "reviewer": "Reviewer_ZiPE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is mostly well-written and nicely structured. The motivation is clearly outlined, and the discussion of related work is extensive. I liked how the contributions are clearly linked to sections in §1, and the use of color to highlight important parts in the equations throughout the text. A major strength is the possibility of *supervised* contrastive learning without additional augmentations (and I think this could be highlighted more prominently as a strength).\n\nA major shortcoming of the paper is a lack of full ImageNet results, and the small batch sizes considered in the paper. I should note, while the numerous analyses added to the paper are nice, a few well-executed experiments including the “gold standard” models for self supervised learning would have made the paper much stronger (and a clear accept).\n\nI will discuss some additional weaknesses worth addressing / commenting on below.\n\n### Weaknesses\n\nI have the following concerns about the experiments:\n\n* The batch size in the supervised contrastive learning paper was 6144 --- how do you justify the must smaller considered batch size in the paper? How would you rule out that a larger batch size could diminish the gains from your method?\n* The supervised contrastive papers outlines a technique for training with batch size 256 (I think effectively 512 due to the augmentation) on a 8 GPU system 79.1% for full ImageNet. If compute was a limiting factor, could the authors comment why they didnt use this setup? There is also a range of methods (like MoCo and related methods) with momentum buffers or other methods that make it possible train CL models on ImageNet scale on 8 GPU setups.\n* Any reason why Table 3 does not include measurements for full-scale images, i.e., ImageNet-100? For practitioners, this would be by far the most relevant result. \n* Fair comparisons: Why does the result for SupervisedContrastive learning in Table 4 gets worse with a larger batch size? Might this be due to suboptimally tuned hyperparameters (especially the learning rate)?\n\nSome weaknesses about the interpretation of results:\n\n* The paper contains a lot of causal statements about a link between quality of MI estimation and the performance in downstream classification performance. Yet, the only evidence is Figure 2, which is a correlation analysis with a few point estimates. What common confounders could alternatively explain this result, and why did you think that there was no additional need to run more control experiment? What happens e.g. when different hyperparameter setups for Self/SupCon are compared; does the correlation between MI and Test Acc hold?\n* Table 2: Since this is a new dataset (as far as I am aware of), could you minimally provide mean and standard deviations across three seeds? I find it hard to judge the variance and what constitutes progress on this task without seeing CIs; also, were hyperparameters selected fairly between the baseline and the tested CL models? To make this result more stronger, what about evaluating the existing full imagenet trained checkpoints of SupCon and including this as a comparison?\n\nThe method section 3 needs work (but please correct me if I am misinterpreting anything here):\n\n* In Eq. 1, could you verify that $1[F(x_i) \\neq F(x_p)]$ is equal to 1 for all cases, i.e. you included this to make the equation consistent with Eq 2? If this is the case, what was the motivation? Why did you not just use $\\omega \\neq \\omega’$ in Eq 2? Could you please clarify if I am missing something? If I am correct that something is off, I would appreciate it if you could let me know what the revised equations would look like. The same concern applies to Eq. 2.\n* “We dropped the dividing term of sums for brevity” -> Could you clarify and re-write this sentence? I do not get the reference.\n* The notation is incomplete; it should be made clear that $P$ depends on $i$. Suggestions: Call it $P_i$ or $P(i)$. Also, $(i, p) \\in I \\times P$ does not work because of this, it should be $i \\in I, p \\in P_i$ (for example). The same applies to $J$, which contains all images except the anchor. Write this explicitly: $J \\setminus\n \\{i\\}.\n\nA few additional minor points, not sorted by priority:\n\n* Use of barplots: Figure 2 and Figure 3 show barplots with truncated y axis, which is a bad practise. Much better and also more informative is an interaction plot (i.e., just plot a dot, along with an error bar). This is a major concern, because strong results are concluded from these plots.\n* In Figure 2, the different points for NWJ, MINE and InfoNCE should not be connected --- the x axis is categorical, not continuous. Also, a much better of getting the point across would be a scatterplot between MI and Test Accuracy.\n* Figure 6: Do not use the jet colormap. It is widely recognized that it distorts the visual perception of results. Please use viridis (as in Figure 5), or a similar variant with less issues. Especially for the dog images, the results conveyed by the jet map are misleading.\n* Table 1, column “single view”: I would e.g. use checkmarks ✓ instead of the current O/X --- I would rather associate the “X” with “yes” then the “O”. So I suggest to disambiguate this.\n* In the text, you wrote $P = i + B \\mod 2B$ --- while this might be technically correct, I recommend to write $P = (i + B) \\mod 2B$, especially readers with a programming background might be confused on a quick pass through the paper since operator precedence in e.g. Python would interpret the original statement as  $P = i + (B \\mod 2B)$\n* I find the distinction between “single view” and “multi view” a bit misleading / counter-intuitive. Could you comment whether you feel this is a standard term used in popular related work?\n* It would be nice to actually include the image sizes in Table 3, for example after a linebreak in the first column (e.g. “CIFAR-100<linebreak>32x32$)\n* Table 3:\n   * It would be more interesting to show time / step, vs. time / epoch; otherwise you need to adapt for the fact that higher batch sizes mean “shorter epochs” due to less batches / epoch.\n   * Why does the cost per training step go up? Is there an issue with data-loading in the implementation? ", "summary_of_the_paper": "The authors propose an improved algorithm for supervised contrastive learning, which they term “self-contrastive learning”. Instead of only contrasting representations at one point in the network, self-contrasting uses readouts at multiple stages in the network (remapped into an appropriate feature space by an additional readout network). This makes it possible to propose two variants: A “multi-batch” variant that first augments the images, and then contrasts views between samples of the same class, both augmented and non-augmented, and a “single-batch” variant that only contrasts between samples of the same class. Improvements are shown for CIFAR-10,100, TinyImageNet, as well as a smaller version of ImageNet with 100 classes, referenced as ImageNet-100.", "summary_of_the_review": "The paper proposes an interesting addition to supervised contrastive learning, which removes the lack for data augmentations during training and yields improved results. The results look compelling to me, although I generally have the impression that they were executed under a lack of computational ressources (seen, e.g. by relatively small batch sizes although large batch sizes are known to improve performance, lack of multiple seeds for some experiments, etc.), and hence I am worried about the significance of them (how well baselines were tuned, etc.).\n\nThat being said, I am willing to improve the assigned score the paper. To consider an accept, the authors need to minimally show a proof of concept result on ImageNet (for example in a very controlled setup, reproducing the SubContrast result, adding their method and showing an improvement with no special additional tuning, to make the comparison as fair as possible.\n\nAs I see it, the method already works on full-scale images. What is the blocker for running a full scale experiment? Are you confident that the improvements hold on full scale ImageNet training? From the data in the paper right now, it is not clear that the improvement will hold in the regime simCLR-like loss functions are typically evaluated on (large batch sizes, ResNet50 models, full ImageNet).\n\nWithout full ImageNet results, the paper in my opinion does not meet the bar for ICLR, as the one crucial experiment that would contribute to adoption of the method is missing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "STrZCGlEoN", "reviewer": "Reviewer_qb4T", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: The paper propose an interesting idea to employ a multi-exit network for conducting contrastive representation learning without using augmentation techniques to create positive pairs. The method has the potential of addressing the various issues of the multi-viewed contrastive learning such as requiring domain knowledge for augmentation and increased memore/computation. Besides, I also appreciate various ablation studies in the experiments and appendix investigating aspects such as gradient norm, the structure/position/number of subnetworks, different encoder structures, etc.\n\nBelow are some of my concerns/questions:\n- There are two versions of the self-contrastive loss, the single-view and multi-view version. In the discussion below Eq (2), the paper defines SelfCon-M loss as exactly Eq (1)? I wonder if this is the best design of the loss since with the multi-exit network (different from previous method that only has one network output), when facing multi-view batch, there could be a similar multi-exit extension for the multi-batch setting and should not be just Eq (1)? Also I think even with multi-exit network, the multi-view setting does not just provide redundant information as inductive bias from data augmentation and inductive bias from encouraging similarity within multiple outputs should be different. Thus I am also confused why SelfCon-M, a \"generalization\" of SelfCon-S with additional information, is mostly worse than SelfCon-S. Is overfitting to each instance the only reason and is there any way to fix it for scenarios where we indeed have prior knowledge for doing augmentation?\n- What is the conclusion for using more than one subnetworks (i.e. multi-exit network instead of just 2-exit network)? It seems the current observation is changing the structure will increase the computation a lot but does not really help?\n- The theoretical analysis is mainly that the proposed loss provides a lower bound for the MI between intermediate features and last features. I think intuitively this is straightforward since if we maximize MI between the two outputs of the network, and the second output totally depends on the intermediate representation, then the intermediate representation has to be closely correlated with the last output such that the second head also contains similar information as the first head. That being said, I think it is still not clear that why such a theoretical result can explain an improved classification performance or better representation learning. A degrade case is, if we put an identity mapping between the intermediate layer and the last output of the backbone, the MI is also very large but it will not help for the downstream task. \n- There are many recent advances in representation learning. For example, BYOL has received a lot of attention for a SOTA performance without using negative pairs. Any discussion and comparison to these methods? Also in section 2.1, BYOL is introduced following \"with negative pairs\", which is not accurate?\n- At the beginning of sec 2.2, \"Mutual information is a measure to quantify the amount of information held in a random variable\" - I think what you are describing is entropy not MI.  MI is a measure of the mutual dependence between two random variables.\n- Overall, I think the method derivation/description (section 3) is hard to follow and not clear enough. An algorithm box may also be helpful. I think the presentation can be significantly improved for better readability. For example, there is no appropriate explanation for the key equations like eq 2 and a formal definition of SelfCon-M. It takes me some time to parse it.\n\n---------------After rebuttal-----------------------  Thanks for the detailed response to my feedback. I appreciate the efforts on updating the paper and I think some of my concerns have been addressed. I thus raised my score to weak accept.", "summary_of_the_paper": "The paper proposed a contrastive learning framework to remove the requirement for additional data augmentation techniques for creating positive pairs, by leveraging the idea of multi-exit network and conduct \"self-contrasts\" among multiple outputs of the model. Theoretical analysis on the designed loss shows that the proposed SelfCon loss is a lower bound of MI between the intermediate and last feature. Extensive experiments and ablation studies demonstrate that the proposed method has better performance than than the supervised contrastive baseline.", "summary_of_the_review": "In summary, I think this paper propose an interesting idea with some potential of improving contrastive learning by removing the requirement of data augmentation. Nevertheless, after reading the paper, I also have a number of concerns/questions on some statements, method derivation, theoretical analysis and empirical evaluation. Overall, I think the presentation can be improved to make it easier to follow.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RIV0LOeOMaP", "reviewer": "Reviewer_MBzi", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The idea of not using multi-views in contrastive learning is important. Good quality of data augmentation is an issue in contrastive learning.\n2. The authors did a lot of analysis and qualitative experiments to explain their method and motivations.\n\nWeakness:\n1. The main theoretical contribution which is self-con can maximize the mutual information between intermediate and last feature is incorrect. To prove this claim (prop 4.3), the authors use the inequality that $I(F(x), G(x)) \\leq I(F(x), T(x))$. Then the authors claim that the negative self-con loss is a lower bound of $I(F(x), T(x))$. Therefore, minimizing the self-con loss can increase the mutual information between $F(x)$ and $T(x)$. However, the equity may not hold. Especially, in the authors' prop, maximize negative self-con loss can only increase $I(F(x), G(x))$ but not $I(F(x), T(x))$. There is a gap between $I(F(x), G(x))$ and $I(F(x), T(x))$.  Maximizing $I(F(x), G(x))$ does not imply maximizing $I(F(x), T(x))$. \n2. On the other hand, since $F(x)$ is a function of $T(x)$, the mutual information of $I(F(x), T(x))$ is either zero or infinity. (https://stats.stackexchange.com/questions/465056/mutual-information-between-x-and-fx) I am not sure why the authors want to maximize it.\nThe authors somehow shows that the mutual information between the intermediate layer and the last layer is finite in figure. I think this is the problem of the estimation.\n3. In Table 2, why does the authors only report one seed? Multiple seeds are needed. Confidence interval needs to be reported.\n4. In Table 4, why does single view benefits from larger batch size? What about batch size 512?\n5. It seems that you use labels when building self-contrastive learning. If you already use the labels, why do not you compare with fully-supervised approaches? What's the motivation of not using fully-supervised approaches?\n\nMinor:\n1. The usage of \"Markov chain\" in the proof of prop 4.3 may be inaccurate. The arrows should be $F \\leftarrow T \\rightarrow G$.\n\n-----------------------\nUpdates:\nI thank the authors for detailed rebuttal and answer further questions. I increased my overall score from 3 to 5 and empirical novelty from 2 to 3. The authors add results for more batch sizes and explain that they post a new empirical method and not a theoretical understanding. However, in the original paper, the authors explain their method from the prop 3. Prop 3 does not tell the readers more than minimizing the mutual information between $G(x)$ and $F(x)$. I am against motivating the method from an invalid theoretical statement. So I am still against accepting the paper. The followup motivations like distilling good information from last layer to the intermediate layer also sounds vague. The authors may rewrite the paper in a fully empirical way without any vague theoretical motivation.\n", "summary_of_the_paper": "This paper provides a new way of supervised contrastive learning which does not require multi-views. The main idea is to use different functions for contrastive pairs which allows the usage of single views in contrastive learning.", "summary_of_the_review": "The authors propose a new method which does not require multiple views in contrastive learning. It has the potential to motivate follow-up works. However, the theoretical motivation seems to be incorrect. The author does not consider the gap in the lower bound. So I vote for rejection. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3_B_N_9jw5V", "reviewer": "Reviewer_hbNU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "-\tTo the best of my knowledge, this is the first work to propose self-contrastive learning via sub-networks. The paper holds significant methodological contributions and provides both theoretical and empirical grounds to support the effectiveness of the proposed model. While I believe the paper is clearly above the acceptance threshold, I have a few comments to clarify or supplement some points.\n-\tWhile the authors state that SelfCon has also potentials for an unsupervised setting (Appendix D.2.), the current version is still mostly focused on the supervised setting. I think it will be better if it is more clearly stated both in the abstract and introduction.\n-\tIn the 3rd paragraph of the introduction, the authors stated that “the multi-exit framework already generates positive pairs from a single image, making the data augmentation redundant.” However, I think this is not entirely correct considering that SupCon-S outperforms SupCon even without the multi-exit framework (Table 1). Section 5.3 seems to provide more logical explanations that multi-view might cause the overfitting to each instance. \n-\tCan you provide additional explanations for why increasing the lower bound of label-conditional MI between the intermediate and the last features is desirable and improved the classification performance of the encoder network? It might be unclear considering that “good representations should get rid of redundant input information.” (Section 4). In addition, can you also explain why increasing the “influence of T(x)” on the amount of shared information between the last feature and the label? (cf. increasing the MI btw the last feature and the label).\n-\tCan you clarify the meaning of the indicator functions I(F(x_i) != F(x_p)) from the equations? I could not quite understand (1) why the indictor functions contain F or G rather than the labels and (2) why those in the numerators have inequalities rather than equalities. To define the positive pairs as samples with the same ground-truth labels, shouldn’t the indicator functions in the numerators be something like I(y_i == y_p)?\n", "summary_of_the_paper": "Recent contrastive learning-based algorithms proposed various ways to make representations from semantically similar (positive) samples closer and those from semantically different (negative) samples farther. While they have shown remarkable performance improvement, they usually require a multi-viewed batch for defining the positive samples. In this paper, the authors proposed Self-Contrastive (SelfCon) Learning which self-contrasts within multiple outputs from the different levels of a multi-exit model. The authors theoretically demonstrated that SelfCon loss guarantees the lower bound of label-conditional mutual information (MI) between the intermediate and the last features. Then, they empirically showed the proposed method significantly outperforms the cross-entropy and Supervised Contrastive Learning (SupCon) baselines.", "summary_of_the_review": "While I believe the paper is clearly above the acceptance threshold, I have a few comments to clarify or supplement some points in terms of the supervised settings, the rationale for increasing the label-conditional MI, and the proposed loss equations. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper modifies the loss of supervised contrastive (SupCon) learning by adding a self-contrastive loss. Utilizing a multi-exit network and contrasting the multiple outputs of this network, the proposed self-contrastive (SelfCon) learning removes the requirement of additional data augmentation samples for creating positive pairs. The proposed SelfCon loss is theoretically connected to the lower bound of a label conditional mutual information between the intermediate and last feature. The paper focuses its study on SupCon & SelfCon-M, which are multi-batch variates that first augment the images, and then contrast the views between both augmented and non-augmented samples of the same class, and on SupCon-S and SelfCon-S, which are single-batch variates that only contrast between the samples of the same class and do not require additional data augmentations. A wide variety of experiments have been done on CIFAR-10, CIFAR-100, TinyImageNet, ImageNet-100, and ImageNet, but mostly with relatively small networks. 

The ratings for the paper were mixed [3,5,5,8 before rebuttal; 5,5,6,8 after rebuttal]. All four reviewers had provided detailed initial reviews, pointing out a long list of issues. The authors had incorporated these reviews to make a large number of improvements to their initial submission. After the author rebuttal period, while one reviewer raised the score from 5 to 6, two reviewers maintained their negative positions: Reviewer ZiPE is clearly concerned about the risk of accepting a method that may break as soon as a slightly larger model (ResNet50 instead 18) is used, the model is trained a bit longer, or the baselines are tuned, while Reviewer MBzi is unsatisfied with how the paper motivates its empirical construction from the perspective of mutual information maximization. 

Given the disagreements between the reviewers, the AC has carefully read the paper to provide an additional review. Some concerning observations of the AC are summarized as follows:

1. Echoing the concern of Reviewer ZiPE, the performance gain of SelfCon-S over SupCon diminishes in ImageNet with ResNet-18, as shown in Table 13, making it become even more important for the authors to conduct experiments following more standard settings (e.g., ResNet-50 on ImageNet).

2. The main paper seems to suggest SelfCon-S outperforms SelfCon-M and SupCon outperforms SupCon-S, while Table 13 in the Appendix suggests the opposite. 

3. Table 3 that compares SelfCon-S with SupCon appears very misleading, as SupCon consumes more memory and computation than SelfCon-S simply because it has used data augmentations. If SupCon-S is used, it would take less memory and computation than SelfCon-S.

4. SelfCon-S adds a subnetwork to the backbone to boost its performance, so technically, it has more parameters than the backbone. Comparing it with a baseline that only uses a backbone model does not seem to be that fair. This point has not been discussed in the paper. 

5. Last but not least, echoing the concerns of Reviewer ZiPE and MBzi, the paper seems to try to validate the motivating of the added loss with mutual information maximization.  However, establishing the causal relationship between maximizing the mutual information of the intermediate and last layers and the classification performance needs much more than the correlation analysis provided in the paper. 

Given the above-mentioned concerns, the AC does not consider the paper to be ready for publication at its current stage.
Decision: Reject