Hierarchical Cross Contrastive Learning of Visual Representations
Authors: Hesen Chen
Ming Lin
Xiuyu Sun
Rong Jin
Venue: ICLR 2022 Submitted (2022)
Keywords: Self-supervised Learning
Unsupervised Learning
Computer Vision

Abstract: The rapid progress of self-supervised learning (SSL) has greatly reduced the labeling cost in computer vision. The key idea of SSL is to learn invariant visual representations by maximizing the similarity between different views of the same input image. In most SSL methods, the representation invariant is measured by a contrastive loss which compares one of the network outputs after the projection head to its augmented version. Albeit being effective, this approach overlooks the information containing in the hidden layer of the projection head therefore could be sub-optimal. In this work,  we propose a novel approach termed Hierarchical Cross Contrastive Learning(HCCL) to further distill the information mismatched by the conventional contrastive loss. The HCCL uses a hierarchical projection head to project the raw representations of the backbone into multiple latent spaces and then compares latent features across different levels and different views. By cross-level contrastive learning, HCCL not only regulates invariant on multiple hidden levels but also crosses different levels, improving the generalization ability of the learned visual representations. As a simple and generic method, HCCL can be applied to different SSL frameworks. We validate the efficacy of HCCL under classification, detection, segmentation, and few-shot learning tasks. Extensive experimental results show that HCCL outperforms most previous methods in various benchmark datasets.

Under review as a conference paper at ICLR 2022
HIERARCHICAL CROSS CONTRASTIVE LEARNING OF VISUAL REPRESENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
The rapid progress of self-supervised learning (SSL) has gready reduced the la- beling cost in computer vision. The key idea of SSL is to learn invariant visual representations by maximizing the similarity between different views of the same input image. In most SSL methods, the representation invariant is measured by a contrastive loss which compares one of the network outputs after the projection head to its augmented version. Albeit being effective, this approach overlooks the information containing in the hidden layer of the projection head therefore could be sub-optimal. In this work, propose novel approach termed Hierarchical Cross Contrastive Learning(HCCL, to further distill the information mismatched by the conventional contrastive loss. The HCCL uses hierarchical projection head to project the raw representations of the backbone into multiple latent spaces and then compares latent features across different levels and different views. By cross-level contrastive learning, HCCL not only regulates invariant on multiple hidden levels but also crosses different levels, improving the generalization ability of the learned visual representations. As simple and generic method, HCCL can be applied different SSL. frameworks. We validate the efficacy of HCCL under classification, detection, segmentation, and few-shot learning tasks. Extensive ex- perimental results show that HCCL outperforms most previous methods various benchmark datasets.
INTRODUCTION
Self-supervised learning (SSL) of visual representation aims to learn representations from image pixels without semantic labels. Recently, contrastive representation learning has gained growing attention by demonstrating state-of-the-aa performance in SSL for large-scale image recognition, closing the gap to supervised baselines (Chen et al.) 2020 Grill et al.) [2200] CCen & He [02T)) Contrastive representation learning is essentially the distance of the representation generated by the same image, as an instance class, while pushing away the representation generated by other images to learn invariant visual representations. In most contrastive learning methods, the representation invariant is measured by a contrastive loss, which compares of the network outputs after the projection head to its augmented version. Some works attempt to explore advanced training methods to keep the contrastive loss stable in SSL, by cluster- ing, memory bank, momentum update or projector head/Caron al., 2020 He et al., [2000] Grill cet al. 2020). Some other works seek to improve representation performance by increasing trans- formation views with complicated augmentations. In this paper, we focus on improving contrastive leariing by studying the projection head.
instance-level discrimination task. It narrows
As shown above, the projection head serves a bridge to squeeze out redundant information in order to retrain invariant information from the backbone for the contrastive loss module. However, only the last output of the projection head used, while hidden layers are overlooked, which loses many valuable information according to the information bottleneck theory(Tishby et al.) 2000 Tishby & Zaslavsky 2015). Inspired by the successful results knowledge distillation in supervised repre- sentation learning, we propose cross-level contrastive learning method for (Sun et al.) 2019). In this framework, contrastive losses are applied in the intermediate layers of projection heads. In particular, cross-layer assignment is adapted to create pairs between different views, as the rep- resentation generated at the same level of projection heads of different transformed views is likely to retrain similar information, which is not conducive to learning extra knowledge through contrast-

---

Under review as a conference paper at ICLR 2022
ing. The consistency assumption over the representation in different views and different layers has been partially explored in many feature regularization approaches Guo et al. 2020; Hou et al. 2019 Wang et al. 2019 Huang al. 2020). This phenomenon is investigated in BYOL, which shows that the use of different projection heads for contrastive learning can achieve better performance.
In this paper, series of experiments and analyses are proposed to validate our contributions. The experimental results show the effectiveness of our method in improving the robustness of the pre- training model for downstream tasks. Specifically, in the ImageNer(Deng uation task, the standard ResNeet550(H et al. 2016) trained by our method achieves 73.5% top-1 accuracy. In the object detection and instance segmentation tasks, our method also outperforms ran- dom initialization and existing image- self--sue methods on PASCAL VOC(Everingham et al. 2010) and COCO(Lin et al., 2014) datasets. Last but not the least, compared to other SSI. methods, HCCL. requires fewer training epoch to learn effective representations. For example, it only takes 100 epochs to achieve 69.5% top-1 accuracy in the ImageNet linear evaluation task, while other methods may take 200 epochs more. Our main contributions can be summarized as follows:
al., 2009) linear eval-
In this paper, we propose Hierarchical Cross Contrastive Learning (HCCL) method to explore information from hidden layers of the projection head. Experiments show that our method can achieve tate-of-the-an results in series of standard benchmarks.
Ablation studies are proposed to evaluate the modules CLIC. Experiments show that cross-level assignments, multi-layer outputs and multi-predictors support the final perfor- mance improvements.
We evaluate the generalization of HCCL method in both SimSiam and BYOL frameworks. The experiments show that HCCL can significantly improve performance and speed up convergence.
2 RELATED WORK
Self-supervised learning or unsupervised visual representation learning has undergone a relatively long period of development. Early works use generative methods to restore the distribution of input
data, usually including auto-encoders (Kingma & Welling [013]] Vincent et al. 2008) and adver- sarial learning (Goodfellow
al., 2014; Donahue et al.] 2016 Dumoulin et al. 2016). However, these methods are intended to restore the high-level details of the image that are not necessary for downstream tasks. On the other hand, pixel-level reconstruction also costs lot of calculations.
Some other methods learn specific representations by handcrafted pretext tasks. These pretext tasks include colorization
et al.) 2016 Larsson et al. 2016). jigsaw puzzle & Favaro, 2016 Misra et al., 2016), relative patch prediction (Doersch et al. 2015). inpainting(Pathak et al. geometric transformations/ Dosovitskiy et al. 2014), etc. However, the design of excuse tasks relies strong prior knowledge, and the representations learned by these methods usually lack generality in downstream tasks.
2016
Another type of works is the clustering based self-training (Bautista al. 2016: Asano al. 2019 Caron et af. 2018) 2019 2020). In general, they cluster the representations and learn to predict the cluster assignment. DeepCluster (Caron al. 2018 2019) uses k-means create pseudo labels for samples. It clusters data using previous versions of representation and uses the cluster index of each sample as classification target for the new representation. SwAV calculates the assignment from one view and predicts it from another (Caron et al. (2020). It produces soft assignment by the Sinkhorn- Sinkhorn-Knopp algorithm (Cuturi 2013) and performs online clustering under a balanced parti- tion constraint for each batch. Clustering-basee methods require a costly clustering phase to avoid collapsing, and large memory bank or large batches to provide sufficient samples for clustering.
Contrastive learning measures the similarity of pairs of samples in the representation space by using contrast loss(Hadsell et al.) 2006). brings the representation of different views the same sam- ple closer and pushes the representation of views from different samples away. Dosovitskiy et al. (014)) considers each image in a dataset as its own class represented by a feature vector and sepa- rates each image. Since this method produces huge classification header that is difficult to handle, Wu et al. (2018) use memory bank that stores replace the classifier. Several later works also follow and extend this approach(Misra & Maaten
instance representation vectors
2

---

Under review as a conference paper at ICLR 2022
predictor 4
encoder f
hierarchical projection hend %b
L
view
imput image
h,
x
contra. loss
IH
X,
view
eneoder Sa
hierarchical projection head db
HCCL's framework. HCCL minimizes the cross contrastive loss between the different level output of hierarchical projection head. The encoder parameters and projection parameters of the two branches are shared. '=' denotes stop gradient.
Figure I:
2020 Tian et al. 2020 Zhuang et al. 2019). MoCo(He et al., 2020) reduces the size of the memory bank by maintaining a dynamic queue that stores negative sample representations. The representa- tion in the queue is generated by moving average encoder to improve consistency with the query representation. SimCLR R(Chen et al. 2020) removes the memory bank and uses the negative sam- ples that exist in the current batch directly for learning. It introduces nonlinear projection heads and complex composition of image transformations to improve the robustness of representations in downstream tasks. BYOL a slow-moving average target network to produce stable targets for the online network. The online network bootstraps its own representation by keeping pace with stable targets. By analysing certain conditions for avoiding collapsing solutions. SimSiam(Chen & He (222)) removed the momentum encoder on the basis B YOL. BARLOW TWINSZbonntar al. 2021) proposes an objective func- tion that naturally avoids collapse by measuring the cross-corre cons-coorelation matrix between the outputs of two siamese networks fed with different views of a sample and bringing it as close as possible to the identity matrix. It causes the representation of different views of sample to be similar, while minimizing the redundancy between the components of these representations. The above- mentioned self-supervised learning methods all express learning image-level predictions using global representations, and the gap between image-level pre-training and dense prediction tasks still exists. Some works propose pixel-level self-supervised learning methods improve the effect of pre-training representations on dense prediction tasks Wang et al. 2021 Roh et al. 2021
al. 2020) further examines the need for negative samples, using
Xie et al.
2021 Pinheiro al. 2020).
METHOD
In this section, we describe the proposed HCCL in detail. First, we briefly introduce the pipeline of the proposed HCCL shown in Figure] I including the design of hierarchical projection network that maps representations to different latent spaces. Then we introduce the cross contradictive loss, which allows explicit information interactions between different latent layers. Finally, we present the implementation details in HCCL, such as image transforms, architecture, and optimization.
3.1 FRAMEWORK
HCCL builds on the cross-view prediction framework introduced in recent successful contrastive learning approaches(Chenn
al. 2020 Grill et al. 2020 Chen & He 2021h. The cross-view pre- diction framework is based on a prediction problem that the representation of a transformed view of an image should be predictive of the representation of another transformed view of the same image. HCCL transforms an image X randomly to two distorted views X,, and X,, by applying transfor- mations sampled from the set T of image transformations. The two distorted views X, and X,,
3

---

Under review as a conference paper at ICLR 2022
then fed to non-linear mapping encoder fe. typically deep network with trainable parameters 8, producing two raw representations yaand yb respectively. For fo. HCCL allows various choices of the network architecture without any constraints. y,and y/, representations used for transfer and yb. a hierarchical neural network projection
tasks. After extracting the raw representation head 9ø that maps them different latent space, produces embedding Z. (Zal; Za2, ..... Zan and 2h = {zb1, .62,.... Zbn J. Further, n prediction MLP head, denoted as h1, hn. trans-
forms the projection embedding žn and b to 2n vectors Pa {Pa1 Po21, Po2....;Par and Pg {P61, P82, Pbn J. The following formulas summarizes the above pipeline:
2 nb (X.))
(1)
2 gø (fe (Xo))
(2)
Pa: = h;(
(3)
Ph: (2
where yo fx y/ (X) and E f1,2,.... n). Follow the same way SimSiam Chen & He He|[202]] (021) and BYOL(Grill et 2020). HCCL matches one view's prediction vector to the other view's projection embedding finally. The difference that we have multiple projection embedding and prediction vectors, and cannot directly use the original contrastive loss, so we designed a cross contrastive loss solve it.
3.2 CROSS CONTRASTIVE LOsS
Since Z" are extracted from different levels of the projection head, the high-level representations are more invariant and the low-level representations are more informative. HCCI, adopts an intu- itive approach, that is, cross contrastive representations of different levels to trade off invariant and informative. Given the projection embedding # and the prediction vector p. we minimize their mean squared error of l2- formomall vectors:
p.z
D(p, x) = |pl ||
(5)
where ||.1|2 is 122 mo-mm.. After that, we define the cross contrastive loss as:
Lerose (P", D(p1, 2n) + Σ D(p..
(6)
1=2
Here each level of prediction vector contrast with the previous level projection representation except the first level one. The first level of the prediction vector contrast with the final level of pro- jection representation. Following BYOL and SimSiam, We symmetrize the loss Leroes in Eq.6 and take stop- gradgnra operation projection embedding Z". The form of total loss implemented as:
Leross (Fn stopgrad(Z{)) Lerosa (Pg sZ)))rad(Z?))
(7)
Here the encoder only receives gradient from branch P. and Pain the two terms. The pseudo-code provided in Algorithm refalgorithm
3.3 IMPLEMENTATION DETAILS
Image augmentations We use the same image augmentation set and parameters as BYOL et al. 2020). Firstly, two different views are randomly cropped from each input image and resized 224 224, followed by horizontal flipping, color distortion, converting to grayscale, Gaussian blurring, and polarization. The color distortion consists of random sequence of brightness, con- trast, saturation, hue adjustments. Random cropping and resizing are always applied, others are applied randomly with certain probability. For gaussian blurring and polarization, the probabilities applicd on the two distorted views are different.
Architecture We use ResNet-50(He et al. 2016) our base parametric encoder network, the fully- connected layer after global average pooling is removed and replaced by hierarchical projection network. Specifically, We adopt 2-level projector network that projects the output of the global average pooling layer to two latent spaces. Each level projector network has three linear layers and each layer has 2048 output units. The first two linear layers of each level projector are followed
4

---

Under review as a conference paper at ICLR 2022
Algorithm PyTorch-style pseudocode for HCCL
f: encoder networks
g: hierarchical projetcion head (with two levels)
h1, h2: two prediction networks
n: batch size
for in loader: # load minibatch with samples
x_å, x_b augment (x)
# two augmented views
y_a, y_b (x_a), (x
# encoder output
z_al, z_a2 gyy_a)
# proj outputs of view a
2_bl, 2_b2 (yy_b)
# proj outputs of view b
P_al, p_a2 hl[_a_l), h2 (z_a2)
4 pred outputs of view a
p_bl, p_b2 h1 hl[bb]),, h2 (z_b2)
# pred outputs of view b
loss = [p((p,, z_b2) /2 L(p_a2,
z_b1) + \
L(p_bl, 2_a2) /2
z_al)
loss (backaadd)))) optimizer.step
# back-propagate
def Ltt, s)
s detach ()
stop gradient
t = normalize (t)
s normalize (s)
return - (t s) sum (dim=l) mean ()
by batch normalization layer and rectified linear units(ReLU). Each level projector network is followed by a prediction network which has two linear layers. The dimension of the prediction network's input and output is 2048 and the hidden layer's dimension is 512. The prediction network has batch normalization and ReLU applied to its hidden layer. Its output layer does not have batch normalization ReLU.
We adapt the same optimization protocol described in SimSiam et al. 2020). We use SGD optimizer( You al. (2017) with momentum parameter of 0.9. We use cosine decay learning rate without restarts and warm-up. We the initial learning rate 0.025 and multiply the learning rate by the batch size and divide it by 256(Goyal et al. 2017). In addition, we use a global weight decay parameter of 1.5-10-4 while excluding the biases and batch normalization parameters from weights. Following SimCLR(CChe et al., (2020) We use batch normalization synchronized across devices. Training distributed across V100 GPUs and takes approximately 230 hours for a ResNet-50. For 100, 200, 400, and other less epoch training, we set the initial learning rate to 0.05 and weight decay parameter to 1-10-4. Since SimSiam(Chee et al. 2020) and BYOL explained that the predictor network should adapt to the latest representation, so it is not necessary to force to converge before the representations. one predictor constant and set the learning rate of the other predictor to 10 times the initial learning rate and decay normally.
Optimization
& Hutter 2016) and train for 800 epochs with batch size of 512,
al., (020) have
our experiment, we keep the learning rate of
EXPERIMENTS
In this section, we analyze the performance of HCCL's representations after self-supervised pre- training on the training set of the ImageNet ILSVRC-2012 dataset(Deng al. 2009) by transfer learning on different datasets and tasks. Firstly, we evaluate the pre-trained representation on Ima- geNet in both linear classification frozen features and semi-supervised learning. Then assess its transfer capabilities on other vision datasets and tasks, including classification, segmentation. and object detection.
5

---

Under review as a conference paper at ICLR 2022
Table Top-1 accuracy under linear evaluation ImageNet. All methods are based ResNet- 50 pre-trained with two 224x224 views. Evaluation is on a single crop and all best results are in bold. 'f" denotes improved reproduction from SimSiam.
Method
Encoder
momentum encoder
100ep
200ep
300ep
400ep
800ep
SimCLR
R50
66.5
68.3
69.8
70.4
SWAVI
R50
66.5
69.1
70.7
71.8
Barlow Twins
R50
71.4
73.2
MoCoV2
R50
67.4
69.9
71.0
72.2
BYOL
R50
66.5
70.6
72.5
73.2
74.3
DINO
VIT-S
67.8
72.5
SimSiam
R50
68.1
70.0
70.8
71.3
HCCL
R50
69.5
71.4
72.4
73.5
4.1
LINEAR AND SEMI-SUPERVISED EVALUATIONS IMAGENET
Linear evaluation ImageNet We first verify our method by training a linear classifier on Ima- geNet on fixed representations, which are from ResNet's global average pooling layer. Following a common protocol, the linear classifier is trained for 90 epochs with batch size of 1024 and cosine learning rate schedule. We set the initial learning rate to 0.1 and multiply the learning rate by the batch size and divide by 256. We minimize the cross-entropy loss with the LARS optimizer with a momentum of 0.9 and weight decay of O. At training time, the input image is randomly cropped, resized 224x224, and randomly flipped horizontally. At test time, the image is resized to 256x256 and center-cropped to a size of 224x224.
We compare with the state-of-the-art frameworks in Table on ImageNet linear evaluation. All competitors are based on original papers or reproduction trained in SimSiam & He 202T). All competitors are based on a standard ResNet50, with two 224x224 views used during pre-training.
Table reports the top-1 accuracies obtained on the ImageNet validation set and the main proper- ties the methods. HCCL achieves competitive results with a top-1 accuracy of 73.5%, which comparable to the state-of-the-ar methods. HCCL is trained with small batch size and used neither negative samples nor momentum encoder. achieved the best results among methods under 100-epoch and 200-epoch pre-training. HCCL only 0.8% below the performance of BYOL et al. 2020). BYOL uses a momentum encoder, which means a greater training cost, because requires 4 times forward in iteration, while HCCL only requires 2 times. In addition, HCCL can also be ficxibly applied to the SSL method with momentum encoder. We evaluated HCCL with momentum encoder, achieved better performance than BSee Appendix A.2)
Semi-supervised learning on ImageNet We simply fine-tune the whole ResNet-50 pre-trained with method small subset of ImageNet's train and evaluate the performance. We sample
Table 2: Semi-supervised learning on Ima- geNet. All models are finetuned with % and 10% training examples. Results for the su- pervised method are from Zhai et al. 2019).
Table Transfer learning on image clas- sification task. We report top-1 accuracy on i8a118 and Places-205 datasets, and classifi- cation mAP on VOCO7.
Method
TOP-1
TOP-5
Method
iNat18
Place-205
VOCO7
1%
10%
1%
10%
Supervised
46.7
53.2
87.5
Supervised
25.4
56.4
48.4
80.4
PIRL
29.7
51.0
78.8
PIRL
57.2
83.8
SimCLR
37.2
52.5
85.5
SimCLR
48.3
65.6
75.5
87.8
MoCov2
38.6
51.8
86.4
BYOL
53.2
68.8
78.4
89.0
SWAV
39.5
52.8
86.4
SWAV
53.9
70.2
78.5
89.9
BYOL
47.6
$4.0
86.6
Barlow Twins
55.0
69.7
79.2
89.3
Barlow Twins
46.5
54.1
86.2
HCCL
56.4
71.3
80.1
90.4
HCCL
47.2
54.5
86.6

---

Under review as a conference paper at ICLR 2022
Table 4: Transfer Learning on object detection and instance segmentation task. All models use the C4-backbone. We benchmark finetuned representations on the object detection task on VOC07+12 using Faster R-CNN and on the detection and instance segmentation task on COCO using Mask R-CNN(Ix schedule).
Method
VOC 07+12 det
COCO Det
COCO instance
AP
AP75
AP
AP30
AP75
AP
AP
AP75
Apm
Random Init.
59.0
31.6
32.8
50.9
35.3
32.8
47.9
32.0
29.9
Supervised
81.3
58.8
53.5
58.2
41.2
38.2
54.7
35.2
33.3
SimCLR
81.8
61.4
55.5
57.7
40.9
37.9
54.6
35.3
33.3
MoCoV2
82.3
63.3
57.0
58.8
42.5
39.2
55.5
36.6
34.3
BYOL
81.4
61.1
55.3
57.8
40.9
37.9
54.3
35.0
33.2
SWAV
81.5
61.4
55.4
57.6
40.3
37.6
54.2
35.1
33.1
SimSiam
82.4
63.7
57.0
59.3
42.1
39.2
56.0
36.7
34.4
Barlow Twins
82.6
63.4
56.8
59.0
42.5
39.2
56.0
36.5
34.3
HCCL
82.6
64.3
57.2
60.1
42.3
39.5
56.5
36.7
34.6
1% and 10% of the labeled ImageNet training datasets in a class-balanced way by following Sim- CLR(Chen et al., 2020). The image augmentations are the same as in the linear evaluation setting. We train for 20 epochs with batch size of 256 and momentum of 0.9. We minimize the cross- entropy loss with the SGD optimizer with Nesterov momentum and do not use any regularization methods. We
learning rate of 0.02 for the conv-net weights and 300 for the final linear layer, and we decay the learning rates with cosine learning rate schedule. Table Table[]shows the comparisons of our semi-supervised results against recent methods. We report both top-1 and top-5 accuracies on the ImageNet validation set. HCCL is either on par or slightly outperforms previous competing semi-supervised and supervipe approaches.
4.2 TRANSFER TO OTHER DATASETS AND TASKS
Image classification with fixed features We follow the exact settings from PIRL Misrra & Maaten 2020) for training and evaluating linear classifiers on the iNaturalist2018 ( Horn et al. (018)) Places205 al. 2014) and VOCO7 (vvrrinhven et al. 2010) datasets. For VOCO7, We train linear SVM on the global average pooled final representations. For iNaturalist0018 and Places-205, we train a linear classifier with SGD using learning rate of 0.01 reduced by factor of 10 two equally spaced intervals, batch size of 256, a weight decay of 0,0001, and SGD momentum of 0.9. We train the linear models for 30 epochs on Places-203 and 84 epochs on iNat18. We report the top-1 accuracy computed using the 224 224 center crop on the validation set. The results in Table 3 show that HCCL almost outperforms all prior works on three datasets.
Object Detection and Instance Segmentation We evaluate them to Docalization-based tasks, like object detection and instance segmentation. Following the setup in MOCO(He et al. 2020), fine-tune all layers of the pre-trained models end-to-end in et al. 2010) and coco(Lin et al. 2014) datasets. We use Faster R- CNn al. 2015) with an R50-C4 backbone on VOC and use Mask R-CNN(He et al., 2017) with an R50-C4 backbone on COCO.
representations by transferring
the 12C07+12
We use the detection models as implemented in Detectron2 library Wu al. 2019) for training and closely follow the finetune and evaluation settings from the public codebase of MoCo. For Pascal VOC, We use the VOC07+12 trainval set for training, and report results on the VOCO7 test set averaged over 5 independent runs. We train Faster R-CNN C4 backbone for 24K iterations using a batch size of 16 across 8 GPUs using SyncBatchNorm. We set the initial learning rate to 0. and reduce it by factor of 10 after 18K and 22K iterations. We use linear warmup(Goyal et al. (017)) with a slope of 0.333 for 1000 iterations. For COCO, We use the COCO 2017 train split for training and report results the val split averaged over independent runs. We set the initial learning rate to 0.03 and keep the other parameters the same as the 1x schedule in MoCo's codebase.
In Table4 we show that HCCL outperforms among these leading methods both Pascal VOC and COCO datasets. means that HCCL's representations are transferable beyond the classification task.

---

Under review as a conference paper at ICLR 2022
bad
piocaka
paraatoo boad
piokekas hand
prokder
prodeses
(a)
(b)
(c)
(d)
Figure 2: Comparison of different projection architectures. The projection head includes all layers that can be shared between both branches. '=' denotes stop gradient.
ABLATIONS
We present extensive ablation experiments HCCL give intuition of its behavior and per- formance and show how each component contributes, We train HCCL for 100 epochs among all ablation experiments, which performs consistent results compared to our baseline training of long epochs. For reproducibility, all the experiment results are averaged over 3 independent trials. For all the experiments weight decay to 10-4 as in SimSiam(Cher) et al. (020).. We report the top-1 accuracy of training linear classifiers on the ImageNet under linear evaluation protocol the same in Section 4
this section, set the initial learning rate to 0.05 with batch size 512, the
5.1 INFLUENCE OF HIERARCHICAL PROJECTION HEAD AND CROSS CONTRASTIVE LOSS
In this subsection, we alter our hierarchical projection head and cross contrastive loss in several ways to test the necessity of each term of them. Figure 2 recapitulates the different structure and learning ways on projection head and Table shows their results on a linear evaluation benchmark of Imagenet.
Deep projection with single level We first confirm that the gain of HCCL deep projection head with a single level. In Figure 2(a), we use 6 layer projection head the same as section only single-level representation (the output of the final linear layer) are used to construct the contrastive loss. Compared to HCCL, the performance drops by 0.8%, which means that the lack of hicrarchical representation and cross contrastive loss will cause performance degradation.
brought about by
Hierarchical projection head without cross contrastive loss In Figure 2(b), use 6 layer projection head the same as in section 4 and multi-level representation (the output of the final linear layer and the 3rd linear layer) used to construct the contrastive loss. Instead of cross contrastive between different levels, we contrast representations of the same level with each other. Compared with HCCL the performance reduced by 0.8%, which means that hierarchical representations must be combined with a cross contrastive loss to improve performance.
Hierarchical projection head with single cross contrastive loss Figure 2(c) and Figure 2(b) the same hicrarchical projection structure, the difference is that in Figure 2(c) we use the high- level feature of viewl I and the low-level feature of view2 for cross contrastive. Compared with the
Table 5: Influence of hierarchical projection head and cross contrastive loss. We report the top-1 accuracy of models trained with different choices on hierarchical projection head, cross contrastive loss, and multi predictor.
Hierarchical
Cross Contrastive Loss
Multi Predictor
Top-1 Acc
68.6
68.6
68.4
68.7
68.4
69,5
8

---

Under review as a conference paper at ICLR 2022
Table 6: The levels and layers of projection head. We examine the effect of different lev- els and layers(in one level) for the hierarchi- cal projection head. We do apply ReLU activation nor a batch normalization on the fi- nal linear layer of our MLPs.
Table 7: Learning rate of Predictor. We re- port top-1 accuracy at 100 epochs when ap- plying a multiplier A to the low-level predic- tor and A to the high-level predictor learn- ing rate, 'fixed' denotes using constant learn- ing rate.
Layers in one level
An
Fixed
10
20
40
3
4
Fixed
68.3
69.2
69.4
69.2
69.4
2
68.4
69.5
68.8
5
68.9
69.1
69.3
69.2
69,5
Levels
69.0
69.1
68.2
A
10
69.4
69.4
69.3
69.3
20
69.5
69.2
69.4
69.1
69.3
69.0
68.9
67.7
40
69.3
69.4
69.4
69.5
standard HCCL structure, the cross contrastive between the low-level feature of view1 and the high- level of view2 is removed. It brings 1.0% drops compared with HCCL. Obviously that only a one- way contrastive between high-level and low-level representations cannot improve the performance.
Hierarchical projection head with single predictor In Figure 2(d), we share a single predictor among all level projections, which brings 0.7% drops. The output representations of different levels of projection heads are distributed in different spaces, and the use of shared single predictor will destroy this distribution and lead to worse performance.
Deep projection with multi predictor We confirm that the gain HCCL not brought about by 6 layer projection head the same in section [ The output of the projection head fed to two different predictors, and then the contrastive loss is constructed. Compared to HCCL,, the performance drops by 1.1 %.
multi predictor. In Figure 2(c),
5.2 LEVELS AND DEPTH PROJECTOR NETWORK
Table 66 shows the influence of projector architecture on HCCL. We examine the effect of different levels and depths for the projector network. Using the default projector network of level 2 and depth 3 yields the best performance. In addition, we find that our model performs worse when the projector network has more layers, with a saturation of the performance for layers and 2 levels.
5.3 PREDICTOR LEARNING RATE
In this subsection, examine the effect of the combination of learning rates used by different pre- dictors in HCCL. BYOL (Grill et al. (220)) and SimSiam(Chen & He (021) explains that predictors should adapt to the latest representations and keep near-optimal, use a sufficiently large learning rate of predictor provides reasonably good performance. We use the default HCCL structure of level 2 and depth with 2 different predictors. We multiply the learning rate of the predictors by constant X compared to the learning rate used for the rest of the network; all other hyper-parameten are unchanged.
provides the performance of the two predictors using constant learning rate or different À times learning rate. Results show that the predictor learning rate needs to be much higher than the projector learning rate.
CONCLUSION
We propose HCCL, simple yet effective method strategy that applies to class of self-supervised learning. The main idea of HCCL is to explore multiple outputs from hidden layers instead of the output of the last one, and to compare latent features on different levels and views. We show that HCCL regulates not only invariant multiple hidden levels, but also across different levels, im- proving the generalization ability of the learned visual representations. We also show that HCCL can work in various SSL frameworks like BYOL and Simsiam. The results of the experiments sug- gest that HCCL is particularly effective when computing resources are limited, which is widespread in practice.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. arXiv preprint arx05:111..0537,, 2019.
Miguel A Bautista, Artsiom Sanakoyeu, Ekaterina Tikhoncheva, and Bjor Ommer. Cliqueenn: Deep unsupervised exemplar learning. Advances in Neural Information Processing Systems, 29: 3846-3854, 2016.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 132-149, 2018.
Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pre-training of image features on non-curated data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2959-2968. 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Thirty-fourth Conference on Neural Information Processing Systems (NeurIPS), 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26:2292-2300, 2013.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pp. 1422-1430, 2015.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell, Adversarial feature learning, arXiv preprint arXhv:1605.09782, 2016.
Alexey Dasovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discrimi- native unsupervised feature learning with convolutional neural networks. Advances neural information processing systems, 27:766-774, 2014.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Ar- jovsky, and Aaron Courville. Adversarially learned inference. arXiv preprint /00.:i::60600000000 2016.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2): 303-338, 2010.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski. Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv: 176.06777 2017.
10

---

Under review as a conference paper at ICLR 2022
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733. 2020.
H. Guo, K. Zheng, X. Fan, Yu, and S. Wang. Visual attention consistency under image transforms for multi-label image classification. In 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2020.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference Computer Vision and Pattern Recogni- tion (CVPR 06), volume 2, pp. 1735-1742. IEEE, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9729-9738, 2020.
Yuenan Hou, Zheng Ma, Chunxiao Liu, and Chen Change Loy. Learning lightweight lane detection cnns by self attention distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1013-1021, 2019.
Zeyi Huang, Yang Zou, Vijayakumar Bhagavatula, and Dong Huang. Comprehensive attention self- distillation for weakly-supervised object detection. arXiv preprint arXiv:2010.1202. 2020.
Diederik P Kingma and Max Welling. arXiv:1312.6114 2013.
Auto-encoding variational bayes.
arXiv preprint
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for auto- matic colorization. In European conference on computer vision, pp. 577-593. Springer, 2016.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608. 03983, 2016.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representa- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6707-6717, 2020.
Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In European Conference on Computer Vision, pp. 527-544. Springer, 2016.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning visual representations by solving jigsaw puzzles. In European conference on computer vision, pp. 69-84. Springer, 2016.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2536-2544, 2016.
Pedro O Pinheiro, Amjad Almahairi, Ryan Y Benmalek, Florian Golemo, and Aaron Courville. Unsupervised learning of dense visual representations. arXiv preprint arXiv:201 1.05499, 2020.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28: 91-99, 2015.
11

---

Under review as a conference paper at ICLR 2022
Byungseok Roh, Wuhyum Shin, Ildoo Kim, and Sungwoong Kim. Spatially consistent represen- tation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1144-1153, 2021.
Dawei Sun, Anbang Yao, Aojun Zhou, and Hao Zhao. Deeply-supervised knowledge synergy. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 6997-7006, 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Eesion-ECC2200 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16. pp. 776-794, Springer, 2020.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEFE Information Theory Workshop (ITW), pp. 1-5. IEEE, 2015.
Naftali Tishby, Fernando Pereira, and William Bialek. The information bottleneck method. arXiv preprint physicc/0004005,, 2000.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pictro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8769-8778, 2018.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103, 2008
Lezi Wang, Ziyan Wu, Srikrishna Karanam, Kuan-Chuan Peng, Rajat Vikram Singh, Bo Liu, and Dimitris Metaxas. Sharpen focus: Learning with attention separability and consistency. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 512-521, 2019.
Xinlong Wang. Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pp. 3024-3033, 2021.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Yen Lo, and Ross Girshick. Detectron2. http:///g..tuuu.ccooaaeeeeeeeooa
wwwpeeeeerrreeeteetttteteeeee 2019.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3733-3742, 2018.
Enze Xie, Jian Ding, Wenhai Wang, Xiaohang Zhan, Hang Xu, Zhenguo Li, and Ping Luo. Detco: Unsupervised contrastive learning for object detection. arXiv preprint arXiv:2102.04800 2021.
Yang You, Igor Gitman, and Boris Ginsburg. Scaling sgd batch size to 32k for imagenet training. arXiv preprint arXiv: 777.03888 6:12, 2017.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arXiv:2 103.03230, 2021.
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S41: Self-supervised semi-
supervised learning. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1476-1485, 2019.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European confer- ence on computer vision, pp. 649-666. Springer, 2016.
Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene recognition using places database. Advances in Neural Information Processing Systems, 27:487-495 2014.
Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6002-6012 2019.
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 THE PROOF OF HCCL'S CONVERGENCE
We consider a deep linear system where x E R the input image, W E Rmxd extract d- dimensional latent features from E. The augmented version of is E. The fe layer Wie gener- ates the final label y Wie Wx. The ground-truth is denoted as y² = w W' The ma- trix W1 Rm denotes the stopping-gradient copy of W1 R W1 StopGrad(Wii. Similary, W2, W2, Pi, P2, Pl, P2 E Rmxm, The learning rates for (Wi, W2) and (Pt (Tw, } respectively.
With out loss of generality, we assume the features of be divided into two parts 11 22 where I1 corresponds to the augmentation invariant part and 22 corresponds to the augmentation parts, Similarily, &b = [2] We futher assume that the invariant part and augmentation part are independent and the augmentation is ii.d. generated, that is, B{xx } = Iaxd. E{z 0, E = Aligned with æ, matrix W can be divided into two parts too, W VU Ideally, | 0 and | = 0 gives the optimal solution. The measure the quality of self-supervised learning, we define o | as the recovering error of invariant feature subspace. p means that the
Ixxm E{2y2² ] (dd-m) [[ } = E[
SSL algorithm finds the perfect W Wwithout lossing any userful information.
In our method, the loss function
L - ae {
1 + PPW2Waa-Wiee
The gradients are
= ( - W2 W
=P.WW T - Wiig (::iaag((:00))) W;
-PA(UUT +VVT)- W.UU
Vp l Em.. {PPWW a - Wi W WW," T
= -P3WWWiW W - Widiag(I: (I:) W,
PUU VV W T UU UU,,,
Vw, TwiC=Ban E.. PP T (PW x W. WI
W P. (P T W2 Wi W
-P. - P W, W diagg( (I:0+
+ W P2 P.W.WW, We Ps (Idiag(I:O)
=P P.W, P. W₂ U::0 We Ps P.W.W, - W₂ T P2 T [U:0]
WW,C Ez P. (.W W x W)+)2 W1
=P. WW
T - P2 Widiag( :
=Pa PUU +VV - TUU
From Vw, C, we get
VoL=P. - P W.U - WT P P.W.U WT T P. T
VyL WT P3 P.W.V
13

---

Under review as a conference paper at ICLR 2022
0.1
0.0
-0.1
-0.5
0.0
0.5
Figure 3: Differential equation trajectories
The SGD iterations are then
((++1) W(O))
Fw Vw,
W (++1) W
Tw V W2C
p(+1) p(d) 1/ V. C
((+1) P!d) 7pp NnFC
Clearly, the above SGD iterations correspond to highly non-lineat differential equation systems in high dimensional space. If we do not make further simplifications, there is little we can do analytically. To get deeper insight, we consider special case where all matrices are diagonal ones. This simplification allows us to discuss the dynamics analytically. To ensure that the discussion under such simplification can generalize to the original problem, we verify our main conclusion via numerical simulation.
To continuous our discussion, we assume that 2m, diag (u (2
(m)
W
(uig(w,) (1) wg2 .... w/ (m)), Pi diag(p)) (2) ,... Pi (m)) ). P2 diag(p) P22 P2
P2 mm
To avoid notation clutter, the superscript is omitted in the following discussion when is clear from the context.
The differential equations for the above SGD iterations are
ú (pr W2 - p + p2w2 - -2
à (p + vŹ
ج (u. + 0² )3) Pnu
P1 mp("²² + u² 1²)p1 + 7p upu2w2
P2 (p? u² 20bp2 + 1p 22u2
Again, these equations are high-order ones. We use numerical simulation to plot their trajectories. We initialized all variables in { where = 0.1 is small number. The SGD learning rate The 0.1, 7/p = 1.0. Since there are variables (including {u, v)), we try all the possible initialization combinations and obtain 32 trajectories in Figure 3 The x-axis is u(t) and the y-axis is v(t) The marker X indicates the termination point the numerical simulation.
easy to see that all 32 trajectories converge to lim,-roc v(t) 0 and lim,-to u(t) +0.5 * 0. This means that the differential equations indeed converge to the ideal solution where the coefficient of the augmentation subspace converges zero and the coefficient of the invariant subspace converges to non-zero fixed point.
A.2 HCCL WITH MOMENTUM ENCODER
For fair comparison with BYOL, we also evaluated HCCL with momentum encoder. In this periment, HCCL has the same architecture as Figure I but the parameters of the two branches are no longer shared. As in BYOL, the online network(the branch with predictor) is defined weights 8, and is comprised three stages: an encoder fo. a hierarchical projection head gā and predictor hy h2,.... hy- The target network(the branch without predictor) uses different set of
set of
14

---

Under review as a conference paper at ICLR 2022
Table 8: Top-1 accuracy under linear evaluation on ImageNet. 'f" denotes improved reproduc- tion from SimSiam.
Method
Encoder
100ep
200ep
400ep
800ep
BYOL
R50
66.5
70.6
73.2
74.3
HCCL Momentum Encoder
R50
70.9
72.8
74.2
74.9
weights Ç. and is comprised of two stages: encoder fg. hierarchical projection head gç. For coder and projection head, the target parameters ç are an exponential moving average of the online parameters Given a target decay rate [0, 1]. after each training step perform the following update:
= (1 -)0
We set 0.996, other training settings are the same as section Limited by training resources, we give the results under 100, 200, 400, 800 epoch in Table [ Compared with BYOL, HCCL with momentum encoder is significantly improved.
A.3 MEMORY FOOTPRINT AND TRAINING SPEED.
We compared the memory footprint and speed of SimSiam, BYOL, and HCCL during training. All experiments are trained on 8 x V100 GPU with batchsize of 512 and mixed precision. (hblesssowss that HCCL hardly increases memory footprint and training time compared SimSiam and BYOL.
A.4 DIFFERENT WAY OF PROSS-CORRELATION.
We further study the performance of establishing cross-correlation relationships in different ways. When the level of the hierarchical projection head increases, the cross-correlation combination be- tween different levels will become correlation combination based on the 3 levels hierarchical projection heads. Table [10 shows that cross-correlation combinations between different levels can help improve performance. With more levels of hierarchical projection architecture, the cross coos-correlation method will become very compli- cated and cannot be verified one by one through experiments. We believe that the more cross-level cross-correlation relationships are established, the better the performance will be. We will explore this in future work.
complicated. In this subsection, explored the cross-
A.5 ANALYZE OF HIERARCHICAL FEATURE.
In this subsection, analyze the features extracted by the hierarchical projection head further explore the mystery behind HCCL.
A.5.1 Similarity between features different levels of hierarchical projection head.
Using SimSiam and HCCL, we extracted the features of 50000 images on the Imagenet Val dataset. Each image is randomly augmented obtain views. For SimSiam, each image can extract
Table 9: Memory footprint and Training speed. 'Memory' refers to the memory foot- print single gpu during training. 'ME' means momentum encoder.
Table 10: Top-1 accuracy under different
way of cross-correlation. All method use level hierarchical projection head.
Method
Top-1 Acc
Method
Encoder
Memory
Speed
68.4
SimSiam
R50
8220M
1290 im/s
BYOL
R50
8400M
975 im/s
b
68.7
HCCL
R50
8320M
1220 im/s
69.2
HCCL ME
R50
8660M
940 im/s
15

---

Under review as a conference paper at ICLR 2022
hicrarchical projection head s
hicrarchical projection head si
hierarchical projection head di
predictor
predictor
predictor h
hg
loss
loss
loss
hierarchical projection head d
hierarchical projection head
hierarehical projection head o;
(b)
Figure 4: Different way of cross-correlation on 3 level hierarchical projection head. ==' denotes stop gradient.
two features, and calculate the similarity (cosine distance) of the two features. For HCCL, use 2-level hierarchical projection head, so each image can extract four features, including two high-level projectors features and two low-level projector features. We calculate the similarity of high-level features, the similarity of low-level features, and the similarity between high-level and low-level features. Figure shows the cosine distance distribution of all samples in the data set. We observed that the features extracted in the same layer of the projector are very similar in both HCCL and SimSiam, and most of their cosine distances less than 0.2. For HCCL, there large differences between low-level and high-level projector features. This is consistent with our view that different layers the projector extract different information. Intuitively, contrastive learning between too similar features is not good choice.
A.5.2 Feature similarity within and between classes.
We extracted the hierarchical projector features of 50000 images on the Imagenet Val dataset. Each sample extracts low-level and high-level features respectively. For each class, calculate the aver- age cosine distance between all samples in the class, and the results are shown in Figure 6(a). Then we calculate the average cosine distance between the samples of this class and the samples of other classes, the results are shown in Figure 6(b). We can observe that the average cosine distance within classes is smaller than that between classes, whether it is low-level or high-level projector features. The difference is that the average cosine distance of high-level projector features is greater than that low-level projector features, both within and between classes. We believe that the high-level projector makes semantic disturbance the low-level features, making the distribution of features more divergent. HCCL's hierarchical projection head introduces diversity to the features, which is equivalent different 'data augmentation' at the feature level. Finally, We cross contrast loss to learn invariance in these features that experience different disturbances.
16

---

Under review as a conference paper at ICLR 2022
Distance Slemšiam Proj Feet
Distance Of HCCL Proj1 Fest
1750
1020
1500
1250
550
Numbar
400
530-
250
3.0
0.2
0.4
06
6.
1.0
1.2
0.0
0.2
0.4
0.6
088
Cosini Distance
Coolni Distance
(a)
(b)
Distance Of HCCL Proj1 And Proy2 Feat
Distance Of HCCL Proj2 Feet
2500
200
150
kumbers
1300
1200
0.50
1.00
1.05
1.10
0.4
3.0
Coslnee Datance
Cosine Distance
(d)
Figure 5: Cosine distance distribution of viewl and view2 features. (a): SimSiam's projector
features. (b) HCCL's low level projector features. (c) HCCL's low level and hightlevel projector
features. (d) HCCL's high level projector features.
Mean cosine distance within class
Mean cosine distance between class
lon, Jove feature
1.0
aodd fusture
HighLkooe future
light Joes feature
6.8
Moan Cashe
Mean Cashe
6.2
0.0
0.0
Classss
Classes
(a)
(b)
Figure 6: Average cosine distance within and between classes. (a): Average cosine distance of features in each class. (b): Average cosine distance of features between different class.
17


Reviews:
[{"review_id": "qYc0-fF03GD", "reviewer": "Reviewer_RrYz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper starts with the motivation of self-supervised learning methods followed by the summarisation of the important methods. Afterward, the paper observes the trend of the existing methods being imag-level predictions utilizing global representations. Paper argues that the gap between the image-level pre-training and downstream tasks is not filled yet. It sounds interesting but it is not clearly mentioned how this gap exists in a more detailed manner.  As this paper advocates the need for hierarchical cross contrastive learning, a clear insight on the issue with the methods relying only on a global representation learning would make this paper stronger. \n\nThe method is straight forward and it is clearly present in the paper. Pseudocode has made even easier to understand the pipeline. \n\nAnother strong point about this paper is the extensive evaluations. Although the performance improvement over the existing methods is marginal in most of the cases, the pair have done evaluations on multiple benchmarks and also present the ablation studies.\n\n", "summary_of_the_paper": "This paper presents a hierarchical cross contrastive self-supervised learning framework for learning visual representation. This paper proposes to project the representations of an image and its augmented version to multiple latent spaces and also make predictions on each of the latent spaces.  A contrastive loss between the features of different projection levels is minimized to learn the parameters. Experiments on the image classification and detection benchmarks are evaluated. A comparison between important existing methods is also done in the paper. ", "summary_of_the_review": "The research problem is interesting.  Novelty on the idea is modest and the performance compared with the existing method is also modest.\n\nThe presentation of the paper is good but it lacks a clear insight on the motivation for hierarchical architecture and also the way cross-correlation is established. What would be the performance when cross-correlation has been established in a different manner than that present in the paper? As there is potential to have different combinations of cross-correlation dependencies between the layers.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "oQ9993zTm9S", "reviewer": "Reviewer_mW6T", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. The proposed approach, especially the cross contrastive loss, is novel and interesting.\n2. The proposed approach is fairly easy to implement.\n3. The proposed approach is evaluated on multiple benchmarks and achieves noticeable improvements compared with previous states of the art.\n\nWeaknesses\n1. Although empirically evaluated, the intuition behind the cross contrastive loss is still not clear. More theoretical analysis or interpretation should be provided to understand why cross contrastive loss works better. Specifically, what is the difference in features learned with and without cross contrastive loss? Will the features across different projector layers become closer after trained with cross contrastive loss?\n2. The training protocol for serval downstream tasks is not consistent with previous work. Since the authors simply report the performance from existing literature, it would be more critical to use consistent training protocols for a rigorous comparison. For example,\n    - Table 2 in this manuscript reported the results of Table 1 in Barlow Twins [1]. However, Barlow Twins uses a learning rate of 0.002 for the conv-net weights and 0.5 for the final linear layer, while HCCL uses 0.02 and 300, respectively.\n    - Similarly, Table 3 reported the results of Table 3 in Barlow Twins. But the training epochs for Places-205 is 14 for Barlow Twins, but 30 for HCCL.\n3. On page 6, the authors pointed out that \"BYOL has a greater training cost, because It[it] requires 4 times forward in one iteration.\" However, HCCL has much more trainable parameters by introducing more projectors and predictors. Taking two projector heads as example, HCCL has about 35% more parameters compared with BYOL (~49M for HCCL and ~36M for BYOL). It would more reasonable to compare the memory footprint and time during training instead of forward-propagation times.\n4. In Table 7, the \"Fixed\" learning rate is confusing. Does it mean the predictors us the same learning rate as the rest of the network?\n5. In Table 8, what is the performance of HCCL+Momentum Encode for 400ep and 800ep? Can it outperform BYOL? If not, what is the possible reason that HCCL cannot benefit more from longer training time compared with BYOL and Barlow Twins?\n6. There are several typos in the manuscript:\n   - In the caption of Figure 1, ’̸=’ should be `=’;\n   - Page 6, \"because It requires 4 times...\" should be \"because it requires 4 times...\";\n   - Page 7, 2nd line, missing space in \"Table2\";\n   - Page 8, 3rd paragraph, missing space in \"section4\".\n\n[1] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Ste ́phane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arXiv:2103.03230, 2021.", "summary_of_the_paper": "This manuscript proposed a new contrastive self-supervised learning approach (HCCL). Compared with exiting work such as BYOL and SimSiam, HCCL introduced (1) multiple hierarchical projectors and predictors; and (2) contrastive loss calculated across different layers of projectors. HCCL is empirically evaluated on several self-supervised learning benchmarks (e.g., iNat18, Place-205, and COCO instance seg, etc.) and achieves noticeable improvement compared with previous states of the art (e.g., SWAV, BYOL, SimSiam, and Barlow Twins). Additionally, ablation studies are provided to verify the significance of hierarchical projectors, cross contrastive loss, and higher learning rates of the predictor.", "summary_of_the_review": "The authors proposed an interesting hierarchical comtrastive learning approach that achieves noticeable improvement over serval benchmarks. \n\nHowever, it would be more convincing if some theoretical analysis or empirical interpretation could be provided to help the readers understand why hierarchical project heads and cross contrastive loss works better. Additionally, some justification regarding downstream task training protocols should be provided for the sake of fair and rigorous comparison with previous work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jkT4R54PjCw", "reviewer": "Reviewer_TpLR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The consistency assumption over the representation in different views and different layers has been partially explored in many feature regularization approaches. In most cases, this assumption was directly implemented with a consistency loss between feature pairs.  While the consistency across views is well established as the basis of contrastive learning, most practices showed that the features are only partially similar among the last few layers. For this reason, researchers have to carefully select the layers and parameters in tuning the performance. see examples of related work in [1-4].\n\nThe authors address the limitation of the assumption above with several transformation modules between compared features. The performance did show good improvement in the cases of limited training data and short training epochs. The improvement for the full training sets and full training epochs are, however, limited. This may lead to the concern that the information introduced by the proposed approach has been naturally learned with the existing datasets and training settings.\n\nReference:\n[1] Hao Guo, Kang Zheng, Xiaochuan Fan, Hongkai Yu, and Song Wang. Visual attention consistency under image transforms for multi-label image classification. CVPR, 2019. \n[2] Yuenan Hou, Zheng Ma, Chunxiao Liu, and Chen Change Loy. Learning lightweight lane detection CNN by self-attention distillation. ICCV, pages 1013–1021, 2019. \n[3] Lezi Wang, ZiyanWu, Srikrishna Karanam, Kuan-Chuan Peng, Rajat Vikram Singh, Bo Liu, and Dimitris N Metaxas. Sharpen focus: Learning with attention separability and consistency. ICCV, 2019.\n[4] Zeyi Huang, Yang Zou, Vijayakumar Bhagavatula, Dong Huang, Comprehensive Attention Self-Distillationfor Weakly-Supervised Object Detection, NeurIPS, 2020.", "summary_of_the_paper": "The authors propose an extension to the contrastive learning based representation learning approach. The proposed method, call Hierarchical Cross Contrastive Learning(HCCL), leverages the features in different levels and views for more consistent features over the standard CL using only the features of the final layers. The effectiveness is validated in classification, detection, segmentation, and few-shot learning tasks. ", "summary_of_the_review": "The approach proposed in this paper is straightforward and well implemented. The DNNs trained by this approach benefit mostly in the case of small training sets and epochs. The advancement over the full training settings is still small due to the natural limitation of the consistency assumption mentioned above. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JITKcCheBPZ", "reviewer": "Reviewer_NpsJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strenghs\n- The paper writing is clear.\n- The cross-level contrastive learning shows the improvements in comparison to previous methods.\n\n\nWeaknesses\n\nIn general, the contributions of the paper is incremental from previous works as most of the building blocks of the proposed method are adopted from SimSiam (Chen & He, 2021) and BYOL (Grill et al. 2020).\n\n1. Novelty: the main difference between the proposed and SimSiam approaches are the hierarchical projection heads and the cross contrastive loss between each head in a branch with the head of previous level in the other branch. \nAlthough the experimental results show some improvements, the intuition of cross contrastive loss to make improvement is unclear.\nFor example, as the encoder f already extract high-level features, which information is further embedded in the hierarchical projection heads?\nWhich aspect (i.e. feature distributions, embedded information, etc.) can be enhanced in cross contrastive loss to help the improvements?\n\n2. Experiments: As illustrated in ablation study (Table 5, 6, 7), the improvements seem to be the results of several tuning steps (i.e. number of additional levels, predictor learning rate). In particular, the adaptation of Hierarchical and Contrastive loss only cannot help to improve the performance as presented in Table 5.\nThe authors should analyze in details to show the actual improvements obtained by the proposed approach.\n", "summary_of_the_paper": "In this paper, the authors proposed a Hierarchical Cross Contrastive Learning (HCCL) method for Self-supervised learning (SSL) of visual representation.\nThe proposed method include a design of a hierarchical projection network that produces multi-level latent representations. A cross contrastive loss is also introduced to learn invariant visual representations.\nHCCL is validated on several downstream tasks including classification, segmentation and object detection.", "summary_of_the_review": "Although the experimental results show some improvements, the contributions of the paper is incremental from previous works and the intuition for the improvement is unclear.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In general, the reviewers were lukewarm about the paper. They all acknowledged the strength of the paper: it is well written, HCCL showed (somewhat) improvements over previous methods, and it is easy to implement. However, it still feels incremental, and the improvement over the full training setting is small due to the natural limitation of consistency assumption. The AC feels that while there is merit of the proposed method, the impact seems to be limited to specific scenarios such as limited epochs.
Decision: Reject