Hierarchical Cross Contrastive Learning of Visual Representations

1 INTRODUCTION . Self-supervised learning ( SSL ) of visual representation aims to learn representations from image pixels without semantic labels . Recently , contrastive representation learning has gained growing attention by demonstrating state-of-the-art performance in SSL for large-scale image recognition , closing the gap to supervised baselines ( Chen et al. , 2020 ; Grill et al. , 2020 ; Chen & He , 2021 ) . Contrastive representation learning is essentially an instance-level discrimination task . It narrows the distance of the representation generated by the same image , as an instance class , while pushing away the representation generated by other images to learn invariant visual representations . In most contrastive learning methods , the representation invariant is measured by a contrastive loss , which compares one of the network outputs after the projection head to its augmented version . Some works attempt to explore advanced training methods to keep the contrastive loss stable in SSL , by clustering , memory bank , momentum update or projector head ( Caron et al. , 2020 ; He et al. , 2020 ; Grill et al. , 2020 ) . Some other works seek to improve representation performance by increasing transformation views with complicated augmentations . In this paper , we focus on improving contrastive learning by studying the projection head . As shown above , the projection head serves as a bridge to squeeze out redundant information in order to retrain invariant information from the backbone for the contrastive loss module . However , only the last output of the projection head is used , while hidden layers are overlooked , which loses many valuable information according to the information bottleneck theory ( Tishby et al. , 2000 ; Tishby & Zaslavsky , 2015 ) . Inspired by the successful results of knowledge distillation in supervised representation learning , we propose a cross-level contrastive learning method for SSL ( Sun et al. , 2019 ) . In this framework , contrastive losses are applied in the intermediate layers of projection heads . In particular , a cross-layer assignment is adapted to create pairs between different views , as the representation generated at the same level of projection heads of different transformed views is likely to retrain similar information , which is not conducive to learning extra knowledge through contrast- ing . The consistency assumption over the representation in different views and different layers has been partially explored in many feature regularization approaches ( Guo et al. , 2020 ; Hou et al. , 2019 ; Wang et al. , 2019 ; Huang et al. , 2020 ) . This phenomenon is investigated in BYOL , which shows that the use of different projection heads for contrastive learning can achieve better performance . In this paper , a series of experiments and analyses are proposed to validate our contributions . The experimental results show the effectiveness of our method in improving the robustness of the pretraining model for downstream tasks . Specifically , in the ImageNet ( Deng et al. , 2009 ) linear evaluation task , the standard ResNet-50 ( He et al. , 2016 ) trained by our method achieves 73.5 % top-1 accuracy . In the object detection and instance segmentation tasks , our method also outperforms random initialization and existing image-level self-supervision methods on PASCAL VOC ( Everingham et al. , 2010 ) and COCO ( Lin et al. , 2014 ) datasets . Last but not the least , compared to other SSL methods , HCCL requires fewer training epoch to learn effective representations . For example , it only takes 100 epochs to achieve 69.5 % top-1 accuracy in the ImageNet linear evaluation task , while other methods may take 200 epochs or more . Our main contributions can be summarized as follows : • In this paper , we propose a Hierarchical Cross Contrastive Learning ( HCCL ) method to explore information from hidden layers of the projection head . Experiments show that our method can achieve state-of-the-art results in a series of standard benchmarks . • Ablation studies are proposed to evaluate the modules of CLIC . Experiments show that cross-level assignments , multi-layer outputs and multi-predictors support the final performance improvements . • We evaluate the generalization of HCCL method in both SimSiam and BYOL frameworks . The experiments show that HCCL can significantly improve performance and speed up convergence . 2 RELATED WORK . Self-supervised learning or unsupervised visual representation learning has undergone a relatively long period of development . Early works use generative methods to restore the distribution of input data , usually including auto-encoders ( Kingma & Welling , 2013 ; Vincent et al. , 2008 ) and adversarial learning ( Goodfellow et al. , 2014 ; Donahue et al. , 2016 ; Dumoulin et al. , 2016 ) . However , these methods are intended to restore the high-level details of the image that are not necessary for downstream tasks . On the other hand , pixel-level reconstruction also costs a lot of calculations . Some other methods learn specific representations by handcrafted pretext tasks . These pretext tasks include colorization ( Zhang et al. , 2016 ; Larsson et al. , 2016 ) , jigsaw puzzle ( Noroozi & Favaro , 2016 ; Misra et al. , 2016 ) , relative patch prediction ( Doersch et al. , 2015 ) , inpainting ( Pathak et al. , 2016 ) , geometric transformations ( Dosovitskiy et al. , 2014 ) , etc . However , the design of excuse tasks relies on strong prior knowledge , and the representations learned by these methods usually lack generality in downstream tasks . Another type of works is the clustering based self-training ( Bautista et al. , 2016 ; Asano et al. , 2019 ; Caron et al. , 2018 ; 2019 ; 2020 ) . In general , they cluster the representations and learn to predict the cluster assignment . DeepCluster ( Caron et al. , 2018 ; 2019 ) uses k-means to create pseudo labels for samples . It clusters data using previous versions of representation and uses the cluster index of each sample as a classification target for the new representation . SwAV calculates the assignment from one view and predicts it from another ( Caron et al. , 2020 ) . It produces soft assignment by the Sinkhorn-Knopp algorithm ( Cuturi , 2013 ) and performs online clustering under a balanced partition constraint for each batch . Clustering-based methods require a costly clustering phase to avoid collapsing , and a large memory bank or large batches to provide sufficient samples for clustering . Contrastive learning measures the similarity of pairs of samples in the representation space by using contrast loss ( Hadsell et al. , 2006 ) . It brings the representation of different views of the same sample closer and pushes the representation of views from different samples away . Dosovitskiy et al . ( 2014 ) considers each image in a dataset as its own class represented by a feature vector and separates each image . Since this method produces a huge classification header that is difficult to handle , Wu et al . ( 2018 ) use a memory bank that stores previously-computed instance representation vectors to replace the classifier . Several later works also follow and extend this approach ( Misra & Maaten , 2020 ; Tian et al. , 2020 ; Zhuang et al. , 2019 ) . MoCo ( He et al. , 2020 ) reduces the size of the memory bank by maintaining a dynamic queue that stores negative sample representations . The representation in the queue is generated by a moving average encoder to improve consistency with the query representation . SimCLR ( Chen et al. , 2020 ) removes the memory bank and uses the negative samples that exist in the current batch directly for learning . It introduces nonlinear projection heads and complex composition of image transformations to improve the robustness of representations in downstream tasks . BYOL ( Grill et al. , 2020 ) further examines the need for negative samples , using a slow-moving average target network to produce stable targets for the online network . The online network bootstraps its own representation by keeping pace with stable targets . By analysing certain conditions for avoiding collapsing solutions , SimSiam ( Chen & He , 2021 ) removed the momentum encoder on the basis of BYOL . BARLOW TWINS ( Zbontar et al. , 2021 ) proposes an objective function that naturally avoids collapse by measuring the cross-correlation matrix between the outputs of two siamese networks fed with different views of a sample and bringing it as close as possible to the identity matrix . It causes the representation of different views of a sample to be similar , while minimizing the redundancy between the components of these representations . The abovementioned self-supervised learning methods all express learning as image-level predictions using global representations , and the gap between image-level pre-training and dense prediction tasks still exists . Some works propose pixel-level self-supervised learning methods to improve the effect of pre-training representations on dense prediction tasks ( Wang et al. , 2021 ; Roh et al. , 2021 ; Xie et al. , 2021 ; Pinheiro et al. , 2020 ) . 3 METHOD . In this section , we describe the proposed HCCL in detail . First , we briefly introduce the pipeline of the proposed HCCL shown in Figure 1 , including the design of a hierarchical projection network that maps representations to different latent spaces . Then we introduce the cross contradictive loss , which allows explicit information interactions between different latent layers . Finally , we present the implementation details in HCCL , such as image transforms , architecture , and optimization . 3.1 FRAMEWORK . HCCL builds on the cross-view prediction framework introduced in recent successful contrastive learning approaches ( Chen et al. , 2020 ; Grill et al. , 2020 ; Chen & He , 2021 ) . The cross-view prediction framework is based on a prediction problem that the representation of a transformed view of an image should be predictive of the representation of another transformed view of the same image . HCCL transforms an image X randomly to two distorted views Xa and Xb , by applying transformations sampled from the set T of image transformations . The two distorted views Xa and Xb are then fed to non-linear mapping encoder fθ , typically a deep network with trainable parameters θ , producing two raw representations yaand yb respectively . For fθ , HCCL allows various choices of the network architecture without any constraints . yaand yb are representations used for transfer tasks . After extracting the raw representation ya and yb , a hierarchical neural network projection head gnθ that maps them to different latent space , produces n embedding Zna = { za1 , za2 , . . . , zan } and Znb = { zb1 , zb2 , . . . , zbn } . Further , n prediction MLP head , denoted as h1 , h2 , . . . , hn , transforms the projection embedding Zna and Znb to 2n vectors as Pna = { pa1 , pa2 , . . . , pan } and Pnb = { pb1 , pb2 , . . . , pbn } . The following formulas summarizes the above pipeline : Zna = gnθ ( fθ ( Xa ) ) ( 1 ) Znb = gnθ ( fθ ( Xb ) ) ( 2 ) pai = hi ( zai ) ( 3 ) pbi = hi ( zbi ) ( 4 ) where ya = fθ ( Xa ) , yb = fθ ( Xb ) and i ∈ { 1 , 2 , . . . , n } . Follow the same way as SimSiam ( Chen & He , 2021 ) and BYOL ( Grill et al. , 2020 ) , HCCL matches one view ’ s prediction vector to the other view ’ s projection embedding finally . The difference is that we have multiple projection embedding and prediction vectors , and can not directly use the original contrastive loss , so we designed a cross contrastive loss to solve it .

Summary: This manuscript proposed a new contrastive self-supervised learning approach (HCCL). Compared with exiting work such as BYOL and SimSiam, HCCL introduced (1) multiple hierarchical projectors and predictors; and (2) contrastive loss calculated across different layers of projectors. HCCL is empirically evaluated on several self-supervised learning benchmarks (e.g., iNat18, Place-205, and COCO instance seg, etc.) and achieves noticeable improvement compared with previous states of the art (e.g., SWAV, BYOL, SimSiam, and Barlow Twins). Additionally, ablation studies are provided to verify the significance of hierarchical projectors, cross contrastive loss, and higher learning rates of the predictor.