Hierarchical Cross Contrastive Learning of Visual Representations

1 INTRODUCTION . Self-supervised learning ( SSL ) of visual representation aims to learn representations from image pixels without semantic labels . Recently , contrastive representation learning has gained growing attention by demonstrating state-of-the-art performance in SSL for large-scale image recognition , closing the gap to supervised baselines ( Chen et al. , 2020 ; Grill et al. , 2020 ; Chen & He , 2021 ) . Contrastive representation learning is essentially an instance-level discrimination task . It narrows the distance of the representation generated by the same image , as an instance class , while pushing away the representation generated by other images to learn invariant visual representations . In most contrastive learning methods , the representation invariant is measured by a contrastive loss , which compares one of the network outputs after the projection head to its augmented version . Some works attempt to explore advanced training methods to keep the contrastive loss stable in SSL , by clustering , memory bank , momentum update or projector head ( Caron et al. , 2020 ; He et al. , 2020 ; Grill et al. , 2020 ) . Some other works seek to improve representation performance by increasing transformation views with complicated augmentations . In this paper , we focus on improving contrastive learning by studying the projection head . As shown above , the projection head serves as a bridge to squeeze out redundant information in order to retrain invariant information from the backbone for the contrastive loss module . However , only the last output of the projection head is used , while hidden layers are overlooked , which loses many valuable information according to the information bottleneck theory ( Tishby et al. , 2000 ; Tishby & Zaslavsky , 2015 ) . Inspired by the successful results of knowledge distillation in supervised representation learning , we propose a cross-level contrastive learning method for SSL ( Sun et al. , 2019 ) . In this framework , contrastive losses are applied in the intermediate layers of projection heads . In particular , a cross-layer assignment is adapted to create pairs between different views , as the representation generated at the same level of projection heads of different transformed views is likely to retrain similar information , which is not conducive to learning extra knowledge through contrast- ing . The consistency assumption over the representation in different views and different layers has been partially explored in many feature regularization approaches ( Guo et al. , 2020 ; Hou et al. , 2019 ; Wang et al. , 2019 ; Huang et al. , 2020 ) . This phenomenon is investigated in BYOL , which shows that the use of different projection heads for contrastive learning can achieve better performance . In this paper , a series of experiments and analyses are proposed to validate our contributions . The experimental results show the effectiveness of our method in improving the robustness of the pretraining model for downstream tasks . Specifically , in the ImageNet ( Deng et al. , 2009 ) linear evaluation task , the standard ResNet-50 ( He et al. , 2016 ) trained by our method achieves 73.5 % top-1 accuracy . In the object detection and instance segmentation tasks , our method also outperforms random initialization and existing image-level self-supervision methods on PASCAL VOC ( Everingham et al. , 2010 ) and COCO ( Lin et al. , 2014 ) datasets . Last but not the least , compared to other SSL methods , HCCL requires fewer training epoch to learn effective representations . For example , it only takes 100 epochs to achieve 69.5 % top-1 accuracy in the ImageNet linear evaluation task , while other methods may take 200 epochs or more . Our main contributions can be summarized as follows : • In this paper , we propose a Hierarchical Cross Contrastive Learning ( HCCL ) method to explore information from hidden layers of the projection head . Experiments show that our method can achieve state-of-the-art results in a series of standard benchmarks . • Ablation studies are proposed to evaluate the modules of CLIC . Experiments show that cross-level assignments , multi-layer outputs and multi-predictors support the final performance improvements . • We evaluate the generalization of HCCL method in both SimSiam and BYOL frameworks . The experiments show that HCCL can significantly improve performance and speed up convergence . 2 RELATED WORK . Self-supervised learning or unsupervised visual representation learning has undergone a relatively long period of development . Early works use generative methods to restore the distribution of input data , usually including auto-encoders ( Kingma & Welling , 2013 ; Vincent et al. , 2008 ) and adversarial learning ( Goodfellow et al. , 2014 ; Donahue et al. , 2016 ; Dumoulin et al. , 2016 ) . However , these methods are intended to restore the high-level details of the image that are not necessary for downstream tasks . On the other hand , pixel-level reconstruction also costs a lot of calculations . Some other methods learn specific representations by handcrafted pretext tasks . These pretext tasks include colorization ( Zhang et al. , 2016 ; Larsson et al. , 2016 ) , jigsaw puzzle ( Noroozi & Favaro , 2016 ; Misra et al. , 2016 ) , relative patch prediction ( Doersch et al. , 2015 ) , inpainting ( Pathak et al. , 2016 ) , geometric transformations ( Dosovitskiy et al. , 2014 ) , etc . However , the design of excuse tasks relies on strong prior knowledge , and the representations learned by these methods usually lack generality in downstream tasks . Another type of works is the clustering based self-training ( Bautista et al. , 2016 ; Asano et al. , 2019 ; Caron et al. , 2018 ; 2019 ; 2020 ) . In general , they cluster the representations and learn to predict the cluster assignment . DeepCluster ( Caron et al. , 2018 ; 2019 ) uses k-means to create pseudo labels for samples . It clusters data using previous versions of representation and uses the cluster index of each sample as a classification target for the new representation . SwAV calculates the assignment from one view and predicts it from another ( Caron et al. , 2020 ) . It produces soft assignment by the Sinkhorn-Knopp algorithm ( Cuturi , 2013 ) and performs online clustering under a balanced partition constraint for each batch . Clustering-based methods require a costly clustering phase to avoid collapsing , and a large memory bank or large batches to provide sufficient samples for clustering . Contrastive learning measures the similarity of pairs of samples in the representation space by using contrast loss ( Hadsell et al. , 2006 ) . It brings the representation of different views of the same sample closer and pushes the representation of views from different samples away . Dosovitskiy et al . ( 2014 ) considers each image in a dataset as its own class represented by a feature vector and separates each image . Since this method produces a huge classification header that is difficult to handle , Wu et al . ( 2018 ) use a memory bank that stores previously-computed instance representation vectors to replace the classifier . Several later works also follow and extend this approach ( Misra & Maaten , 2020 ; Tian et al. , 2020 ; Zhuang et al. , 2019 ) . MoCo ( He et al. , 2020 ) reduces the size of the memory bank by maintaining a dynamic queue that stores negative sample representations . The representation in the queue is generated by a moving average encoder to improve consistency with the query representation . SimCLR ( Chen et al. , 2020 ) removes the memory bank and uses the negative samples that exist in the current batch directly for learning . It introduces nonlinear projection heads and complex composition of image transformations to improve the robustness of representations in downstream tasks . BYOL ( Grill et al. , 2020 ) further examines the need for negative samples , using a slow-moving average target network to produce stable targets for the online network . The online network bootstraps its own representation by keeping pace with stable targets . By analysing certain conditions for avoiding collapsing solutions , SimSiam ( Chen & He , 2021 ) removed the momentum encoder on the basis of BYOL . BARLOW TWINS ( Zbontar et al. , 2021 ) proposes an objective function that naturally avoids collapse by measuring the cross-correlation matrix between the outputs of two siamese networks fed with different views of a sample and bringing it as close as possible to the identity matrix . It causes the representation of different views of a sample to be similar , while minimizing the redundancy between the components of these representations . The abovementioned self-supervised learning methods all express learning as image-level predictions using global representations , and the gap between image-level pre-training and dense prediction tasks still exists . Some works propose pixel-level self-supervised learning methods to improve the effect of pre-training representations on dense prediction tasks ( Wang et al. , 2021 ; Roh et al. , 2021 ; Xie et al. , 2021 ; Pinheiro et al. , 2020 ) . 3 METHOD . In this section , we describe the proposed HCCL in detail . First , we briefly introduce the pipeline of the proposed HCCL shown in Figure 1 , including the design of a hierarchical projection network that maps representations to different latent spaces . Then we introduce the cross contradictive loss , which allows explicit information interactions between different latent layers . Finally , we present the implementation details in HCCL , such as image transforms , architecture , and optimization . 3.1 FRAMEWORK . HCCL builds on the cross-view prediction framework introduced in recent successful contrastive learning approaches ( Chen et al. , 2020 ; Grill et al. , 2020 ; Chen & He , 2021 ) . The cross-view prediction framework is based on a prediction problem that the representation of a transformed view of an image should be predictive of the representation of another transformed view of the same image . HCCL transforms an image X randomly to two distorted views Xa and Xb , by applying transformations sampled from the set T of image transformations . The two distorted views Xa and Xb are then fed to non-linear mapping encoder fθ , typically a deep network with trainable parameters θ , producing two raw representations yaand yb respectively . For fθ , HCCL allows various choices of the network architecture without any constraints . yaand yb are representations used for transfer tasks . After extracting the raw representation ya and yb , a hierarchical neural network projection head gnθ that maps them to different latent space , produces n embedding Zna = { za1 , za2 , . . . , zan } and Znb = { zb1 , zb2 , . . . , zbn } . Further , n prediction MLP head , denoted as h1 , h2 , . . . , hn , transforms the projection embedding Zna and Znb to 2n vectors as Pna = { pa1 , pa2 , . . . , pan } and Pnb = { pb1 , pb2 , . . . , pbn } . The following formulas summarizes the above pipeline : Zna = gnθ ( fθ ( Xa ) ) ( 1 ) Znb = gnθ ( fθ ( Xb ) ) ( 2 ) pai = hi ( zai ) ( 3 ) pbi = hi ( zbi ) ( 4 ) where ya = fθ ( Xa ) , yb = fθ ( Xb ) and i ∈ { 1 , 2 , . . . , n } . Follow the same way as SimSiam ( Chen & He , 2021 ) and BYOL ( Grill et al. , 2020 ) , HCCL matches one view ’ s prediction vector to the other view ’ s projection embedding finally . The difference is that we have multiple projection embedding and prediction vectors , and can not directly use the original contrastive loss , so we designed a cross contrastive loss to solve it .

Summary: In this paper, the authors proposed a Hierarchical Cross Contrastive Learning (HCCL) method for Self-supervised learning (SSL) of visual representation. The proposed method include a design of a hierarchical projection network that produces multi-level latent representations. A cross contrastive loss is also introduced to learn invariant visual representations. HCCL is validated on several downstream tasks including classification, segmentation and object detection.