Consistent Instance Classification for Unsupervised Representation Learning

1 INTRODUCTION . Learning good representations from unlabeled images is a land-standing and challenging problem . The mainstream methods include : generative modeling ( Hinton et al. , 2006 ; Kingma & Welling , 2014 ) , colorization ( Zhang et al. , 2016 ) , transformation or spatial relation prediction ( Doersch et al. , 2015 ; Noroozi & Favaro , 2016 ; Gidaris et al. , 2018 ) , and discriminative methods , such as instance classification ( Dosovitskiy et al. , 2016 ; He et al. , 2019 ) , and contrastive learning ( Chen et al. , 2020b ) . The instance discrimination methods show promising performance for downstream tasks . There are two basic objectives that are optimized ( He et al. , 2019 ; Chen et al. , 2020b ; Yu et al. , 2020 ; Wang & Isola , 2020 ) : contraction and separation . Contraction means that the features of the augmented views from the same instance should be as close as possible . Separation means that the features of the augmented views from one instance should lie in a region different from other instances . The instance classification framework , such as InstDisc ( Wu et al. , 2018 ) , and MoCo ( He et al. , 2019 ; Chen et al. , 2020c ) , adopts a prototype-based classifier , where the prototype is estimated as the moving average of the corresponding features of previous epoches Wu et al . ( 2018 ) or as the output of an moving-average network He et al . ( 2019 ) ; Chen et al . ( 2020c ) . The prototype-based schemes ease the optimization of the classification loss in the challenging case that there is over one million categories . BYOL ( Grill et al. , 2020 ) computes the prototype in a way similar to MoCo , and only aligns the feature of augmented views with its prototype leaving the separation objective implicitly optimized . The prototype , computed from a single view rather than many views and from networks with different parameters , might not be reliable enough , making the contraction and separation optimization quality not guaranteed . The contrastive learning framework1 , such as SimCLR ( Chen et al. , 2020b ) and Ye et al . ( 2019 ) , simultaneously maximizes the similarities between each view pair from the same instance and min- 1InstDisc ( Wu et al. , 2018 ) and MoCo ( He et al. , 2019 ; Chen et al. , 2020c ) are also closely related to contrastive learning and are regarded as contrastive learning methods by some researchers . imizes the similarities between the view pair from different instances . This framework directly compares the feature of one view to a different view other than to a prototype , avoiding the unreliability of the prototype estimation . It , however , requires large batch size for each SGD iteration to compare enough number of negative instances for imposing the separation constraint2 , increasing the difficulty in large batch training . We propose a simple unsupervised representation learning approach , consistent instance classification ( ConIC ) , to improve the optimization and feature quality . Our approach jointly minimizes two losses : instance classification loss and consistency loss . The instance classification loss is formulated by regarding each instance as a category . Its optimization encourages that different instances lie in different regions . The consistency loss is formulated to directly compares the features of the augmented views from the same instance and encourages high similarity between them . One benefit from the consistency optimization is to directly and explicitly make the features of the same instances compact and thus to accelerate the optimization of the classification loss . This is different from Wu et al . ( 2018 ) , He et al . ( 2019 ) , heuristically estimating the classifier weights using the prototypes and does not suffer from the prototype estimation reliability issue . On the other hand , our approach does not rely on large batch training , that is essential for SimCLR ( Chen et al. , 2020b ) , because the whole loss in our formulation can be decomposed as a sum of components each of which only depends on one instance . Furthermore , we observed that jointly optimizing the consistency and classification losses leads to that the representation is more focused on the textured region , as shown in Figure 1 . This implies that the learned representation is more capable of characterizing the objects , and thus potentially more helpful for downstream tasks like object detection and segmentation . We demonstrate the effectiveness of our approach in unsupervised representation learning on ImageNet . Our approach achieves competitive performance under the linear evaluation protocol . When finetuned on downstream tasks , such as object detection on VOC , object detection and instance segmentation on COCO , instance segmentation on Cityscapes and LVIS , as well as semantic segmentation on Citeyscapes , COCO Stuff , ADE and VOC , our approach performs better than InstDisc , MoCo and SimCLR , and competitively or superior compared to other methods with stronger training setting ( e.g. , InfoMin and SwAV ) . 2 RELATED WORK . Generative approaches . Generative models , such as auto-encoders ( Hinton et al. , 2006 ; Kingma & Welling , 2014 ; Vincent et al. , 2008 ) , context encoders ( Pathak et al. , 2016 ) , GANs ( Donahue & Simonyan , 2019 ) , and GPTs ( Chen et al. , 2020a ) , learn an unsupervised representation by faithfully reconstructing the pixels . Later self-supervised models , such as colorization ( Zhang et al. , 2016 ) and split-brain encoders ( Zhang et al. , 2017 ) , improve generative models by withholding some part of the data and predicting it . 2We will show one possible reason that it requires large batch . Spatial relation prediction . The representation is learned by solving pretext tasks related to image patch spatial relation prediction , such as predicting the spatial relation between two patches sampled from an image , e.g. , a patch is on the left of another patch , ( Doersch et al. , 2015 ) ; solving Jigsaw Puzzles and determining the spatial configuration for the shuffled ( typically 9 ) patches ( Noroozi & Favaro , 2016 ) ; and predicting the rotation ( Gidaris et al. , 2018 ) . Instance classification . Exemplar-CNN ( Dosovitskiy et al. , 2016 ) regards the views formed by augmenting each instance as a class , and formulates an instance classification problem . InstDisc ( Wu et al. , 2018 ) , MoCo ( He et al. , 2019 ) , CMC ( Tian et al. , 2019 ) and PIRL ( Misra & van der Maaten , 2019 ) generalize exemplar-CNN by heuristically estimating the classifier weights using prototypes for easing the optimization . Our proposed approach follows the instance classification approach , and exploit an additional consistency loss to help optimization . Instance clustering . Rather than regarding each instance as a category , the instance clustering frameworks ( Caron et al. , 2018 ; 2019 ; 2020 ; Asano et al. , 2020 ; Huang et al. , 2019 ; Xie et al. , 2016 ; Yan et al. , 2020 ; Yang et al. , 2016 ) learn representations in which the instances are well optimized . DeepCluster ( Caron et al. , 2018 ) simply adopt the k-means clustering method by simultaneously optimizing the network parameters , and uses k-means assignments as pseudo-labels to learn representations . SwAV ( Caron et al. , 2020 ) simultaneously clusters the data while enforcing consistency between cluster assignments for different views of the same instance . Contrastive learning . Contrastive predictive coding ( van den Oord et al. , 2018 ; Hénaff et al. , 2019 ) predicts the representations of patches below a certain position from those above it by optimizing contrastive loss . DIM ( Hjelm et al. , 2019 ) and ANDIM ( Bachman et al. , 2019 ) achieves global-tolocal/local-to-neighbor patch representation prediction ( overlapping ) across augmented views using the contrastive loss . The contrastive learning framework ( Ye et al. , 2019 ; Chen et al. , 2020b ) formulates a contrastive loss encouraging the high similarity between the augmented views from the same instance , and low similarity between the instance and other instances . Wang & Isola ( 2020 ) presents a novel formulation based on two measures : alignment and uniformity , and shows that it is an alternative of contrastive loss . Yu et al . ( 2020 ) connects contractive and contrastive learning , cross-entropy , and so on , and provides theoretical guarantees for learning diverse and discriminative features . Consistency in semi-supervised learning . Consistency regularization , enforcing the similarity between the predictions or features of different views for the same unlabeled instance , has been widely applied in semi-supervised learning , such as Π Model ( Laine & Aila , 2017 ) , Temporal Ensemble ( Laine & Aila , 2017 ) , and Mean Teacher ( Tarvainen & Valpola , 2017 ) . We exploit the consistency loss to help optimize the classification loss for unsupervised representation learning . 3 APPROACH . Given a set of image instances without any labels , I = { I1 , I2 , . . . , IN } , the goal is to learn a feature extractor ( a neural network ) x = f ( I ) . The discrimination approach expands each image In to a set of augmented views { I1n , I2n , . . . , IKn } , and formulates the problem in a way that the features of the augmented views of each instance are similar ( contraction ) and the features of different instances are distributed separately ( separation ) . In the following , we first review three related instance classification methods , then we introduce our approach and present the analysis . 3.1 INSTANCE CLASSIFICATION . Exemplar CNN . Exemplar-CNN ( Dosovitskiy et al. , 2016 ) formulates unsupervised representation learning as an instance classification problem . The augmented views from one instance are regarded as one category , and the augmented views from different instances are regarded as different categories . The softmax loss is used and written for the kth view of the nth instance : ` s ( x k n ) = − log ew > n x̃ k n/τ∑N j=1 e w > j x̃ k n/τ , ( 1 ) where τ is the temperature . Exemplar-CNN uses the standard backpropagation algorithm to learn the network f ( · ) and the classification weights { w1 , w2 , . . . , wN } . InstDisc . The InstDisc approach ( Wu et al. , 2018 ) optimizes the network parameters , and heuristically estimates the classifier weights { w1 , w2 , . . . , wN } in each epoch using a feature moving average scheme , i.e. , compute the exponential averages of the features of the corresponding instances ( stored in a memory bank ) in the previous epochs . The heuristic weight estimation scheme eases the network optimization . MoCo . MoCo ( He et al. , 2019 ) instead adopts a network moving average scheme . In each SGD iteration , MoCo updates a momentum network whose parameters are moving average of the previous network parameters . It computes the features from the momentum network as the classifier weights , which are further maintained by a queue . This leads to better classifier weight estimates . 3.2 CONSISTENT INSTANCE CLASSIFICATION . We introduce a consistency loss to explicitly penalize the dissimilarity between augmented views from the same instance . Let sim ( u , v ) = u > v/‖u‖‖v‖ denote the inner product between ` 2 normalized u and v , i.e . cosine similarity . The consistency loss for two views xin and x j n from the image In is formed as ` c ( x i n , x j n ) = ( 1− sim ( xin , xjn ) ) 2 = ( 1− x̃in > x̃jn ) 2 . ( 2 ) Here , we normalize the feature vector x̃ = x/‖x‖2 as done in InstDisc and MoCo . The consistency loss for the N images each with K augmented views is written as Lc = ∑N n=1 ∑K i , j=1 , i6=j ` c ( x i n , x j n ) = ∑N n=1 ∑K i , j=1 , i6=j ( 1− x̃in > x̃jn ) 2 . ( 3 ) The classification loss for the N images each with K augmented views is written as Ls = ∑N n=1 ∑K k=1 ` s ( x k n ) = − ∑N n=1 ∑K k=1 log ew > n x̃ k n/τ∑N j=1 e w > j x̃ k n/τ . ( 4 ) where we let the classifier weight be an ` 2-normalized vector : ‖w‖2 = 1 , which is similar to normalizing the prototype vector as done in InstDisc and MoCo . We combine the two losses together , L = Ls + αLc , ( 5 ) where α is a weight for the consistency loss to balance the two losses , avoiding over-optimizing the consistency loss or merely optimizing the classification loss . Consistency helps optimizing the classification loss . In general , when the features for each class are more compact , different classes are more easily separated and the softmax classification are more efficiently optimized . Our approach has the benefits : the feature distribution for the same instance is compact and the distributions for different instances are well separable , because of maximizing the consistency . Figure 2 ( a ) illustrates the benefit from simultaneously optimizing the consistency loss and the classification loss . Figure 2 ( b ) shows the insufficiency of only optimizing the classification loss . One can see that the distributions of different instances in Figure 2 ( a ) are better separated and the distribution for each instance is more compact . Let ’ s see how the consistency term makes the gradient of the classifier weight more effective . We have the gradient for the classification loss ` s ( xkn ) with respect to the classifier weight wn : ∂ ` s ( x k n ) ∂wn = ( P knn − 1 ) x̃kn , where P knn = e w > n x̃ k n/τ∑N r=1 e w > r x̃ k n/τ . The gradient from two views xin and x j n is gw = ∂ ` s ( x i n ) ∂wn + ∂ ` s ( x j n ) ∂wn = ( P inn − 1 ) x̃in + ( P jnn − 1 ) x̃jn . ( 6 ) According to the law of cosines , ‖x̃in‖2 = 1 and ‖x̃jn‖2 = 1 , we have ‖gw‖22 = ( P inn − 1 ) 2 + ( P jnn − 1 ) 2 + 2| ( P inn − 1 ) || ( P jnn − 1 ) |x̃in > x̃jn . ( 7 ) When the consistency term is included , x̃in and x̃ j n are very close , implying that x̃ i n > x̃jn is larger . In the case P inn and P j nn are not changed , the magnitude ‖gw‖2 is larger , and accordingly the classifier weight wn is updated effectively and quickly . In contrast , when the consistency term is not included , x̃in and x̃ j n might be very diverse as discussed ( see the discussion in “ Optimizing the classification loss is not direct to optimize the consistency loss. ” ) This results in that ‖gw‖2 is smaller , and thus the classifier weight wn is updated less effectively and less quickly . Figure 3 shows the final classification loss Ls and consistency loss Lc value with different consistency loss weights . We can see that increasing the consistency weight when smaller than 2.5 helps optimizing the classification loss , and when larger than 2.5 harms the classification loss optimization . In Appendix B , we discuss the reason : over-weighting the consistency loss could lead to a trivial solution . In our experiments , we set α to 2.5 in which case the training classification loss is minimum . Optimizing the classification loss is not direct to optimize the consistency loss . Optimizing the classification loss intuitively expects that each instance lies in a different region . We expect that the augmented views of an instance xn are assigned to the nth region and compactly distributed . We find that merely optimizing the classification loss Ls is not easy to make the features of the augmented views of the same instance contractive , consequently the features are not compactly distributed . The reason is that larger similarity between augmented views is not explicitly encouraged , and is implicitly imposed through the classifier weight . The angle between x̃in and x̃ j n , θ ( x̃ i n , x̃ j n ) ( reflecting the similarity between x̃in and x̃ j n , θ ( x̃ i n , x̃ j n ) ) , is upbounded : θ ( x̃in , x̃ j n ) ≤ θ ( x̃in , wn ) + θ ( wn , x̃jn ) . ( 8 ) Minimizing the classifier loss Ls if given wn , it is possible that the numerators ( e.g , w > n x̃in and w > n x̃ j n ) , are larger and accordingly the upbound θ ( x̃ i n , wn ) +θ ( wn , x̃ j n ) is smaller . However , we find that there exist many transformations R so that the upbound is the same : w > n ( R x̃ i n ) = w > n x̃ i n , and θ ( x̃in , wn ) = θ ( Rx̃ i n , wn ) . In this case , θ ( x̃ i n , x̃ j n ) is likely to be very different from θ ( Rx̃ i n , x̃ j n ) . This implies that there is still a gap between optimizing the upbound θ ( x̃in , wn ) + θ ( wn , x̃ j n ) and directly optimizing θ ( x̃in , x̃ j n ) . As a result , merely optimizing the classification loss is not easy to make the features for one instance compactly distributed in the corresponding region . Batch size . We present rough analysis showing that instance classification , including our approach , MoCo , and InstDisc , does not require large batch ( see He et al . ( 2019 ) and the empirical validation in Figure 4 for our approach ) . We rewrite the loss function in Equation 5 as L = ∑N n=1 [ α ∑K i , j=1 , i6=j ( 1− sim ( xin , xjn ) ) 2 + ∑K k=1 log ew > n x̃ k n/τ∑N j=1 e w > j x̃ k n/τ ] . ( 9 ) The reformulation indicates that the loss can be decomposed to the sum of components , where each component depends on a different instance . The separation between instances is got through the classifier weights each of which encodes the information of the corresponding instance . The decomposability property leads to that the optimization of L using SGD behaves similarly to the standard classification problem with SGD : large batch size is not necessary . Figure 4 shows that the performances with batch sizes 256 , 512 and 1024 are similar . In contrast , the contrastive loss over all the N instances in SimCLR , is L = ∑N n=1 ( log ex̃ 1 n > x̃2n/τ∑N j=1 e x̃1n > x̃1j/τ + ex̃ 1 n > x̃2j/τ + log ex̃ 2 n > x̃1n/τ∑N j=1 e x̃2n > x̃1j/τ + ex̃ 2 n > x̃2j/τ ) . ( 10 ) We can see that this can not be decomposed as a sum of components each of which depends on a different instance , which is a general requirement for SGD . Each instance depends on other instances . We believe that this is the reason why SimCLR needs large batch size ( Chen et al. , 2020b ) .

Summary: This paper proposes adding an additional loss term to instance classification, within the context of self-supervised pre-training.  Specifically, in addition to the standard classification loss term that views each image (and its augmentation) as a separate category, a second loss term is added, which is 1 minus the inner product of the representations for two augmentation views of the same image.  In a sense, this is incorporating an aspect of the contrasting learning approach, as two views of the same image are being explicitly pulled toward one another due to the consistency loss.