Consistent Instance Classification for Unsupervised Representation Learning

1 INTRODUCTION . Learning good representations from unlabeled images is a land-standing and challenging problem . The mainstream methods include : generative modeling ( Hinton et al. , 2006 ; Kingma & Welling , 2014 ) , colorization ( Zhang et al. , 2016 ) , transformation or spatial relation prediction ( Doersch et al. , 2015 ; Noroozi & Favaro , 2016 ; Gidaris et al. , 2018 ) , and discriminative methods , such as instance classification ( Dosovitskiy et al. , 2016 ; He et al. , 2019 ) , and contrastive learning ( Chen et al. , 2020b ) . The instance discrimination methods show promising performance for downstream tasks . There are two basic objectives that are optimized ( He et al. , 2019 ; Chen et al. , 2020b ; Yu et al. , 2020 ; Wang & Isola , 2020 ) : contraction and separation . Contraction means that the features of the augmented views from the same instance should be as close as possible . Separation means that the features of the augmented views from one instance should lie in a region different from other instances . The instance classification framework , such as InstDisc ( Wu et al. , 2018 ) , and MoCo ( He et al. , 2019 ; Chen et al. , 2020c ) , adopts a prototype-based classifier , where the prototype is estimated as the moving average of the corresponding features of previous epoches Wu et al . ( 2018 ) or as the output of an moving-average network He et al . ( 2019 ) ; Chen et al . ( 2020c ) . The prototype-based schemes ease the optimization of the classification loss in the challenging case that there is over one million categories . BYOL ( Grill et al. , 2020 ) computes the prototype in a way similar to MoCo , and only aligns the feature of augmented views with its prototype leaving the separation objective implicitly optimized . The prototype , computed from a single view rather than many views and from networks with different parameters , might not be reliable enough , making the contraction and separation optimization quality not guaranteed . The contrastive learning framework1 , such as SimCLR ( Chen et al. , 2020b ) and Ye et al . ( 2019 ) , simultaneously maximizes the similarities between each view pair from the same instance and min- 1InstDisc ( Wu et al. , 2018 ) and MoCo ( He et al. , 2019 ; Chen et al. , 2020c ) are also closely related to contrastive learning and are regarded as contrastive learning methods by some researchers . imizes the similarities between the view pair from different instances . This framework directly compares the feature of one view to a different view other than to a prototype , avoiding the unreliability of the prototype estimation . It , however , requires large batch size for each SGD iteration to compare enough number of negative instances for imposing the separation constraint2 , increasing the difficulty in large batch training . We propose a simple unsupervised representation learning approach , consistent instance classification ( ConIC ) , to improve the optimization and feature quality . Our approach jointly minimizes two losses : instance classification loss and consistency loss . The instance classification loss is formulated by regarding each instance as a category . Its optimization encourages that different instances lie in different regions . The consistency loss is formulated to directly compares the features of the augmented views from the same instance and encourages high similarity between them . One benefit from the consistency optimization is to directly and explicitly make the features of the same instances compact and thus to accelerate the optimization of the classification loss . This is different from Wu et al . ( 2018 ) , He et al . ( 2019 ) , heuristically estimating the classifier weights using the prototypes and does not suffer from the prototype estimation reliability issue . On the other hand , our approach does not rely on large batch training , that is essential for SimCLR ( Chen et al. , 2020b ) , because the whole loss in our formulation can be decomposed as a sum of components each of which only depends on one instance . Furthermore , we observed that jointly optimizing the consistency and classification losses leads to that the representation is more focused on the textured region , as shown in Figure 1 . This implies that the learned representation is more capable of characterizing the objects , and thus potentially more helpful for downstream tasks like object detection and segmentation . We demonstrate the effectiveness of our approach in unsupervised representation learning on ImageNet . Our approach achieves competitive performance under the linear evaluation protocol . When finetuned on downstream tasks , such as object detection on VOC , object detection and instance segmentation on COCO , instance segmentation on Cityscapes and LVIS , as well as semantic segmentation on Citeyscapes , COCO Stuff , ADE and VOC , our approach performs better than InstDisc , MoCo and SimCLR , and competitively or superior compared to other methods with stronger training setting ( e.g. , InfoMin and SwAV ) . 2 RELATED WORK . Generative approaches . Generative models , such as auto-encoders ( Hinton et al. , 2006 ; Kingma & Welling , 2014 ; Vincent et al. , 2008 ) , context encoders ( Pathak et al. , 2016 ) , GANs ( Donahue & Simonyan , 2019 ) , and GPTs ( Chen et al. , 2020a ) , learn an unsupervised representation by faithfully reconstructing the pixels . Later self-supervised models , such as colorization ( Zhang et al. , 2016 ) and split-brain encoders ( Zhang et al. , 2017 ) , improve generative models by withholding some part of the data and predicting it . 2We will show one possible reason that it requires large batch . Spatial relation prediction . The representation is learned by solving pretext tasks related to image patch spatial relation prediction , such as predicting the spatial relation between two patches sampled from an image , e.g. , a patch is on the left of another patch , ( Doersch et al. , 2015 ) ; solving Jigsaw Puzzles and determining the spatial configuration for the shuffled ( typically 9 ) patches ( Noroozi & Favaro , 2016 ) ; and predicting the rotation ( Gidaris et al. , 2018 ) . Instance classification . Exemplar-CNN ( Dosovitskiy et al. , 2016 ) regards the views formed by augmenting each instance as a class , and formulates an instance classification problem . InstDisc ( Wu et al. , 2018 ) , MoCo ( He et al. , 2019 ) , CMC ( Tian et al. , 2019 ) and PIRL ( Misra & van der Maaten , 2019 ) generalize exemplar-CNN by heuristically estimating the classifier weights using prototypes for easing the optimization . Our proposed approach follows the instance classification approach , and exploit an additional consistency loss to help optimization . Instance clustering . Rather than regarding each instance as a category , the instance clustering frameworks ( Caron et al. , 2018 ; 2019 ; 2020 ; Asano et al. , 2020 ; Huang et al. , 2019 ; Xie et al. , 2016 ; Yan et al. , 2020 ; Yang et al. , 2016 ) learn representations in which the instances are well optimized . DeepCluster ( Caron et al. , 2018 ) simply adopt the k-means clustering method by simultaneously optimizing the network parameters , and uses k-means assignments as pseudo-labels to learn representations . SwAV ( Caron et al. , 2020 ) simultaneously clusters the data while enforcing consistency between cluster assignments for different views of the same instance . Contrastive learning . Contrastive predictive coding ( van den Oord et al. , 2018 ; Hénaff et al. , 2019 ) predicts the representations of patches below a certain position from those above it by optimizing contrastive loss . DIM ( Hjelm et al. , 2019 ) and ANDIM ( Bachman et al. , 2019 ) achieves global-tolocal/local-to-neighbor patch representation prediction ( overlapping ) across augmented views using the contrastive loss . The contrastive learning framework ( Ye et al. , 2019 ; Chen et al. , 2020b ) formulates a contrastive loss encouraging the high similarity between the augmented views from the same instance , and low similarity between the instance and other instances . Wang & Isola ( 2020 ) presents a novel formulation based on two measures : alignment and uniformity , and shows that it is an alternative of contrastive loss . Yu et al . ( 2020 ) connects contractive and contrastive learning , cross-entropy , and so on , and provides theoretical guarantees for learning diverse and discriminative features . Consistency in semi-supervised learning . Consistency regularization , enforcing the similarity between the predictions or features of different views for the same unlabeled instance , has been widely applied in semi-supervised learning , such as Π Model ( Laine & Aila , 2017 ) , Temporal Ensemble ( Laine & Aila , 2017 ) , and Mean Teacher ( Tarvainen & Valpola , 2017 ) . We exploit the consistency loss to help optimize the classification loss for unsupervised representation learning . 3 APPROACH . Given a set of image instances without any labels , I = { I1 , I2 , . . . , IN } , the goal is to learn a feature extractor ( a neural network ) x = f ( I ) . The discrimination approach expands each image In to a set of augmented views { I1n , I2n , . . . , IKn } , and formulates the problem in a way that the features of the augmented views of each instance are similar ( contraction ) and the features of different instances are distributed separately ( separation ) . In the following , we first review three related instance classification methods , then we introduce our approach and present the analysis . 3.1 INSTANCE CLASSIFICATION . Exemplar CNN . Exemplar-CNN ( Dosovitskiy et al. , 2016 ) formulates unsupervised representation learning as an instance classification problem . The augmented views from one instance are regarded as one category , and the augmented views from different instances are regarded as different categories . The softmax loss is used and written for the kth view of the nth instance : ` s ( x k n ) = − log ew > n x̃ k n/τ∑N j=1 e w > j x̃ k n/τ , ( 1 ) where τ is the temperature . Exemplar-CNN uses the standard backpropagation algorithm to learn the network f ( · ) and the classification weights { w1 , w2 , . . . , wN } . InstDisc . The InstDisc approach ( Wu et al. , 2018 ) optimizes the network parameters , and heuristically estimates the classifier weights { w1 , w2 , . . . , wN } in each epoch using a feature moving average scheme , i.e. , compute the exponential averages of the features of the corresponding instances ( stored in a memory bank ) in the previous epochs . The heuristic weight estimation scheme eases the network optimization . MoCo . MoCo ( He et al. , 2019 ) instead adopts a network moving average scheme . In each SGD iteration , MoCo updates a momentum network whose parameters are moving average of the previous network parameters . It computes the features from the momentum network as the classifier weights , which are further maintained by a queue . This leads to better classifier weight estimates . 3.2 CONSISTENT INSTANCE CLASSIFICATION . We introduce a consistency loss to explicitly penalize the dissimilarity between augmented views from the same instance . Let sim ( u , v ) = u > v/‖u‖‖v‖ denote the inner product between ` 2 normalized u and v , i.e . cosine similarity . The consistency loss for two views xin and x j n from the image In is formed as ` c ( x i n , x j n ) = ( 1− sim ( xin , xjn ) ) 2 = ( 1− x̃in > x̃jn ) 2 . ( 2 ) Here , we normalize the feature vector x̃ = x/‖x‖2 as done in InstDisc and MoCo . The consistency loss for the N images each with K augmented views is written as Lc = ∑N n=1 ∑K i , j=1 , i6=j ` c ( x i n , x j n ) = ∑N n=1 ∑K i , j=1 , i6=j ( 1− x̃in > x̃jn ) 2 . ( 3 ) The classification loss for the N images each with K augmented views is written as Ls = ∑N n=1 ∑K k=1 ` s ( x k n ) = − ∑N n=1 ∑K k=1 log ew > n x̃ k n/τ∑N j=1 e w > j x̃ k n/τ . ( 4 ) where we let the classifier weight be an ` 2-normalized vector : ‖w‖2 = 1 , which is similar to normalizing the prototype vector as done in InstDisc and MoCo . We combine the two losses together , L = Ls + αLc , ( 5 ) where α is a weight for the consistency loss to balance the two losses , avoiding over-optimizing the consistency loss or merely optimizing the classification loss . Consistency helps optimizing the classification loss . In general , when the features for each class are more compact , different classes are more easily separated and the softmax classification are more efficiently optimized . Our approach has the benefits : the feature distribution for the same instance is compact and the distributions for different instances are well separable , because of maximizing the consistency . Figure 2 ( a ) illustrates the benefit from simultaneously optimizing the consistency loss and the classification loss . Figure 2 ( b ) shows the insufficiency of only optimizing the classification loss . One can see that the distributions of different instances in Figure 2 ( a ) are better separated and the distribution for each instance is more compact . Let ’ s see how the consistency term makes the gradient of the classifier weight more effective . We have the gradient for the classification loss ` s ( xkn ) with respect to the classifier weight wn : ∂ ` s ( x k n ) ∂wn = ( P knn − 1 ) x̃kn , where P knn = e w > n x̃ k n/τ∑N r=1 e w > r x̃ k n/τ . The gradient from two views xin and x j n is gw = ∂ ` s ( x i n ) ∂wn + ∂ ` s ( x j n ) ∂wn = ( P inn − 1 ) x̃in + ( P jnn − 1 ) x̃jn . ( 6 ) According to the law of cosines , ‖x̃in‖2 = 1 and ‖x̃jn‖2 = 1 , we have ‖gw‖22 = ( P inn − 1 ) 2 + ( P jnn − 1 ) 2 + 2| ( P inn − 1 ) || ( P jnn − 1 ) |x̃in > x̃jn . ( 7 ) When the consistency term is included , x̃in and x̃ j n are very close , implying that x̃ i n > x̃jn is larger . In the case P inn and P j nn are not changed , the magnitude ‖gw‖2 is larger , and accordingly the classifier weight wn is updated effectively and quickly . In contrast , when the consistency term is not included , x̃in and x̃ j n might be very diverse as discussed ( see the discussion in “ Optimizing the classification loss is not direct to optimize the consistency loss. ” ) This results in that ‖gw‖2 is smaller , and thus the classifier weight wn is updated less effectively and less quickly . Figure 3 shows the final classification loss Ls and consistency loss Lc value with different consistency loss weights . We can see that increasing the consistency weight when smaller than 2.5 helps optimizing the classification loss , and when larger than 2.5 harms the classification loss optimization . In Appendix B , we discuss the reason : over-weighting the consistency loss could lead to a trivial solution . In our experiments , we set α to 2.5 in which case the training classification loss is minimum . Optimizing the classification loss is not direct to optimize the consistency loss . Optimizing the classification loss intuitively expects that each instance lies in a different region . We expect that the augmented views of an instance xn are assigned to the nth region and compactly distributed . We find that merely optimizing the classification loss Ls is not easy to make the features of the augmented views of the same instance contractive , consequently the features are not compactly distributed . The reason is that larger similarity between augmented views is not explicitly encouraged , and is implicitly imposed through the classifier weight . The angle between x̃in and x̃ j n , θ ( x̃ i n , x̃ j n ) ( reflecting the similarity between x̃in and x̃ j n , θ ( x̃ i n , x̃ j n ) ) , is upbounded : θ ( x̃in , x̃ j n ) ≤ θ ( x̃in , wn ) + θ ( wn , x̃jn ) . ( 8 ) Minimizing the classifier loss Ls if given wn , it is possible that the numerators ( e.g , w > n x̃in and w > n x̃ j n ) , are larger and accordingly the upbound θ ( x̃ i n , wn ) +θ ( wn , x̃ j n ) is smaller . However , we find that there exist many transformations R so that the upbound is the same : w > n ( R x̃ i n ) = w > n x̃ i n , and θ ( x̃in , wn ) = θ ( Rx̃ i n , wn ) . In this case , θ ( x̃ i n , x̃ j n ) is likely to be very different from θ ( Rx̃ i n , x̃ j n ) . This implies that there is still a gap between optimizing the upbound θ ( x̃in , wn ) + θ ( wn , x̃ j n ) and directly optimizing θ ( x̃in , x̃ j n ) . As a result , merely optimizing the classification loss is not easy to make the features for one instance compactly distributed in the corresponding region . Batch size . We present rough analysis showing that instance classification , including our approach , MoCo , and InstDisc , does not require large batch ( see He et al . ( 2019 ) and the empirical validation in Figure 4 for our approach ) . We rewrite the loss function in Equation 5 as L = ∑N n=1 [ α ∑K i , j=1 , i6=j ( 1− sim ( xin , xjn ) ) 2 + ∑K k=1 log ew > n x̃ k n/τ∑N j=1 e w > j x̃ k n/τ ] . ( 9 ) The reformulation indicates that the loss can be decomposed to the sum of components , where each component depends on a different instance . The separation between instances is got through the classifier weights each of which encodes the information of the corresponding instance . The decomposability property leads to that the optimization of L using SGD behaves similarly to the standard classification problem with SGD : large batch size is not necessary . Figure 4 shows that the performances with batch sizes 256 , 512 and 1024 are similar . In contrast , the contrastive loss over all the N instances in SimCLR , is L = ∑N n=1 ( log ex̃ 1 n > x̃2n/τ∑N j=1 e x̃1n > x̃1j/τ + ex̃ 1 n > x̃2j/τ + log ex̃ 2 n > x̃1n/τ∑N j=1 e x̃2n > x̃1j/τ + ex̃ 2 n > x̃2j/τ ) . ( 10 ) We can see that this can not be decomposed as a sum of components each of which depends on a different instance , which is a general requirement for SGD . Each instance depends on other instances . We believe that this is the reason why SimCLR needs large batch size ( Chen et al. , 2020b ) .

Summary: This paper studies the instance classification solution for an unsupervised representation learning problem. Particularly, this paper proposes an additional consistency loss that is simultaneously optimized with classification loss, in order to penalize feature dissimilarity between augmented views of the same instance. Such consistency loss makes classification loss optimization easier and avoids large batch size. Extensive experiments on downstream tasks, e.g., segmentation and detection, show the effectiveness of the proposed method.