Are all negatives created equal in contrastive instance discrimination?

1 INTRODUCTION . In recent years , there has been tremendous progress on self-supervised learning ( SSL ) , a paradigm in which representations are learned using a pre-training task that uses only unlabeled data . These representations are then used on downstream tasks , such as classification or object detection . Since SSL pre-training does not require labels , it can leverage unlabeled data , which is can be abundant and cheaper to obtain than labeled data . In computer vision , representations learned from unlabeled data have historically underperformed representations learned directly from labeled data . Recently , however , newly proposed SSL methods such as MoCo ( He et al. , 2019 ; Chen et al. , 2020c ) , SimCLR ( Chen et al. , 2020a ; b ) , SwAV ( Caron et al. , 2020 ) , and BYOL ( Grill et al. , 2020 ) have dramatically reduced this performance gap . The MoCo and SimCLR pre-training tasks use contrastive instance discrimination ( CID ) , in which a network is trained to recognize different augmented views of the same image ( sometimes called the query and the positive and discriminate between the query and the augmented views of other random images from the dataset ( called negatives ) .1 ) Despite the empirical successes of CID , the mechanisms underlying its strong performance remain unclear . Recent theoretical and empirical works have investigated the role of mutual information between augmentations ( Tian et al. , 2020 ) , analyzed properties of the learned representations such as alignment and uniformity ( Wang & Isola , 2020 ) , and proposed a theoretical framework ( Arora et al. , 2019 ) , among others . However , existing works on CID have not investigated the relative importance or semantic properties of different negatives , even though negatives play a central role in CID . In other areas , works on hard negative mining in metric learning ( Kaya & Bilge , 2019 ) and on the impact of different training examples in supervised learning ( Birodkar et al. , 2019 ) suggest that understanding the relative importance of different training data can be fruitful . 1In MoCo , these are called the query and positive and are treated slightly differently ; in SimCLR , both are treated the same and are called positives . The other SSL methods listed ( not SimCLR and MoCo ) are not CID . In this work , we empirically investigate how the difficulty of negatives affects the downstream performance of the learned representation . We measure difficulty using the dot product between the normalized contrastive-space embeddings of the query and the negative . A dot product closer to 1 suggests a negative that is more difficult to distinguish from the query . We ask how different negatives , by difficulty , affect training . Are some negatives more important than others for downstream accuracy ? If so , we ask : Which ones ? To what extent ? And what makes them different ? We focus on MoCo v2 ( Chen et al. , 2020c ) and the downstream task of linear classification on ImageNet ( Deng et al. , 2009 ) , and have similar results for SimCLR in Appendix A.2 . We make the following contributions ( see Figure 1 for summary ) : • The easiest 95 % of negatives are unnecessary and insufficient , while the top 5 % hardest negatives are necessary and sufficient : We reached within 0.7 percentage points of full accuracy by training on the 5 % of hardest negatives for each query , suggesting that the 95 % easiest negatives are unnecessary . In contrast , the easiest negatives are insufficient ( and , therefore , the hardest negatives are necessary ) : accuracy drops substantially when training on only the easiest 95 % of negatives . The hardest 5 % of negatives are especially important : training on only the next hardest 5 % lowers accuracy by 15 percentage points . • The hardest 0.1 % of negatives are unnecessary and sometimes detrimental : Downstream accuracy was unchanged or improved when we removed these hardest negatives . These negatives were more often in the same ImageNet class as the query , compared to easier negatives , suggesting that semantically identical ( but superficially dissimilar ) negatives were unhelpful or detrimental . • Properties of negatives : Based on our observations that the importance of a negative varies with its difficulty , we investigate the properties of negatives that affect their difficulty . – We found that hard negatives were more semantically similar to the query than were easy negatives : the hardest 5 % of negatives were more likely to be of the same ImageNet class as the query , compared to easier negatives . These hard negatives were also closer to the query as measured by depth of the least common ancestor for the negative and the query in the WordNet tree ( which ImageNet is built upon ) . – We also observed that the pattern is reversed for the ≈50 % of easier negatives : there , the easier the negative , the more semantically similar it is to the query . – There exist negatives that are more consistently hard across queries than would be expected by random chance . We emphasize that our primary aim is to better understand the differences between negatives and the impact of these differences on existing methods rather than to propose a new method . However , our results suggest that there may be unexploited opportunities to reduce the cost of modern CID methods ( Chen et al. , 2020c ) . For any particular query , only a small fraction of the negatives are necessary . Our observations on differences in negative importance can serve as a valuable building block for the understanding and improvement of contrastive learning methods.For example , there may be further room to choose specific examples for training , similar to hard negative mining and curriculum learning ( Chen et al. , 2020a ; Chuang et al. , 2020 ; Kaya & Bilge , 2019 ) , to reduce costs and improve performance per data sample . 2 METHODS AND PRELIMINARIES . Contrastive instance discrimination and momentum contrast . Momentum Contrast ( MoCo v2 ) is a CID method that reaches accuracy within 6 percentage points of supervised accuracy on ImageNet with ResNet-50 ( Chen et al. , 2020c ) . In MoCo , the task is to learn a representation that succeeds at the following : given a query ( an augmented view of an image ) , correctly pick a positive ( a different augmented view of the same image ) from a large set of negatives ( augmented views of randomly chosen images ) . Our experiments focus on aspects that are common between CID methods rather than those specific to MoCo . We discuss implementation details that may be specific to MoCo v2 here . The MoCo v2 encoder is a ResNet-50 network . For pre-training , the outputs of this base network are fed into a multi-layer perceptron ( MLP ) head ; we refer to the normalized output from the MLP head as the contrastive-space embedding . For downstream tasks , the MLP head is discarded and only the base network is used ; we refer to the output of the base network as the learned representation . A distinguishing feature of MoCo is that it has two encoders , one of which is actively trained ( used for the query ) and the other which is a moving average of the trained encoder ( used for the positive and negatives ) . MoCo stores the embeddings of each batch of positives in a large queue and uses them as negatives for future batches , enabling the use of more negatives than can fit in a batch . MoCo uses the InfoNCE loss ( Gutmann & Hyvärinen , 2010 ; van den Oord et al. , 2018 ) : Lq = − log exp ( q · k+/τ ) ∑K i=1 exp ( q · ki/τ ) where q is the embedding of a query ( using the learned encoder ) , k+ is the embedding of a positive ( using the momentum encoder ) , and ki are the embeddings of the negatives in the queue ( added using previous states of the momentum encoder ) . τ is a temperature hyperparameter . Difficulty of negatives . To compute the difficulty for a set of negatives given a particular query , we calculate the dot product between the normalized contrastive-space embedding of each negative with the normalized contrastive-space embedding of the query . We then sort the dot products and consider the negatives with dot products closer to 1 to be harder negatives and those with smaller dot products to be easier negatives . We use this terminology because it fits intuition : all else being equal , harder negatives increase the loss . Since embeddings are normalized , the dot product is the cosine of the angle between the embeddings of the instances and ranges from -1 to 1 . Note that difficulty is defined per query and that it is a function of the current state of the network . Thus , a negative can be easy for some queries and hard for others , and the hardness of a negative for a given query can vary over training epochs and across different training runs and configurations . Experimental setting . Our experiments focus on MoCo v2 ( Chen et al. , 2020c ) , an improved version of MoCo which combines MoCo v1 ( He et al. , 2019 ) with several features of SimCLR ( Chen et al. , 2020a ) . We use ImageNet for pre-training and evaluate performance using linear classification on ImageNet from the representation learned in the pre-training CID task . The network used , as in MoCo v2 , is a ResNet-50 with MLP head , and trained for 200 epochs . Unless otherwise noted , we run each experiment three times with different seeds ; error bars represent mean± standard deviation . 3 WHICH NEGATIVES ARE NECESSARY OR SUFFICIENT ? . We examine which negatives , by difficulty , are necessary or sufficient for learning representations that have strong downstream performance . Outside of CID , there are varying perspectives on the value of easy negatives . Research on hard negative mining suggests that harder negatives can be more important than easier negatives for relevant tasks ( Kaya & Bilge , 2019 ) . However , in some supervised contexts , much or all training data seems important for reaching the highest accuracy ( Birodkar et al. , 2019 ) . We aim to experimentally assess which of these perspectives applies when using MoCo v2 on ImageNet classification . To determine whether a set of negatives was necessary , we removed the corresponding negatives on each pre-training step ; if the resulting representations still led to accuracy close to baseline on the downstream task , then we considered those negatives to have been unnecessary . To determine whether a set of negatives was sufficient , we removed all negatives except those in that range on each pre-training step ; if the resulting representations still led to strong accuracy on the downstream task , then we considered the negatives in that range to have been sufficient.2 The easy negatives are unnecessary ; the hard negatives are sufficient . First , we asked whether the easy negatives were necessary ( or equivalently , whether the hard negatives were sufficient ) . That is , does the network maintain downstream accuracy when it is pre-trained without the easy negatives ? To test this , we evaluated how accuracy changed as different subsets of negatives were removed . Interestingly , we found that using only the hardest 5 % of negatives was largely sufficient to recover baseline accuracy ( Figure 2a-b , 95-100 % ) , suggesting that the overwhelming majority of the easier negatives were unnecessary . Moreover , the hardest 5 % ( 95-100 % ) were substantially more informative than the next 5 % ( 90-95 % ) : top-1 accuracy dropped by only 0.7 percentage points when trained on only the hardest 5 % vs. 15 percentage points for the next hardest 5 % ( 90-95 % ) and 47 percentage points for the third 5 % ( 85-90 % ; Figure 2a-b ) . Going forward , we use 5 % as a cutoff and call the negatives harder than this cutoff hard and those easier than this cutoff easy . The easy negatives are largely insufficient ; the hard negatives are necessary . We next asked whether the easy negatives were sufficient ( or , equivalently , whether the hard negatives were necessary ) . Although we found in the previous section that the easy negatives were unnecessary , that does not necessarily mean they are insufficient . For example , it could be that the easy negatives , while individually less important , collectively provide sufficient signal for learning good representations on the downstream task . Alternatively , it is possible that the information contained in all of the easy negatives still lacks enough signal ; in this case , the easy negatives , even when combined together , would still be insufficient . We found that even when the easiest 95 % of negatives were combined together , accuracy was 5.4 % below baseline ( Figure 2c-d ) . In contrast , recall that using only the hardest 5 % of negatives ( 19x fewer ) achieved top-1 performance within 0.7 % of baseline ( Figure 2a ) . Using the easiest 90 % of negatives harms accuracy even further ( 0-90 % ; Figure 2c-d ) . Together , these results demonstrate that the easiest negatives , even when they comprise the vast majority of negatives , are still insufficient . The very hardest negatives are harmful at lower temperatures . We have found that the hard negatives , i.e . the 5 % hardest , are largely necessary and sufficient for CID . However , top-1 accuracy actually improved slightly when we removed the very hardest 0.1 % of negatives ( p = 0.03 for an 2We removed sets of negatives by treating them as through they were not present in the queue . unpaired t-test ) .3 This effect was most pronounced at lower temperatures ( Figure 3 ) ; for example , at temperature 0.1 ( as opposed to baseline temperature 0.2 ) , training without the hardest 0.1 % of negatives improved downstream top-1 accuracy by 0.23 % ( p = 0.0003 ) and top-5 accuracy by 0.67 % . One might expect such a difference between temperatures because the hardest negatives are weighted more in the loss at lower temperatures . Interestingly , the effect was larger for top-5 accuracy than top-1 accuracy ( compare Figure 3b with 3a ) . One hypothesis for why the hardest negatives hurt is that because negatives are randomly sampled , some negatives can be too similar to the query ( e.g . augmentations of near-duplicates to the query ) . Because negatives are randomly sampled , they can included augmented views of images that are near-duplicates of the query or otherwise visually very similar to the query . If both the query and negative contain identical semantic content , the contrastive loss might rely on non-semantic features to distinguish between them , thus emphasizing these non-semantic features in the representation ( Figure 1 ) . These same-class negatives may thus be harmful to learning representations for downstream linear classification . If this is the case , we would expect that removing same class negatives would improve performance , perhaps even more than removing the hardest 0.1 % of negatives overall . As shown in Table 1 , removing same-class negatives indeed leads to slightly higher accuracy than removing the hardest 0.1 % of negatives . Removing only the subset of the hardest 0.1 % of negatives with the same class as the query accounts for all of the improvement from removing the hardest 0.1 % of negatives . Alternatively , removing only the subset of the 0.1 % hardest negatives with different classes shows no improvement over baseline and in fact decreases top-1 accuracy at low temperature . These results demonstrate that the accuracy benefit of removing the 0.1 % hardest negatives can entirely be accounted for by the fact that it removes many elements of the same class as the query , so that removing the 0.1 % hardest negatives approximates removing the same-class negatives without requiring access to privileged label data . This observation is also consistent with recent work which has attempted to “ debias ” contrastive learning away from same-class negatives ( Chuang et al. , 2020 ) .

Summary: This paper argues that in contrastive self-supervised learning, different negative instances have different importance. This importance is relevant to the ``difficulty" of negative instances. On ImageNet and MoCo2, the authors show that using the most difficult 5% negative instances can achieve similar performance compared with using all negative instances. However, the most difficult 0.1% of negative instances yield bad performance.