Are all negatives created equal in contrastive instance discrimination?

1 INTRODUCTION . In recent years , there has been tremendous progress on self-supervised learning ( SSL ) , a paradigm in which representations are learned using a pre-training task that uses only unlabeled data . These representations are then used on downstream tasks , such as classification or object detection . Since SSL pre-training does not require labels , it can leverage unlabeled data , which is can be abundant and cheaper to obtain than labeled data . In computer vision , representations learned from unlabeled data have historically underperformed representations learned directly from labeled data . Recently , however , newly proposed SSL methods such as MoCo ( He et al. , 2019 ; Chen et al. , 2020c ) , SimCLR ( Chen et al. , 2020a ; b ) , SwAV ( Caron et al. , 2020 ) , and BYOL ( Grill et al. , 2020 ) have dramatically reduced this performance gap . The MoCo and SimCLR pre-training tasks use contrastive instance discrimination ( CID ) , in which a network is trained to recognize different augmented views of the same image ( sometimes called the query and the positive and discriminate between the query and the augmented views of other random images from the dataset ( called negatives ) .1 ) Despite the empirical successes of CID , the mechanisms underlying its strong performance remain unclear . Recent theoretical and empirical works have investigated the role of mutual information between augmentations ( Tian et al. , 2020 ) , analyzed properties of the learned representations such as alignment and uniformity ( Wang & Isola , 2020 ) , and proposed a theoretical framework ( Arora et al. , 2019 ) , among others . However , existing works on CID have not investigated the relative importance or semantic properties of different negatives , even though negatives play a central role in CID . In other areas , works on hard negative mining in metric learning ( Kaya & Bilge , 2019 ) and on the impact of different training examples in supervised learning ( Birodkar et al. , 2019 ) suggest that understanding the relative importance of different training data can be fruitful . 1In MoCo , these are called the query and positive and are treated slightly differently ; in SimCLR , both are treated the same and are called positives . The other SSL methods listed ( not SimCLR and MoCo ) are not CID . In this work , we empirically investigate how the difficulty of negatives affects the downstream performance of the learned representation . We measure difficulty using the dot product between the normalized contrastive-space embeddings of the query and the negative . A dot product closer to 1 suggests a negative that is more difficult to distinguish from the query . We ask how different negatives , by difficulty , affect training . Are some negatives more important than others for downstream accuracy ? If so , we ask : Which ones ? To what extent ? And what makes them different ? We focus on MoCo v2 ( Chen et al. , 2020c ) and the downstream task of linear classification on ImageNet ( Deng et al. , 2009 ) , and have similar results for SimCLR in Appendix A.2 . We make the following contributions ( see Figure 1 for summary ) : • The easiest 95 % of negatives are unnecessary and insufficient , while the top 5 % hardest negatives are necessary and sufficient : We reached within 0.7 percentage points of full accuracy by training on the 5 % of hardest negatives for each query , suggesting that the 95 % easiest negatives are unnecessary . In contrast , the easiest negatives are insufficient ( and , therefore , the hardest negatives are necessary ) : accuracy drops substantially when training on only the easiest 95 % of negatives . The hardest 5 % of negatives are especially important : training on only the next hardest 5 % lowers accuracy by 15 percentage points . • The hardest 0.1 % of negatives are unnecessary and sometimes detrimental : Downstream accuracy was unchanged or improved when we removed these hardest negatives . These negatives were more often in the same ImageNet class as the query , compared to easier negatives , suggesting that semantically identical ( but superficially dissimilar ) negatives were unhelpful or detrimental . • Properties of negatives : Based on our observations that the importance of a negative varies with its difficulty , we investigate the properties of negatives that affect their difficulty . – We found that hard negatives were more semantically similar to the query than were easy negatives : the hardest 5 % of negatives were more likely to be of the same ImageNet class as the query , compared to easier negatives . These hard negatives were also closer to the query as measured by depth of the least common ancestor for the negative and the query in the WordNet tree ( which ImageNet is built upon ) . – We also observed that the pattern is reversed for the ≈50 % of easier negatives : there , the easier the negative , the more semantically similar it is to the query . – There exist negatives that are more consistently hard across queries than would be expected by random chance . We emphasize that our primary aim is to better understand the differences between negatives and the impact of these differences on existing methods rather than to propose a new method . However , our results suggest that there may be unexploited opportunities to reduce the cost of modern CID methods ( Chen et al. , 2020c ) . For any particular query , only a small fraction of the negatives are necessary . Our observations on differences in negative importance can serve as a valuable building block for the understanding and improvement of contrastive learning methods.For example , there may be further room to choose specific examples for training , similar to hard negative mining and curriculum learning ( Chen et al. , 2020a ; Chuang et al. , 2020 ; Kaya & Bilge , 2019 ) , to reduce costs and improve performance per data sample . 2 METHODS AND PRELIMINARIES . Contrastive instance discrimination and momentum contrast . Momentum Contrast ( MoCo v2 ) is a CID method that reaches accuracy within 6 percentage points of supervised accuracy on ImageNet with ResNet-50 ( Chen et al. , 2020c ) . In MoCo , the task is to learn a representation that succeeds at the following : given a query ( an augmented view of an image ) , correctly pick a positive ( a different augmented view of the same image ) from a large set of negatives ( augmented views of randomly chosen images ) . Our experiments focus on aspects that are common between CID methods rather than those specific to MoCo . We discuss implementation details that may be specific to MoCo v2 here . The MoCo v2 encoder is a ResNet-50 network . For pre-training , the outputs of this base network are fed into a multi-layer perceptron ( MLP ) head ; we refer to the normalized output from the MLP head as the contrastive-space embedding . For downstream tasks , the MLP head is discarded and only the base network is used ; we refer to the output of the base network as the learned representation . A distinguishing feature of MoCo is that it has two encoders , one of which is actively trained ( used for the query ) and the other which is a moving average of the trained encoder ( used for the positive and negatives ) . MoCo stores the embeddings of each batch of positives in a large queue and uses them as negatives for future batches , enabling the use of more negatives than can fit in a batch . MoCo uses the InfoNCE loss ( Gutmann & Hyvärinen , 2010 ; van den Oord et al. , 2018 ) : Lq = − log exp ( q · k+/τ ) ∑K i=1 exp ( q · ki/τ ) where q is the embedding of a query ( using the learned encoder ) , k+ is the embedding of a positive ( using the momentum encoder ) , and ki are the embeddings of the negatives in the queue ( added using previous states of the momentum encoder ) . τ is a temperature hyperparameter . Difficulty of negatives . To compute the difficulty for a set of negatives given a particular query , we calculate the dot product between the normalized contrastive-space embedding of each negative with the normalized contrastive-space embedding of the query . We then sort the dot products and consider the negatives with dot products closer to 1 to be harder negatives and those with smaller dot products to be easier negatives . We use this terminology because it fits intuition : all else being equal , harder negatives increase the loss . Since embeddings are normalized , the dot product is the cosine of the angle between the embeddings of the instances and ranges from -1 to 1 . Note that difficulty is defined per query and that it is a function of the current state of the network . Thus , a negative can be easy for some queries and hard for others , and the hardness of a negative for a given query can vary over training epochs and across different training runs and configurations . Experimental setting . Our experiments focus on MoCo v2 ( Chen et al. , 2020c ) , an improved version of MoCo which combines MoCo v1 ( He et al. , 2019 ) with several features of SimCLR ( Chen et al. , 2020a ) . We use ImageNet for pre-training and evaluate performance using linear classification on ImageNet from the representation learned in the pre-training CID task . The network used , as in MoCo v2 , is a ResNet-50 with MLP head , and trained for 200 epochs . Unless otherwise noted , we run each experiment three times with different seeds ; error bars represent mean± standard deviation . 3 WHICH NEGATIVES ARE NECESSARY OR SUFFICIENT ? . We examine which negatives , by difficulty , are necessary or sufficient for learning representations that have strong downstream performance . Outside of CID , there are varying perspectives on the value of easy negatives . Research on hard negative mining suggests that harder negatives can be more important than easier negatives for relevant tasks ( Kaya & Bilge , 2019 ) . However , in some supervised contexts , much or all training data seems important for reaching the highest accuracy ( Birodkar et al. , 2019 ) . We aim to experimentally assess which of these perspectives applies when using MoCo v2 on ImageNet classification . To determine whether a set of negatives was necessary , we removed the corresponding negatives on each pre-training step ; if the resulting representations still led to accuracy close to baseline on the downstream task , then we considered those negatives to have been unnecessary . To determine whether a set of negatives was sufficient , we removed all negatives except those in that range on each pre-training step ; if the resulting representations still led to strong accuracy on the downstream task , then we considered the negatives in that range to have been sufficient.2 The easy negatives are unnecessary ; the hard negatives are sufficient . First , we asked whether the easy negatives were necessary ( or equivalently , whether the hard negatives were sufficient ) . That is , does the network maintain downstream accuracy when it is pre-trained without the easy negatives ? To test this , we evaluated how accuracy changed as different subsets of negatives were removed . Interestingly , we found that using only the hardest 5 % of negatives was largely sufficient to recover baseline accuracy ( Figure 2a-b , 95-100 % ) , suggesting that the overwhelming majority of the easier negatives were unnecessary . Moreover , the hardest 5 % ( 95-100 % ) were substantially more informative than the next 5 % ( 90-95 % ) : top-1 accuracy dropped by only 0.7 percentage points when trained on only the hardest 5 % vs. 15 percentage points for the next hardest 5 % ( 90-95 % ) and 47 percentage points for the third 5 % ( 85-90 % ; Figure 2a-b ) . Going forward , we use 5 % as a cutoff and call the negatives harder than this cutoff hard and those easier than this cutoff easy . The easy negatives are largely insufficient ; the hard negatives are necessary . We next asked whether the easy negatives were sufficient ( or , equivalently , whether the hard negatives were necessary ) . Although we found in the previous section that the easy negatives were unnecessary , that does not necessarily mean they are insufficient . For example , it could be that the easy negatives , while individually less important , collectively provide sufficient signal for learning good representations on the downstream task . Alternatively , it is possible that the information contained in all of the easy negatives still lacks enough signal ; in this case , the easy negatives , even when combined together , would still be insufficient . We found that even when the easiest 95 % of negatives were combined together , accuracy was 5.4 % below baseline ( Figure 2c-d ) . In contrast , recall that using only the hardest 5 % of negatives ( 19x fewer ) achieved top-1 performance within 0.7 % of baseline ( Figure 2a ) . Using the easiest 90 % of negatives harms accuracy even further ( 0-90 % ; Figure 2c-d ) . Together , these results demonstrate that the easiest negatives , even when they comprise the vast majority of negatives , are still insufficient . The very hardest negatives are harmful at lower temperatures . We have found that the hard negatives , i.e . the 5 % hardest , are largely necessary and sufficient for CID . However , top-1 accuracy actually improved slightly when we removed the very hardest 0.1 % of negatives ( p = 0.03 for an 2We removed sets of negatives by treating them as through they were not present in the queue . unpaired t-test ) .3 This effect was most pronounced at lower temperatures ( Figure 3 ) ; for example , at temperature 0.1 ( as opposed to baseline temperature 0.2 ) , training without the hardest 0.1 % of negatives improved downstream top-1 accuracy by 0.23 % ( p = 0.0003 ) and top-5 accuracy by 0.67 % . One might expect such a difference between temperatures because the hardest negatives are weighted more in the loss at lower temperatures . Interestingly , the effect was larger for top-5 accuracy than top-1 accuracy ( compare Figure 3b with 3a ) . One hypothesis for why the hardest negatives hurt is that because negatives are randomly sampled , some negatives can be too similar to the query ( e.g . augmentations of near-duplicates to the query ) . Because negatives are randomly sampled , they can included augmented views of images that are near-duplicates of the query or otherwise visually very similar to the query . If both the query and negative contain identical semantic content , the contrastive loss might rely on non-semantic features to distinguish between them , thus emphasizing these non-semantic features in the representation ( Figure 1 ) . These same-class negatives may thus be harmful to learning representations for downstream linear classification . If this is the case , we would expect that removing same class negatives would improve performance , perhaps even more than removing the hardest 0.1 % of negatives overall . As shown in Table 1 , removing same-class negatives indeed leads to slightly higher accuracy than removing the hardest 0.1 % of negatives . Removing only the subset of the hardest 0.1 % of negatives with the same class as the query accounts for all of the improvement from removing the hardest 0.1 % of negatives . Alternatively , removing only the subset of the 0.1 % hardest negatives with different classes shows no improvement over baseline and in fact decreases top-1 accuracy at low temperature . These results demonstrate that the accuracy benefit of removing the 0.1 % hardest negatives can entirely be accounted for by the fact that it removes many elements of the same class as the query , so that removing the 0.1 % hardest negatives approximates removing the same-class negatives without requiring access to privileged label data . This observation is also consistent with recent work which has attempted to “ debias ” contrastive learning away from same-class negatives ( Chuang et al. , 2020 ) .

Summary: In this paper, the authors carried out a series of experiments to analyze the impact of negative samples in contrastive learning (instance discrimination - CID). In particular, they try to identify which difficulty range is important for representation learning. Of the many recent self-supervised learning approaches, they chose MOCO V2 as the testbed. They trained the MOCO model from an ImageNet pre-trained one. Various settings, which correspond to various ways of filtering our hard or easy negatives, were used. Hardness of samples are measured based on embedding distance to the query. I.e. ones with large distance are easy. Their main findings are, for negative samples, 1) Using the 5% hardest is enough for downstream tasks, 2) the easiest 95% of them were unnecessary and insufficient, 3) The hardest 0.1% is harmful and  4) hard negatives were more semantically similar to the query.