Conditional Negative Sampling for Contrastive Learning of Visual Representations

1 INTRODUCTION . Supervised learning has given rise to human-level performance in several visual tasks ( Russakovsky et al. , 2015 ; He et al. , 2017 ) , relying heavily on large image datasets paired with semantic annotations . These annotations vary in difficulty and cost , spanning from simple class labels to more granular descriptions like bounding boxes and key-points . As it is impractical to scale high quality annotations , this reliance on supervision poses a barrier to widespread adoption . While supervised pretraining is still the dominant approach in computer vision , recent studies using unsupervised “ contrastive ” objectives , have achieved remarkable results in the last two years , closing the gap to supervised baselines ( Wu et al. , 2018 ; Oord et al. , 2018 ; Hjelm et al. , 2018 ; Zhuang et al. , 2019 ; Hénaff et al. , 2019 ; Misra & Maaten , 2020 ; He et al. , 2019 ; Chen et al. , 2020a ; b ; Grill et al. , 2020 ) . Many contrastive algorithms are estimators of mutual information ( Oord et al. , 2018 ; Hjelm et al. , 2018 ; Bachman et al. , 2019 ) , capturing the intuition that a good low-dimensional “ representation ” is one that linearizes the useful information embedded within a high-dimensional data point . In vision , these estimators maximize the similarity of encodings for two augmentations of the same image . This is trivial ( e.g . assign all image pairs maximum similarity ) unless the similarity function is normalized . This is typically done by comparing an image to “ negative examples ” , which a model must assign low similarity to . We hypothesize that how we choose these negatives greatly impacts the representation quality . With harder negatives , the encoder is encouraged to capture more granular information that may improve performance on downstream tasks . While research in contrastive learning has explored architectures , augmentations , and pretext tasks , there has been little attention given to the negative sampling procedure . Meanwhile , there is a rich body of work in deep metric learning showing semi-hard negative mining to improve the efficacy of triplet losses . Inspired by this , we hope to bring harder negative sampling to modern contrastive learning . Naively choosing difficult negatives may yield an objective that no longer bounds mutual information , removing a theoretical connection that is core to contrastive learning and has been shown to be important for downstream performance ( Tian et al. , 2020 ) . In this paper , we present a new estimator of mutual information based on the popular noise-contrastive estimator ( NCE ) that supports sampling negatives from conditional distributions . We summarize our contributions below : 1 . We prove our Conditional-NCE ( CNCE ) objective to lower bound mutual information . Further , we show that although CNCE is a looser bound than NCE , it has lower variance . This motivates its value for representation learning . 2 . We use CNCE to generalize contrastive algorithms that utilize a memory structure like IR , CMC , and MoCo to sample semi-hard negatives in just a few lines of code and minimal compute overhead . 3 . We find that the naive strategy of sampling hard negatives throughout training can be detrimental . We then show that slowly introducing harder negatives yields good performance . 4 . On four image classification benchmarks , we find improvements of 2-5 % absolute points . We also find consistent improvements ( 1 ) when transferring features to new image datasets and ( 2 ) in object detection , instance segmentation , and key-point detection . 2 BACKGROUND . We focus on exemplar-based contrastive objectives , where examples are compared to one another to learn a representation . Many of these objectives ( Hjelm et al. , 2018 ; Wu et al. , 2018 ; Bachman et al. , 2019 ; Tian et al. , 2019 ; Chen et al. , 2020a ) are equivalent to NCE ( Oord et al. , 2018 ; Poole et al. , 2019 ) , a popular lower bound on the mutual information , denoted by I , between two random variables . This connection is well-known and stated in several works ( Chen et al. , 2020a ; Tschannen et al. , 2019 ; Tian et al. , 2020 ; Wu et al. , 2020 ) . To review , recall : I ( X ; Y ) INCE ( X ; Y ) = Exi , yi⇠p ( x , y ) Ey1 : k⇠p ( y ) '' log ef✓ ( xi , yi ) 1 k+1 P j2 { i,1 : k } e f✓ ( xi , yj ) # ( 1 ) where x , y are realizations of two random variables , X and Y , and f✓ : X ⇥ Y ! R is a similarity function . We call y1 : k = { y1 , . . . yk } negative examples , being other realizations of Y . Suppose the two random variables in Eq . 1 are both transformations of a common random variable X . Let T be a family of transformations where each member t is a composition of cropping , color jittering , gaussian blurring , among others ( Wu et al. , 2018 ; Bachman et al. , 2019 ; Chen et al. , 2020a ) . We call a transformed input t ( x ) a “ view ” of x . Let p ( t ) denote a distribution over T , a common choice being uniform . Next , introduce an encoder g✓ : X ! Sn 1 that maps an example to a L2-normalized representation . Suppose we have a dataset D = { xi } ni=1 of n values for X sampled from a distribution p ( x ) . Then , the contrastive objective for the i-th example is : L ( xi ) = Et , t0 , t1 : k⇠p ( t ) Ex1 : k⇠p ( x ) '' log eg✓ ( t ( xi ) ) T g✓ ( t 0 ( xi ) ) /⌧ 1 k+1 P j2 { i,1 : k } e g✓ ( t ( xi ) ) T g✓ ( tj ( xj ) ) /⌧ # ( 2 ) where ⌧ is a temperature . The equivalence of Eq . 2 to NCE is immediate given f✓ ( x , y ) = g✓ ( x ) T g✓ ( y ) /⌧ . Maximizing Eq . 2 chooses an embedding that pulls two views of the same example together while pushing two views of distinct examples apart . A drawback to this framework is that the number of negatives k must be large to faithfully approximate the true partition . In practice , k is limited by memory . Recent innovations have focused on tackling this challenge : Instance Discrimination ( Wu et al. , 2018 ) , or IR , introduces a memory bank of n entries to cache embeddings of each example throughout training . Since every epoch we observe each example once , the memory bank will save the embedding of the view of the i-th example observed last epoch in its ith entry . Representations stored in the memory bank are removed from the automatic differentiation tape , but in return , we can choose a large k by querying M . A follow up work , Contrastive Multiview Coding ( Tian et al. , 2019 ) , or CMC , decomposes an image into two color modalities . Then , CMC sums two IR losses where the memory banks for each modality are swapped . Momentum Contrast ( He et al. , 2019 ) , or MoCo , observed that the representations stored in the memory bank grow stale , since possibly thousands of optimization steps pass before updating an entry again . So , MoCo makes two important changes . First , it replaces the memory bank with a first-in first-out ( FIFO ) queue of size k. During each minibatch , representations are cached into the queue while the most stale ones are removed . Second , MoCo introduces a second ( momentum ) encoder g0✓0 as a copy of g✓ . The primary encoder g✓ is used to embed one view of xi whereas the momentum encoder is used to embed the other . Again , gradients are not propagated to g0✓0 . In this work , we focus on contrastive algorithms that utilize a memory structure that we repurpose in Sec . 4 to efficiently sample hard negatives from . In Sec . 7 , we briefly discuss generalizations to contrastive algorithms that do not use a memory structure . 3 CONDITIONAL NOISE CONTRASTIVE ESTIMATION . In NCE , the negative examples are sampled i.i.d . from the marginal distribution , p ( y ) . Indeed , the existing proof that NCE lower bounds mutual information ( Poole et al. , 2019 ) assumes this to be true . However , choosing negatives in this manner may not be the best choice for learning a good representation . For instance , prior work in metric learning has shown the effectiveness of semihard negative mining in optimizing triplet losses ( Wu et al. , 2017 ; Yuan et al. , 2017 ; Schroff et al. , 2015 ) . We similarly wish to exploit choosing semi-hard negatives in NCE conditional on the current example but to do so in a manner that preserves the lower bound on mutual information . In presenting the theory , we assume two random variables X and Y , deriving a general bound ; we will return to the contrastive learning setting in Sec . 4 . To begin , in Eq . 1 , suppose we sample negatives from a distribution q ( y|x ) conditional on a value x ⇠ p ( x ) rather than the marginal p ( y ) , which is independent of X . Ideally , we would like to freely choose q ( y|x ) to be any distribution but not all choices preserve a bound on mutual information1 . This does not , however , imply that we can only sample negatives from p ( y ) ( Poole et al. , 2019 ; Oord et al. , 2018 ) . One of our contributions is to formally define a family of conditional distributions Q such that for any q ( y|x ) 2 Q , drawing negative examples from q defines an estimator that lower bounds I ( X ; Y ) . We call this new bound the Conditional Noise Contrastive Estimator , or CNCE . We first prove CNCE to be a bound : Theorem 3.1 . ( The Conditional NCE bound ) Define d-dimensional random variables X and Y by a joint distribution p ( x , y ) and let Y1 , ... , Yk be i.i.d . copies of Y with the marginal distribution p ( y ) . Fix any function f : ( X , Y ) ! R , any realization x of X , and let c = Ey⇠p ( y ) [ ef ( x , y ) ] , the expected exponentiated similarity . Pick a set B ⇢ R strictly lower-bounded by c. Assume the pulled back set SB = { y|ef ( x , y ) 2 B } has non-zero probability ( i.e . p ( SB ) > 0 ) . For A1 , . . . , Ak in the Borel -algebra over Rd , define A = A1 ⇥ . . .⇥Ak and let q ( ( Y1 , . . . , Yk ) 2 A|X = x ) = kY j=1 p ( Aj |SB ) . Let ICNCE ( X ; Y ) = Ex , y⇠p ( x , y ) Ey1 , ... , yk⇠q ( y1 , ... , yk|x ) log e f ( x , y ) 1 k Pk j=1 e f ( x , yj ) . Then ICNCE INCE . Proof . To show ICNCE INCE , we show Ep [ log Pk j=1 e f ( x , yj ) ] < Eq [ log Pk j=1 e f ( x , yj ) ] . To see this , apply Jensen ’ s to the left-hand side of logEp [ Pk j=1 e f ( x , yj ) ] < log Pk j=1 e f ( x , yj ) , which holds if yj 2 SB for j = 1 , . . . , k , and then take the expectation Eq of both sides . The last inequality holds by monoticity of log , linearity of expectation , and the fact that Ep [ ef ( x , yj ) ] ef ( x , yj ) . Theorem Intuition . For intuition , although using arbitrary negative distributions in NCE does not bound mutual information , we have found a restricted class of distributions Q where every member q ( y|x ) “ subsets the support ” of the distribution p ( y ) . That is , given some fixed value x , we have defined q ( y|x ) to constrain the support of p ( y ) to a set SB whose members are “ close ” to x as measured by the similarity function f . For every element y 2 SB , the distribution q ( y|x ) wants to assign to it the same probability as p ( y ) . However , as q ( y|x ) is not defined outside of SB , we must renormalize it to sum to one ( hence p ( Aj |SB ) = p ( Aj\SB ) p ( SB ) ) . Intuitively , q ( y|x ) can not change p ( y ) too much : it must redistribute mass proportionally . The primary distinction then , is the smaller 1We provide a counterexample in Sec . A.1 . support of q ( y|x ) , which forces samples from it to be harder for f to distinguish from x. Thm . 3.1 shows that substituting q ( y|x ) for p ( y ) in NCE still bounds mutual information . Theorem Example 3.1 . We give a concrete example for the choice B that will be used in Sec . 4 . For any realization x , suppose we define two similarity thresholds ! ` , ! u 2 R where c < ! ` < ! u . Then , choose B = [ w ` , wu ] . In this case , the set SB , which defines the support of the distribution q ( y|x ) , contains values of y that are not “ too-close ” to x but not “ too-far ” . In contrastive learning , we might pick these similarity thresholds to vary the difficulty of negative samples . Interestingly , Thm . 3.1 states that CNCE is looser than NCE , which raises the question : when is a looser bound useful ? In reply , we show that while CNCE is a more biased estimator than NCE , in return it has lower variance . Intuitively , because q ( y|x ) is the result of restricting p ( y ) to a smaller support , samples from q ( y|x ) have less opportunity to deviate , hence lower variance . Formally : Theorem 3.2 . ( Bias and Variance Tradeoff ) Pick any x , y ⇠ p ( x , y ) . Fix the distribution q ( y1 : k|x ) as stated in Theorem 3.1 . Define a new random variable Z ( y1 : k ) = log ✓ ef ( x , y ) 1 k Pk j=1 e f ( x , yj ) ◆ repre- senting the normalized similarity . By Theorem 3.1 , the expressions Ep ( y1 : k ) [ Z ] and Eq ( y1 : k|x ) [ Z ] are estimators for I ( X ; Y ) . Suppose that the set SB is chosen to ensure Varq ( y1 : k|x ) [ Z ] Varq̃ ( y1 : k|x ) [ Z ] , where q̃ ( A ) = p ( A| complement of SB ) . That is , we assume the variance of the normalized similarity when using y1 : k 2 SB is smaller than when using y1 : k /2 SB . Then Biasp ( y1 : k ) ( Z ) Biasq ( y1 : k|x ) ( Z ) and Varp ( y1 : k ) ( Z ) Varq ( y1 : k|x ) ( Z ) . The proof can be found in Sec . A.2 . Thm . 3.2 provides one answer to our question of looseness . In stochastic optimization , a lower variance objective may lead to better local optima . For representation learning , using CNCE to sample more difficult negatives may ( 1 ) encourage the representation to distinguish fine-grained features useful in transfer tasks , and ( 2 ) provide less noisy gradients .

Summary: This paper adopts semi-hard negative mining, a sampling strategy widely used for metric learning, for contrastive self-supervised learning. Specifically, the paper chooses the negative samples in the range of $[w_l, w_u]$ percentiles (close, but not too close) in terms of the normalized feature distance. As the initial representation is not informative, the paper anneals down the percentile range. This sampling strategy improves the contrastive learning methods (IR, CMC, MoCO).