Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning

1 Introduction . Self-supervised representation learning is a popular class of unsupervised representation learning algorithms in the domains of vision [ Bachman et al. , 2019 , Chen et al. , 2020a , He et al. , 2020 , Caron et al. , 2020 , Grill et al. , 2020 , Chen and He , 2021 ] and language [ Mikolov et al. , 2013 , Devlin et al. , 2019 , Brown et al. , 2020 ] . Generally , it trains a feature extractor by solving a pretext task constructed on a large unlabeled dataset . The learned feature extractor yields generic feature representations for other machine learning tasks such as classification . Recent self-supervised representation learning algorithms help a linear classifier to attain classification accuracy comparable to a supervised method from scratch , especially in a few amount of labeled data regime [ Newell and Deng , 2020 , Hénaff et al. , 2020 , Chen et al. , 2020b ] . For example , SwAV [ Caron et al. , 2020 ] with ResNet-50 has a top-1 validation accuracy of 75.3 % on the ImageNet-1K classification [ Deng et al. , 2009 ] compared with 76.5 % by using the fully supervised method . InfoNCE [ van den Oord et al. , 2018 , Eq . 4 ] or its modification is a de facto standard loss function used in many state-of-the-art self-supervised methods [ Logeswaran and Lee , 2018 , Bachman et al. , 2019 , He et al. , 2020 , Chen et al. , 2020a , Hénaff et al. , 2020 , Caron et al. , 2020 ] . Intuitively , the minimization of InfoNCE can be viewed as the minimization of cross-entropy loss on K+1 instancewise classification , where K is the number of negative samples . Despite the empirical success of self-supervised learning , we still do not understand why the self-supervised learning algorithms with InfoNCE perform well for downstream tasks . Arora et al . [ 2019 ] propose the first theoretical framework for contrastive unsupervised representation learning ( CURL ) . However , there exists a gap between the theoretical analysis and empirical observation as in Figure 1 . Precisely , we expect that a large number of negative samples degrade a supervised loss on a downstream task from the analysis by Arora et al . [ 2019 ] . In practice , however , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . a large number of negative samples are commonly used in self-supervised representation learning algorithms [ He et al. , 2020 , Chen et al. , 2020a ] . Contributions . We show difficulty to explain why large negative samples empirically improve supervised accuracy on the downstream task from the CURL framework when we use learned representations as feature vectors for the supervised classification in Section 3 . To fill the gap , we propose a novel lower bound to theoretically explain this empirical observation regarding negative samples using the coupon collector ’ s problem in Section 4 . 2 InfoNCE-based Self-supervised Representations Learning . We focus on Chen et al . [ 2020a ] ’ s selfsupervised representation learning formulation , namely , SimCLR.1 Let X be an input space , e.g. , X ⊂ Rchannel×width×height for color images . We can only access a unlabeled training dataset { xi } Ni=1 , where input x ∈ X . SimCLR learns feature extractor f : X → Rh modeled by neural networks on the dataset , where h is the dimensionality of feature representation . Let z = f ( a ( x ) ) that is a feature representation of x after applying data augmentation a : X → X . Data augmentation is a pre-defined stochastic function such as a composition of the horizontal flipping and cropping . Note that the output of f is normalized by its L2 norm . Let z+ = f ( a+ ( x ) ) that is a positive feature representation created from x with different data augmentation a+ ( · ) . SimCLR minimizes the following InfoNCEbased loss with K negative samples for each pair of ( z , z+ ) : ℓInfo ( z , Z ) : = − ln exp ( z · z+/t ) ∑ zk∈Z exp ( z · zk/t ) , ( 1 ) where Z = { z+ , z−1 , . . . , z−K } that is a set of positive representation z+ and K negative representations { z−1 . . . , z−K } created from other samples , ( . · . ) is an inner product of two representations , and t ∈ R+ is a temperature parameter.2 Intuitively , this loss function approximates an instance-wise classification loss by using K random negative samples [ Wu et al. , 2018 ] . From the definition of Eq . ( 1 ) , we expect that f learns an invariant encoder with respect to data augmentation a . After minimizing Eq . ( 1 ) , f works as a feature extractor for downstream tasks such as classification . 3 Extension of Contrastive Unsupervised Representation Learning Framework . We focus on the direct relationship between the self-supervised loss and a supervised loss to understand the role of the number of negative samples K. For a similar problem , the CURL framework [ Arora et al. , 2019 ] shows that the averaged supervised loss is bounded by the contrastive unsupervised loss . Thus , we extend the analysis of CURL to the self-supervised representation learning in Section 2 and point out that we have difficulty explaining the empirical observation as 1In fact , our theoretical analysis is valid with asymmetric feature extractors such as MoCo [ He et al. , 2020 ] , where the positive and negative features do not come from feature extractor f . 2We perform the analysis with t = 1 for simplicity , but our analysis holds with any temperature . in Figure 1 from the existing analysis . Table 2 in Appendix A summarizes notations used in this paper for convenience . 3.1 CURL Formulation for SimCLR By following the CURL analysis [ Arora et al. , 2019 ] , we introduce the learning process in two steps : self-supervised representation learning step and supervised learning step ( see Section 3.1.1 and Section 3.1.2 , respectively ) . 3.1.1 Self-supervised representation learning step . The purpose of the self-supervised learning step is to learn a feature extractor f on an unlabeled dataset . During this step , we can only access input samples from X without any relationship between samples , unlike metric learning [ Kulis , 2012 ] or similar unlabeled learning [ Bao et al. , 2018 ] . We formulate the data generation process for Eq . ( 1 ) . The key idea of the CURL analysis is the existence of latent classes C that are associated with supervised classes.3 Let ρ be a probability distribution over C , and let x be an input sample drawn from a data distribution Dc conditioned on a latent class c ∈ C. We draw two data augmentations ( a , a+ ) from the distribution of data augmentation A and apply them independently to the sample . As a result , we have two augmented samples ( a ( x ) , a+ ( x ) ) and call a+ ( x ) as a positive sample of a ( x ) . Similarly , we draw K negative samples from Dc−k for each c − k ∈ { c−1 , . . . , c−K } ∼ ρK and K data augmentations from A , and then we apply them to negative samples . Definition 1 summarizes the data generation process . Note that we suppose data augmentation a ∼ A does not change the latent class of x ∈ X . Definition 1 ( Data Generation Process in Self-supervised Representation Learning Step ) . 1 . Draw latent classes : c , { c−k } Kk=1 ∼ ρK+1 ; 2 . Draw input sample : x ∼ Dc ; 3 . Draw data augmentations : ( a , a+ ) ∼ A2 ; 4 . Apply data augmentations : a ( x ) , a+ ( x ) ; 5 . Draw negative samples : { x−k } Kk=1 ∼ DKc−k ; 6 . Draw data augmentations : { a − k } Kk=1 ∼ AK ; 7 . Apply data augmentations : { a−k ( x−k ) } Kk=1 . Note that the original data generation process of CURL samples a positive sample of x from Dc independently and does not use any data augmentations . From the data generation process and InfoNCE loss ( 1 ) , we give the following formal definition of self-supervised loss . Definition 2 ( Expected Self-supervised Loss ) . LInfo ( f ) : = E c , { c−k } Kk=1∼ρK+1 E x∼Dc ( a , a+ ) ∼A2 E { x−k ∼Dc− k } Kk=1 { a−k } Kk=1∼AK ℓInfo ( z , Z ) , ( 2 ) where recall that z = f ( a ( x ) ) and Z = { f ( a+ ( x ) ) , f ( a−1 ( x−1 ) ) , . . . , f ( a−K ( x−K ) ) } . Since we can not directly minimize Eq . ( 2 ) , we minimize the empirical counterpart , L̂Info ( f ) , by sampling from the training dataset . After minimizing L̂Info ( f ) , learned f̂ works as a feature extractor in the supervised learning step . 3.1.2 Supervised learning step . At the supervised learning step , we can observe label y ∈ Y = { 1 , . . . , Y } for each x that is used in the self-supervised learning step . Let S be a supervised data distribution over X × Y , and g ◦ f̂ : X → RY be a classifier , where g : Rh → RY and f̂ is the frozen feature extractor . Given the 3Technically , we do not need c to draw x for self-supervised representation learning . However , we explicitly include it in the data generation process to understand the relationship with a supervised task in Section 3.2 . In the main analysis , we assume that the supervised classes in the downstream task are subset of C ; however , different relationship between supervised and latent classes is discussed in Appendix B. supervised data distribution and classifier , our goal is to minimize the following supervised loss : Lsup ( g ◦ f̂ ) : = E x , y∼S a∼A − ln exp ( gy ( f̂ ( a ( x ) ) ) ) ∑ j∈Y exp ( gj ( f̂ ( a ( x ) ) ) ) . ( 3 ) By following the CURL analysis , we introduce a mean classifier as a simple instance of g because its loss is an upper bound of Eq . ( 3 ) .4 The mean classifier is the linear classifier whose weight of label y is computed by averaging representations : µy = Ex∼Dy Ea∼A f ( a ( x ) ) , where Dy is a data distribution conditioned on the supervise label y . We introduce the definition of the mean classifier ’ s supervised loss as follows : Definition 3 ( Mean Classifier ’ s Supervised Loss ) . Lµsup ( f̂ ) : = E x , y∼S a∼A − ln exp ( f̂ ( a ( x ) ) · µy ) ∑ j∈Y exp ( f̂ ( a ( x ) ) · µj ) . ( 4 ) We also introduce a sub-class loss function.5 Let Ysub be a subset of Y and Ssub be a data distribution over X × Ysub , then we define the sub-class losses of classifier g and mean classifier : Definition 4 ( Supervised Sub-class Losses of Classifier g and Mean Classifier with f̂ ) . Lsub ( g ◦ f̂ , Ysub ) : = E x , y∼Ssub a∼A − ln exp ( gy ( f̂ ( a ( x ) ) ) ) ∑ j∈Ysub ( gj ( f̂ ( a ( x ) ) ) ) , ( 5 ) Lµsub ( f̂ , Ysub ) : = E x , y∼Ssub a∼A − ln exp ( f̂ ( a ( x ) ) · µy ) ∑ j∈Ysub exp ( f̂ ( a ( x ) ) · µj ) . ( 6 ) The purpose of unsupervised representation learning [ Bengio et al. , 2013 ] is to learn generic feature representation rather than improve the accuracy of the classifier on the same dataset . However , we believe that such a feature extractor tends to transfer well to another task . Indeed , Kornblith et al . [ 2019 ] empirically show a strong correlation between ImageNet ’ s accuracy and transfer accuracy .

Summary: The authors present some theoretical analysis that extend the analysis in CURL [Arora et al 2019]. The task they study is two steps: first learning an encoder with self-supervised learning (SSL) without labels and then freezing the encoder and learning linear classifiers with supervised learning (Sup) and using the labels on the same dataset. The analysis is trying to explain why having a large number of negative samples during contrastive self supervised pretraining with SimCLR  does not degrade the performance of the supervised second stage. They study the relationship between the two losses and validate their bounds on CIFAR.