Conditional Contrastive Learning with Kernel

1 INTRODUCTION . Contrastive learning algorithms ( Oord et al. , 2018 ; Chen et al. , 2020 ; He et al. , 2020 ; Khosla et al. , 2020 ) learn similar representations for positively-paired data and dissimilar representations for negatively-paired data . For instance , self-supervised visual contrastive learning ( Hjelm et al. , 2018 ) define two views of the same image ( applying different image augmentations to each view ) as a positive pair and different images as a negative pair . Supervised contrastive learning ( Khosla et al. , 2020 ) defines data with the same labels as a positive pair and data with different labels as a negative pair . We see that distinct contrastive approaches consider different positive pairs and negative pairs constructions according to their learning goals . In conditional contrastive learning , positive and negative pairs are constructed conditioned on specific variables . The conditioning variables can be downstream labels ( Khosla et al. , 2020 ) , sensitive attributes ( Tsai et al. , 2021c ) , auxiliary information ( Tsai et al. , 2021a ) , or data embedding features ( Robinson et al. , 2020 ; Wu et al. , 2020 ) . For example , in fair contrastive learning ( Tsai et al. , 2021c ) , conditioning on variables such as gender or race , is performed to remove undesirable information from the learned representations . Conditioning is achieved by constructing negative pairs from the same gender . As a second example , in weakly-supervised contrastive learning ( Tsai et al. , 2021a ) , the aim is to include extra information in the learned representations . This extra information could be , for example , some freely available attributes for images collected from social media . The conditioning is performed by constructing positive pairs with similar annotative attributes . The cornerstone of conditional contrastive learning is the conditional sampling procedure : efficiently constructing positive or negative pairs while properly enforcing conditioning . The conditional sampling procedure requires access to sufficient data for each state of the conditioning variables . For example , if we are conditioning on the “ age ” attribute to reduce the age bias , then the †Equal contribution . Code available at : https : //github.com/Crazy-Jack/CCLK-release . conditional sampling procedure will work best if we can create a sufficient number of data pairs for each age group . However , in many real-world situations , some values of the conditioning variable may not have enough data points or even no data points at all . The sampling problem exacerbates when the conditioning variable is continuous . In this paper , we introduce Conditional Contrastive Learning with Kernel ( CCL-K ) , to help mitigate the problem of insufficient data , by providing an alternative formulation using similarity kernels ( see Figure 1 ) . Given a specific value of the conditioning variable , instead of sampling data that are exactly associated with this specific value , we can also sample data that have similar values of the conditioning variable . We leverage the Kernel Conditional Embedding Operator ( Song et al. , 2013 ) for the sampling process , which considers kernels ( Schölkopf et al. , 2002 ) to measure the similarities between the values of the conditioning variable . This new formulation with a weighting scheme based on similarity kernel allows us to use all training data when conditionally creating positive and negative pairs . It also enables contrastive learning to use continuous conditioning variables . To study the generalization of CCL-K , we conduct experiments on three tasks . The first task is weakly supervised contrastive learning , which incorporates auxiliary information by conditioning on the annotative attribute to improve the downstream task performance . For the second task , fair contrastive learning , we condition on the sensitive attribute to remove its information in the representations . The last task is hard negative contrastive learning , which samples negative pairs to learn dissimilar representations where the negative pairs have similar outcomes from the conditioning variable . We compare CCL-K with the baselines tailored for each of the three tasks , and CCL-K outperforms all baselines on downstream evaluations . 2 CONDITIONAL SAMPLING IN CONTRASTIVE LEARNING . In Section 2.1 , we first present the technical preliminaries of contrastive learning . Next , we introduce the conditional sampling procedure in Section 2.2 , showing that it instantiates recent conditional contrastive frameworks . Last , in Section 2.3 , we discuss the limitation of insufficient data in the current framework , presenting to convert existing objectives into alternative forms with kernels to alleviate the limitation . In the paper , we use uppercase letters ( e.g. , X ) for random variables , P·· for the distribution ( e.g. , PX denotes the distribution of X ) , lowercase letters ( e.g. , x ) for the outcome from a variable , and the calligraphy letter ( e.g. , X ) for the sample space of a variable . 2.1 TECHNICAL PRELIMINARIES - UNCONDITIONAL CONTRASTIVE LEARNING . Contrastive methods learn similar representations for positive pairs and dissimilar representations for negative pairs ( Chen et al. , 2020 ; He et al. , 2020 ; Hjelm et al. , 2018 ) . In prior literature ( Oord et al. , 2018 ; Tsai et al. , 2021b ; Bachman et al. , 2019 ; Hjelm et al. , 2018 ) , the construction of the positive and negative pairs can be understood as sampling from the joint distribution PXY and product of marginal PXPY . To see this , we begin by reviewing one popular contrastive approach , the InfoNCE objective ( Oord et al. , 2018 ) : InfoNCE : = sup f E ( x , ypos ) ∼PXY , { yneg , i } ni=1∼PY ⊗n [ log ef ( x , ypos ) ef ( x , ypos ) + ∑n i=1 e f ( x , yneg , i ) ] , ( 1 ) where X and Y represent the data and x is the anchor data . ( x , ypos ) are positively-paired and are sampled from PXY ( x and y are different views to each other ; e.g. , augmented variants of the same image ) , and { ( x , yneg , i ) } ni=1 are negatively-paired and are sampled from PXPY ( e.g. , x and y are two random images ) . f ( · , · ) defines a mapping X × Y → R , which is parameterized via neural nets ( Chen et al. , 2020 ) as : f ( x , y ) : = cosine similarity ( gθX ( x ) , gθY ( y ) ) /τ , ( 2 ) where gθX ( x ) and gθY ( y ) are embedded features , gθX and gθY are neural networks ( gθX can be the same as gθY ) parameterized by parameters θX and θY , and τ is a hyper-parameter that rescales the score from the cosine similarity . The InfoNCE objective aims to maximize the similarity score between a data pair sampled from the joint distribution ( i.e. , ( x , ypos ) ∼ PXY ) and minimize the similarity score between a data pair sampled from the product of marginal distribution ( i.e. , ( x , yneg ) ∼ PXPY ) ( Tschannen et al. , 2019 ) . 2.2 CONDITIONAL CONTRASTIVE LEARNING . Recent literature ( Robinson et al. , 2020 ; Tsai et al. , 2021a ; c ) has modified the InfoNCE objective to achieve different learning goals by sampling positive or negative pairs under conditioning variable Z ( and its outcome z ) . These different conditional contrastive learning frameworks have one common technical challenge : the conditional sampling procedure . The conditional sampling procedure samples the positive or negative data pairs from the product of conditional distribution : ( x , y ) ∼ PX|Z=zPY |Z=z , where x and y are sampled given the same outcome from the conditioning variable ( e.g. , two random images with blue sky background , when selecting z as blue sky background ) . We summarize how different frameworks use the conditional sampling procedure in Table 1 . Weakly Supervised Contrastive Learning . Tsai et al . ( 2021a ) consider the auxiliary information from data ( e.g. , annotation attributes of images ) as a weak supervision signal and propose a contrastive objective to incorporate the weak supervision in the representations . This work is motivated by the argument that the auxiliary information implies semantic similarities . With this motivation , the weakly supervised contrastive learning framework learns similar representations for data with the same auxiliary information and dissimilar representations for data with different auxiliary information . Embracing this idea , the original InfoNCE objective can be modified into the weakly supervised InfoNCE ( abbreviated as WeaklySup-InfoNCE ) objective : WeaklySupInfoNCE : = sup f E z∼PZ , ( x , ypos ) ∼PX|Z=zPY |Z=z , { yneg , i } ni=1∼P ⊗n Y [ log e f ( x , ypos ) ef ( x , ypos ) + ∑n i=1 e f ( x , yneg , i ) ] . ( 3 ) Here Z is the conditioning variable representing the auxiliary information from data , and z is the outcome of auxiliary information that we sample from PZ . ( x , ypos ) are positive pairs sampled from PX|Z=zPY |Z=z . In this design , the positive pairs always have the same outcome from the conditioning variable . { ( x , yneg , i ) } ni=1 are negative pairs that are sampled from PXPY . Fair Contrastive Learning . Another recent work ( Tsai et al. , 2021c ) presented to remove undesirable sensitive information ( such as gender ) in the representation , by sampling negative pairs conditioning on sensitive attributes . The paper argues that fixing the outcome of the sensitive variable prevents the model from using the sensitive information to distinguish positive pairs from negative pairs ( since all positive and negative samples share the same outcome ) , and the model will ignore the effect of the sensitive attribute during contrastive learning . Embracing this idea , the original InfoNCE objective can be modified into the Fair-InfoNCE objective : FairInfoNCE : = sup f E z∼PZ , ( x , ypos ) ∼PXY |Z=z , { yneg , i } ni=1∼P ⊗n Y |Z=z [ log e f ( x , ypos ) ef ( x , ypos ) + ∑n i=1 e f ( x , yneg , i ) ] . ( 4 ) Here Z is the conditioning variable representing the sensitive information ( e.g. , gender ) , z is the outcome of the sensitive information ( e.g. , female ) , and the anchor data x is associated with z ( e.g. , a data point that has the gender attribute being female ) . ( x , ypos ) are positively-paired that are sampled from PXY |Z=z , and x and ypos are constructed to have the same z . { ( x , yneg , i ) } ni=1 are negativelypaired that are sampled from PX|Z=zPY |Z=z . In this design , the positively-paired samples and the negatively-paired samples are always having the same outcome from the conditioning variable . Hard-negative Contrastive Learning . Robinson et al . ( 2020 ) and Kalantidis et al . ( 2020 ) argue that contrastive learning can benefit from hard negative samples ( i.e. , samples y that are difficult to distinguish from an anchor x ) . Rather than considering two arbitrary data as negatively-paired , these methods construct a negative data pair from two random data that are not too far from each other1 . Embracing this idea , the original InfoNCE objective is modified into the Hard Negative InfoNCE ( abbreviated as HardNeg-InfoNCE ) objective : HardNegInfoNCE : = sup f E ( x , ypos ) ∼PXY , z∼PZ|X=x , { yneg , i } ni=1∼P ⊗n Y |Z=z [ log e f ( x , ypos ) ef ( x , ypos ) + ∑n i=1 e f ( x , yneg , i ) ] . ( 5 ) Here Z is the conditioning variable representing the embedding feature of X , in particular z = gθX ( x ) ( see definition in equation 2 , we refer gθX ( x ) as the embedding feature of x and gθY ( y ) as the embedding feature of y ) . ( x , ypos ) are positively-paired that are sampled from PXY . To construct negative pairs { ( x , yneg , i ) } ni=1 , we sample { ( yneg , i ) } ni=1 from PY |Z=z=gθX ( x ) . We realize the sampling from PY |Z=z=gθX ( x ) as sampling data points from Y whose embedding features are close to z = gθX ( x ) : sampling y such that gθY ( y ) is close to gθX ( x ) .

Summary: This paper presents Conditional Contrastive Learning with Kernel, a kernel sampling method for contrastive learning.  This sampling method leverages the Kernel Conditional Embedding Operator and measures the similarity between the conditioning variables. The authors test the performance of CCL-K in three application scenarios: weakly supervised contrastive learning, fair contrastive learning, and hard negative contrastive learning.