Conditional Contrastive Learning with Kernel

1 INTRODUCTION . Contrastive learning algorithms ( Oord et al. , 2018 ; Chen et al. , 2020 ; He et al. , 2020 ; Khosla et al. , 2020 ) learn similar representations for positively-paired data and dissimilar representations for negatively-paired data . For instance , self-supervised visual contrastive learning ( Hjelm et al. , 2018 ) define two views of the same image ( applying different image augmentations to each view ) as a positive pair and different images as a negative pair . Supervised contrastive learning ( Khosla et al. , 2020 ) defines data with the same labels as a positive pair and data with different labels as a negative pair . We see that distinct contrastive approaches consider different positive pairs and negative pairs constructions according to their learning goals . In conditional contrastive learning , positive and negative pairs are constructed conditioned on specific variables . The conditioning variables can be downstream labels ( Khosla et al. , 2020 ) , sensitive attributes ( Tsai et al. , 2021c ) , auxiliary information ( Tsai et al. , 2021a ) , or data embedding features ( Robinson et al. , 2020 ; Wu et al. , 2020 ) . For example , in fair contrastive learning ( Tsai et al. , 2021c ) , conditioning on variables such as gender or race , is performed to remove undesirable information from the learned representations . Conditioning is achieved by constructing negative pairs from the same gender . As a second example , in weakly-supervised contrastive learning ( Tsai et al. , 2021a ) , the aim is to include extra information in the learned representations . This extra information could be , for example , some freely available attributes for images collected from social media . The conditioning is performed by constructing positive pairs with similar annotative attributes . The cornerstone of conditional contrastive learning is the conditional sampling procedure : efficiently constructing positive or negative pairs while properly enforcing conditioning . The conditional sampling procedure requires access to sufficient data for each state of the conditioning variables . For example , if we are conditioning on the “ age ” attribute to reduce the age bias , then the †Equal contribution . Code available at : https : //github.com/Crazy-Jack/CCLK-release . conditional sampling procedure will work best if we can create a sufficient number of data pairs for each age group . However , in many real-world situations , some values of the conditioning variable may not have enough data points or even no data points at all . The sampling problem exacerbates when the conditioning variable is continuous . In this paper , we introduce Conditional Contrastive Learning with Kernel ( CCL-K ) , to help mitigate the problem of insufficient data , by providing an alternative formulation using similarity kernels ( see Figure 1 ) . Given a specific value of the conditioning variable , instead of sampling data that are exactly associated with this specific value , we can also sample data that have similar values of the conditioning variable . We leverage the Kernel Conditional Embedding Operator ( Song et al. , 2013 ) for the sampling process , which considers kernels ( Schölkopf et al. , 2002 ) to measure the similarities between the values of the conditioning variable . This new formulation with a weighting scheme based on similarity kernel allows us to use all training data when conditionally creating positive and negative pairs . It also enables contrastive learning to use continuous conditioning variables . To study the generalization of CCL-K , we conduct experiments on three tasks . The first task is weakly supervised contrastive learning , which incorporates auxiliary information by conditioning on the annotative attribute to improve the downstream task performance . For the second task , fair contrastive learning , we condition on the sensitive attribute to remove its information in the representations . The last task is hard negative contrastive learning , which samples negative pairs to learn dissimilar representations where the negative pairs have similar outcomes from the conditioning variable . We compare CCL-K with the baselines tailored for each of the three tasks , and CCL-K outperforms all baselines on downstream evaluations . 2 CONDITIONAL SAMPLING IN CONTRASTIVE LEARNING . In Section 2.1 , we first present the technical preliminaries of contrastive learning . Next , we introduce the conditional sampling procedure in Section 2.2 , showing that it instantiates recent conditional contrastive frameworks . Last , in Section 2.3 , we discuss the limitation of insufficient data in the current framework , presenting to convert existing objectives into alternative forms with kernels to alleviate the limitation . In the paper , we use uppercase letters ( e.g. , X ) for random variables , P·· for the distribution ( e.g. , PX denotes the distribution of X ) , lowercase letters ( e.g. , x ) for the outcome from a variable , and the calligraphy letter ( e.g. , X ) for the sample space of a variable . 2.1 TECHNICAL PRELIMINARIES - UNCONDITIONAL CONTRASTIVE LEARNING . Contrastive methods learn similar representations for positive pairs and dissimilar representations for negative pairs ( Chen et al. , 2020 ; He et al. , 2020 ; Hjelm et al. , 2018 ) . In prior literature ( Oord et al. , 2018 ; Tsai et al. , 2021b ; Bachman et al. , 2019 ; Hjelm et al. , 2018 ) , the construction of the positive and negative pairs can be understood as sampling from the joint distribution PXY and product of marginal PXPY . To see this , we begin by reviewing one popular contrastive approach , the InfoNCE objective ( Oord et al. , 2018 ) : InfoNCE : = sup f E ( x , ypos ) ∼PXY , { yneg , i } ni=1∼PY ⊗n [ log ef ( x , ypos ) ef ( x , ypos ) + ∑n i=1 e f ( x , yneg , i ) ] , ( 1 ) where X and Y represent the data and x is the anchor data . ( x , ypos ) are positively-paired and are sampled from PXY ( x and y are different views to each other ; e.g. , augmented variants of the same image ) , and { ( x , yneg , i ) } ni=1 are negatively-paired and are sampled from PXPY ( e.g. , x and y are two random images ) . f ( · , · ) defines a mapping X × Y → R , which is parameterized via neural nets ( Chen et al. , 2020 ) as : f ( x , y ) : = cosine similarity ( gθX ( x ) , gθY ( y ) ) /τ , ( 2 ) where gθX ( x ) and gθY ( y ) are embedded features , gθX and gθY are neural networks ( gθX can be the same as gθY ) parameterized by parameters θX and θY , and τ is a hyper-parameter that rescales the score from the cosine similarity . The InfoNCE objective aims to maximize the similarity score between a data pair sampled from the joint distribution ( i.e. , ( x , ypos ) ∼ PXY ) and minimize the similarity score between a data pair sampled from the product of marginal distribution ( i.e. , ( x , yneg ) ∼ PXPY ) ( Tschannen et al. , 2019 ) . 2.2 CONDITIONAL CONTRASTIVE LEARNING . Recent literature ( Robinson et al. , 2020 ; Tsai et al. , 2021a ; c ) has modified the InfoNCE objective to achieve different learning goals by sampling positive or negative pairs under conditioning variable Z ( and its outcome z ) . These different conditional contrastive learning frameworks have one common technical challenge : the conditional sampling procedure . The conditional sampling procedure samples the positive or negative data pairs from the product of conditional distribution : ( x , y ) ∼ PX|Z=zPY |Z=z , where x and y are sampled given the same outcome from the conditioning variable ( e.g. , two random images with blue sky background , when selecting z as blue sky background ) . We summarize how different frameworks use the conditional sampling procedure in Table 1 . Weakly Supervised Contrastive Learning . Tsai et al . ( 2021a ) consider the auxiliary information from data ( e.g. , annotation attributes of images ) as a weak supervision signal and propose a contrastive objective to incorporate the weak supervision in the representations . This work is motivated by the argument that the auxiliary information implies semantic similarities . With this motivation , the weakly supervised contrastive learning framework learns similar representations for data with the same auxiliary information and dissimilar representations for data with different auxiliary information . Embracing this idea , the original InfoNCE objective can be modified into the weakly supervised InfoNCE ( abbreviated as WeaklySup-InfoNCE ) objective : WeaklySupInfoNCE : = sup f E z∼PZ , ( x , ypos ) ∼PX|Z=zPY |Z=z , { yneg , i } ni=1∼P ⊗n Y [ log e f ( x , ypos ) ef ( x , ypos ) + ∑n i=1 e f ( x , yneg , i ) ] . ( 3 ) Here Z is the conditioning variable representing the auxiliary information from data , and z is the outcome of auxiliary information that we sample from PZ . ( x , ypos ) are positive pairs sampled from PX|Z=zPY |Z=z . In this design , the positive pairs always have the same outcome from the conditioning variable . { ( x , yneg , i ) } ni=1 are negative pairs that are sampled from PXPY . Fair Contrastive Learning . Another recent work ( Tsai et al. , 2021c ) presented to remove undesirable sensitive information ( such as gender ) in the representation , by sampling negative pairs conditioning on sensitive attributes . The paper argues that fixing the outcome of the sensitive variable prevents the model from using the sensitive information to distinguish positive pairs from negative pairs ( since all positive and negative samples share the same outcome ) , and the model will ignore the effect of the sensitive attribute during contrastive learning . Embracing this idea , the original InfoNCE objective can be modified into the Fair-InfoNCE objective : FairInfoNCE : = sup f E z∼PZ , ( x , ypos ) ∼PXY |Z=z , { yneg , i } ni=1∼P ⊗n Y |Z=z [ log e f ( x , ypos ) ef ( x , ypos ) + ∑n i=1 e f ( x , yneg , i ) ] . ( 4 ) Here Z is the conditioning variable representing the sensitive information ( e.g. , gender ) , z is the outcome of the sensitive information ( e.g. , female ) , and the anchor data x is associated with z ( e.g. , a data point that has the gender attribute being female ) . ( x , ypos ) are positively-paired that are sampled from PXY |Z=z , and x and ypos are constructed to have the same z . { ( x , yneg , i ) } ni=1 are negativelypaired that are sampled from PX|Z=zPY |Z=z . In this design , the positively-paired samples and the negatively-paired samples are always having the same outcome from the conditioning variable . Hard-negative Contrastive Learning . Robinson et al . ( 2020 ) and Kalantidis et al . ( 2020 ) argue that contrastive learning can benefit from hard negative samples ( i.e. , samples y that are difficult to distinguish from an anchor x ) . Rather than considering two arbitrary data as negatively-paired , these methods construct a negative data pair from two random data that are not too far from each other1 . Embracing this idea , the original InfoNCE objective is modified into the Hard Negative InfoNCE ( abbreviated as HardNeg-InfoNCE ) objective : HardNegInfoNCE : = sup f E ( x , ypos ) ∼PXY , z∼PZ|X=x , { yneg , i } ni=1∼P ⊗n Y |Z=z [ log e f ( x , ypos ) ef ( x , ypos ) + ∑n i=1 e f ( x , yneg , i ) ] . ( 5 ) Here Z is the conditioning variable representing the embedding feature of X , in particular z = gθX ( x ) ( see definition in equation 2 , we refer gθX ( x ) as the embedding feature of x and gθY ( y ) as the embedding feature of y ) . ( x , ypos ) are positively-paired that are sampled from PXY . To construct negative pairs { ( x , yneg , i ) } ni=1 , we sample { ( yneg , i ) } ni=1 from PY |Z=z=gθX ( x ) . We realize the sampling from PY |Z=z=gθX ( x ) as sampling data points from Y whose embedding features are close to z = gθX ( x ) : sampling y such that gθY ( y ) is close to gθX ( x ) .

Summary: The paper looks into using more conditional data than standard methods by leveraging conditional mean embeddings and this weighting more datapoint when performing contrastive learning. The methodology hence allows to more comparisons (but weighted) and hence allowing the proposed method to achieve higher representational power. The idea of CME allows them to compare the conditional data (ie usually a smaller set) with the batch and then still perform contrastive learning now will the batch. They show improved performance on a variety of experiments such as fairness, representation learning and weakly supervised learning.