Conditional Contrastive Learning with Kernel
Authors: Yao-Hung Hubert Tsai
Tianqin Li
Martin Q. Ma
Han Zhao
Kun Zhang
Louis-Philippe Morency
Ruslan Salakhutdinov
Venue: ICLR 2022 Poster (2022)
Keywords: Contrastive Learning
Conditional Sampling
Kernel methods

Abstract: Conditional contrastive learning frameworks consider the conditional sampling procedure that constructs positive or negative data pairs conditioned on specific variables. Fair contrastive learning constructs negative pairs, for example, from the same gender (conditioning on sensitive information), which in turn reduces undesirable information from the learned representations; weakly supervised contrastive learning constructs positive pairs with similar annotative attributes (conditioning on auxiliary information), which in turn are incorporated into the representations. Although conditional contrastive learning enables many applications, the conditional sampling procedure can be challenging if we cannot obtain sufficient data pairs for some values of the conditioning variable. This paper presents Conditional Contrastive Learning with Kernel (CCL-K) that converts existing conditional contrastive objectives into alternative forms that mitigate the insufficient data problem. Instead of sampling data according to the value of the conditioning variable, CCL-K uses the Kernel Conditional Embedding Operator that samples data from all available data and assigns weights to each sampled data given the kernel similarity between the values of the conditioning variable. We conduct experiments using weakly supervised, fair, and hard negatives contrastive learning, showing CCL-K outperforms state-of-the-art baselines.


Published as a conference paper at ICLR 2022
CONDITIONAL CONTRASTIVE LEARNING WITH KERNEL
Yao-Hung Hubert Tsai Tianqin Li Martin Q. Ma', Han Zhao', Kun Zhang'., Louis-Phillippe Morency', & Ruslan Salakhutdinov' 'Camngge Mellon University University of Illinois at Urbana- Chaaa--hmmppign "Mohamed bin Zayed University of Artificial Intelligence (yaohungt, tianqinn, gianlim, kunzi, morency, rsalakhu) @cs. cmu, edu (hanzhac) @ @illinois.ee.i
ABSTRACT
Conditional contrastive learning frameworks consider the conditional sampling procedure that constructs positíve or negative data pairs conditioned on specific variables. Fair contrastive learning constructs negative pairs, for example, from the same gender (conditioning sensitive information), which in turn reduces unde- sirable information from the learned representations; weakly supervised contrastive learning constructs positive pairs with similar annotative attributes (conditioning on auxiliary information), which in turn are incorporated into the representations. Although conditional contrastive learning enables many applications, the condi- tional sampling procedure can be challenging if we cannot obtain sufficient data pairs for some values of the conditioning variable. This paper presents Conditional Contrastive Learning with Kernel (CCL-K) that converts existing conditional con- trastive objectives into alternative forms that mitigate the insufficient data problem. Instead of sampling data according to the value of the conditioning variable, CCL- K uses the Kernel Conditional Embedding Operator that samples data from all available data and assigns weights to each sampled data given the kernel similarity between the values of the conditioning variable. We conduct experiments using weakly supervised, fair, and hard negatives contrastive learning, showing CCL-K outperforms state-of ootthh--rttttttt baselines.
INTRODUCTION
Contrastive learning algorithms (Oord et al., 2018; Chen et al., 2020; He al., 2020; Khosla et al., 2020) learn similar representations for poxitively-paired data and dissimilar representations for negatively-paired data. For instance, self-supervised visual contrastive learning (Hjelm al., 2018) define two views of the same image (applying different image augmentations to each view) as positive pair and different images as a negative pair. Supervised contrastive learning (Khosla et al., 2020) defines data with the same labels as a positive pair and data with different labels as negative pair. We see that distinct contrastive approaches consider different positive pairs and negative pairs constructions according to their learning goals.
In conditional contrastive learning, positive and negative pairs are constructed conditioned on specific variables. The conditioning variables can be downstream labels (Khosla et al., 2020), sensitive attributes (Tsai et al., 2021c), auxiliary information (Tsai et al., 2021a), or data embedding fea- tures (Robinson et al., 2020; Wu et al., 2020). For example, in fair contrastive learning (Tsai et al., 2021c), conditioning on variables such as gender or race, is performed to remove undesirable infor- mation from the learned representations. Conditioning is achieved by constructing negative pairs from the same gender. As second example, in reakly-supervised contrastive learning (Tsai et al., 2021a), the aim is include extra information in the learned representations. This extra information could be, for example, some freely available attributes for images collected from social media. The conditioning is performed by constructing positive pairs with similar annotative attributes. The cornerstone of conditional contrastive learning is the conditional sampling procedure: efficiently constructing positive or negative pairs while properly enforcing conditioning.
Equull contribution. Code available at: http::///gttibb.cm//rrrryaaaaaa

---

Published as a conference paper at ICLR 2022
Traditional Conditional Sampling
CCL-K (ours)
6
weight
Figure 1: Illustration of the main idea in CCL-K. best viewed in color. Suppose select color the conditioning variable and want to sample red data points. Left figure: The traditional conditional sampling procedure only samples red points (i.e., the points in the circle). Right figure: data points (i.c., the sampled set expands from the inner circle to the outer circle) with weighting scheme based on the similarities between conditioning variables' values. The higher the weight, the higher probability of of a data point being sampled. For example, CCL-K can sample orange data with high probability, because orange resembles red. In this illustration, the weight ranges from to with white 0 and black
proposed CCL-K samples all
The conditional sampling procedure requires access to sufficient data for each state of the conditioning variables. For example, if we are conditioning on the "age" attribute to reduce the age bias, then the conditional sampling procedure will work best we can create a sufficient number of data pairs for each age group. However, in many real- world situations, some values of the conditioning variable may not have enough data points or even no data points at all. The sampling problem exacerbates when the conditioning variable is continuous.
In this paper, we introduce Conditional Contrastive Learning with Kernel (CCL-K), to help mitigate the problem of insufficient data, by providing alternative formulation using similarity kernels (see Figure 1). Given a specific value of the conditioning variable, instead of sampling data that are exactly associated with this specific value,
also sample data that have similar values of the conditioning variable. We leverage the Kernel Conditional Embedding Operator (Song et al., 2013) for the sampling process, which considers kernels (Schölkopf al., 2002) measure the similarities between the values of the conditioning variable. This new formulation with a weighting scheme based on similarity kernel allows us use all training data when conditionally creating positive and negative pairs. It also enables contrastive learning to use continuous conditioning variables.
To study the generalization CCL-K, we conduct experiments on three tasks. The first task weakly supervised contrastive learning which incorporates auxiliary information by conditioning on the annotative attribute to improve the downstream task performance. For the second task, fair contrastive learning, condition the sensitive attribute remove its information in the representations. The last task is hard negative contrastive learning, which samples negative pairs to learn dissimilar representations where the negative pairs have similar outcomes from the conditioning variable. We compare CCL-K with the baselines tailored for each of the three tasks, and CCL-K outperforms all baselines on downstream evaluations.
2 CONDITIONAL SAMPLING IN CONTRASTIVE LEARNING
In Section 2.1, first present the technical preliminaries of contrastive learning. Next, introduce the conditional sampling procedure in Section 2.2, showing that it instantiates recent conditional contrastive frameworks. Last, in Section 2.3, we discuss the limitation of insufficient data in the current framework, presenting to convert existing objectives into alternative forms with kernels to alleviate the limitation. In the paper, we use uppercase letters (e.g., X) for random variables, P. for the distribution (e.g., Px denotes the distribution of X), lowercase letters (e.g., 2)) for the outcome from a variable, and the calligraphy letter (e.g., X)) for the sample space of a variable.
2.1 TECHNICAL PRELIMINARIES UNCONDITIONAL CONTRASTIVE LEARNING
Contrastive methods learn similar representations for positive pairs and dissimilar representations for negative pairs (Chen et al., 2020; He et al., 2020; Hjelm et al., 2018). prior literature (Oord et al., 2018; Tsai et al., 202 Bachman et al., 2019; Hjelm et al., 2018), the construction of the positive and negative pairs can be understood as sampling from the joint distribution Pxy and product of marginal Px Py. To see this, we begin by reviewing popular contrastive approach, the InfoNCE objective (Oord et al., 2018):
(o.Moco))
InfoNCE == sup E (e Pxy (Vong Py on
log el (z. + EE ex
(1)
2

---

Published as a conference paper at ICLR 2022
Framework
Conditioning Variable Z
Positive Pairs from
Negative Pairs from
Weakly Supervised (Tsai et al,, 202
Auxiliary information
PrizzaPrizza PXYIZ==
Px Priz=zPriz=x Pr12=
Fair (Tsai et al., 2021c)
Sensitive information
Hard Negatives (Wu et al., 2020)
Feature embedding of X
Pxy
Table Conditional sampling procedure of weakly supervised, fair, and hard negative contrastive learning. We can regard the data pair (z, y) sampled from Pxy or P as strongly-corren correlate, such as views of the same image by applying different image augmentations; (I: y) sampled from Px|z Py iz as two random data that are both associated with the same outcome of the conditioning variahle, such as two random images with the same annotative attributes; and (ar, y) from Px Py as two uncorrelated data such as two random images.
where X and Y represent the data and x is the anchor data. (a, Upos) are positively-paiin and are sampled from Pxy (x and y are different views to each other; e.g., augmented variants of the same image), and {(x, [heg,i)]== are aggatiellypairee and are sampled from PxPy (e.g, I and y are two random images). f(:))) defines a mapping X x y R, which is parameterized via neural nets (Chen et al., 2020) as:
f(z, v) == cosine similarity (g0x (z), goy (w)
where g0x (x) and g0y (v) are embedded features, gex and g0y are neural networks (gex can be the same as g0y ) parameterized by parameters 0x and fl, and T is a hyper-parameter that rescales the score from the cosine similarity. The InfoNCE objective aims to maximize the similarity score between a data pair sampled from the joint distribution (i.e., (I, Ypos) ~ Pxy) and minimize the similarity score between a data pair sampled from the product of marginal distribution (i.e..
(2t, Vneg) ~ PxPy) (Tschannen et al., 2019).
2.2 CONDITIONAL CONTRASTIVE LEARNING
Recent literature (Robinson et al., 2020; Tsai et al., 202 has modified the InfoNCE objective to achieve different learning goals by sampling positive or negative pairs under conditioning variable Z (and its outcome z). These different conditional contrastive learning frameworks have one common technical challenge: the conditional sampling procedure. The conditional sampling procedure samples the positive or negative data pairs from the product of conditional distribution: (æ, y) ~ Pxiz=aPy|z==. where z and y are sampled given the same outcome from the conditioning variable (e.g., two random images with blue sky background, when selecting as blue sky background). We summarize how different frameworks use the conditional sampling procedure in Table 1.
Weakly Supervised Contrastive Learning. Tsai al. (2021a) consider the auxiliary information from data (e.g., annotation attributes of images) as a weak supervision signal and propose a contrastive objective incorporate the weak supervision in the representations. This work is motivated by the argument that the auxiliary information implies semantic similarities. With this motivation, the weakly supervised contrastive learning framework learns similar representations for data with the same auxiliary information and dissimilar representations for data with different auxiliary information. Embracing this idea, the original InfoNCE objective can be modified into the weakly supervised InfoNCE (abbreviated as WeaklySuppInfoNCCe
objective:
Weakly.Sapintwees = sup B. Pz. (Figp))) Pajz Prizz. ADung H
log 7ix.
Here Z is the conditioning variable representing the auxiliary information from data, and is the outcome of auxiliary information that we sample from Pz. (I, Ypos) are positive pairs sampled from Pylz==PP|z==> In this design, the positive pairs always have the same outcome from the conditioning variable. {(I, )l are negative pairs that are sampled from Px
Fair Contrastive Learning. Another recent work (Tsai et al., 2021c) presented to remove un- desirable sensitive information (such as gender) in the representation, by sampling negative pairs conditioning on sensitive attributes. The paper argues that fixing the outcome of the sensitive variable prevents the model from using the sensitive information to distinguish positive pairs from negative pairs (since all positive and negative samples share the same outcome), and the model will ignore the effect of the sensitive attribute during contrastive learning. Embracing this idea, the original InfoNCE objective can be modified into the Fair- InfoNCE objective:
Fair :- sup Sn (srapoe)) Jr Px Maax (meeg. Piz
log [xx.ppos) 4 EE ef (F.Ravaa.)
(4)
Here Z is the conditioning variable representing the sensitive information (e.g., gender), 2 is the outcome of the sensitive information (e.g., female), and the anchor data is associated with (e.g..
3

---

Published as a conference paper at ICLR 2022
data point that has the gender attribute being female). (z. Upos) are positively-paire that are sampled from Pxy|z==, and x and Ypos are constructed to have the same 2. {(x, Vueg.i) In are negatively- paired that are sampled from Pxiz== Pr In this design, the positively-pate negatively-paired samples are always having the same outcome from the conditioning variable.
samples and the
Hard-negative Contrastive Learning. Robinson et al. (2020) and Kalantidis et al. (2020) argue that contrastive learning can benefit from hard negative samples (i.e., samples y that are difficult to distinguish from an anchor x). Rather than considering two arbitrary data as regatively-paires these methods construct a negative data pair from two random data that are not too far from each other'. Embracing this idea, the original InfoNCE objective is modified into the Hard Negative InfoNCE (abbreviated MardNeg-InfoNCCF)
objective:
HardNegssoNNCe == sup E (Laxppa)~ Pax-x (Mtngg)) 1- p68m
log
(S)
(Co-ypoo) +2 (am) Srog.
Here Z the conditioning variable representing the embedding feature of X, in particular 2 g0x (1r) (see definition in equation 2, we refer 9ex (x) as the embedding feature of x and Bey (v) as the embedding feature of y). (I, Mpos) are positively-paire that are sampled from Pxy. To
construct negative pairs {(z, Mlogg,) Jim we sample { ()neg,)) J from Pr|z===00 (z) We realize the sampling from Priz=a=ga gex (x) as sampling data points from Y whose embedding features are close to z = g (z): sampling y such that g0y (y) is close to gex (2)).
2.3 CONDITIONAL CONTRASTIVE LEARNING WITH KERNEL.
The conditional sampling procedure common all these conditional contrastive frameworks has limitation when we have insufficient data points associated with some outcomes of the conditioning variable. In particular, given an anchor data and its corresponding conditioning variable's 's outcome 2, if 2 is uncommon, then it will be challenging for us to sample y that is associated with via y ~ Priz=a. 2. The insufficient data problem can be more serious when the cardinality of the conditioning variable [Z] is large, which happens when Z contains many discrete values, or when Z is continuous variable (cardinality |Z] = oo). In light of this limitation, we present to convert these objectives into alternative forms that can avoid the need to sample data from Priz and can retain the same functions as the original forms. We name this new family of formulations Conditional Contrastive Learning with Kernel (CCL.-K.
High Level Intuition. The high level idea of our method is that, instead of sampling y from Pyiz=e, we sample from existing data of Y whose associated conditioning variable's outcome is close to z. For example, assuming the conditioning variable Z be age and 2 to be 80 years old, instead of sampling the data points at the age of 80 directly, we sample from all data points, assigning highest weights to the data points at the age from 70 to 90, given their proximity to 80. Our intuition is that data with similar outcomes from the conditioning variable should be used in support of the conditional sampling. Mathematically, instead of sampling from Pyiz= we sample from
distribution proportional to the weighted sum Σø= ( P) where w(2),2) 2) represents how similar in the space of of the conditioning variable Z. This similarity is computed for all data points j = 1... .N. In this paper, we use the Kernel Conditional Embedding Operator (Song et al., 2013) for such approximation, where we represent the similarity using kernel (Schölkopf et al., 2002).
Step I Problem Setup. We want to avoid the conditional sampling procedure from existing conditional learning objectives (equation 3, 4, 5), and hence we are not supposed to have access to data pairs from the conditional distribution Px|z Priz. Instead, the only given data will be batch of
triplets {zz, Si, 24) Ki which are independently sampled from the joint distribution P PxYz with b being the batch size. In particular, when (d:;, Vi, 2;) ~ Pxyz, (Fi, yi) is a pair of data sampled from the joint distribution Pxy (e.g., two augmented views of the same image) and 2; is the associated conditioning variable's 's outcome (e.g., the annotative attribute of the image). To convert previous objectives into alternative forms that avoid the need of the conditional sampling procedure, we need to perform an estimation of the scoring function (((a)) for (a. y) ~ PxizPrie iz in equation 3, 4, given only (x, yos 2
PXYz.
Wu al. (2020) argues that better construction of negative data pairs is selecting two random data that are neither too far nor too close to each other.
PIf at is the anchor data and is its corresponding variable's outcome, then for y Py 12 the data pair (ar, 2) can he seen as sampling from Pxiz== Pr|z=
4

---

Published as a conference paper at ICLR 2022
(Kz + A
Kz
1.0
0.8
0.5
0.2
0.0
Figure 2: (K2 + K2 v.s. Kz. We apply min max normalization (x min(x)) (max(x)) mi)()))
for both matrices for better visualization. We see that (K2 + +Z) - K2 can be seen as a smoothed version of K2, which suggests that cach entry in (K2 + XI) --'KK, K2 represents the similarities between 2s.
Step II Kernel Formulation. We present to reformulate previous objectives into kernel expressions. We denote Kxy E Rxxb as kernel gram matrix between X and Y: let the ith row and j.h column of xx be the exponential scoring function e/(2.)5) (see equation 2) with [Kxy == ((
KXY is a gram matrix because the design of e satisfies kernel between gex (x)) and goy (v):
(or.)) = =xx (ousine similarity (gex (2), goy gey
(w))/=) = (dilgox (4)), Blavy (v)) /2
(6)
where (s is the inner product in a Reproducing Kernel Hilbert Space (RKHS) H and ø the corresponding feature map. Kxy can also be represented KXY 4 x 2b with x [0(yox (zi)), (gax (xx))] and Øy = [Ø( (u.)).... , (wb))] Similarly, denote Kz E Rbxb kernel gram matrix for Z, where [Kz] represents the similarity between 2; and Zy: [Kzli == (1())) where 7(:) is arbitrary kernel embedding for Z and G is its corresponding RKHS space, Kz can also be represented as Kg l'zl' with
Tz (((.)))
Step III- - Kernel-based Scoring Function ef(.)) Estimation. We present the following:
Definition 2.1 (Kemel Conditional Embedding Operator (Song et al., 2013)). By Kernel Conditional Embedding Operator (Song et al,, 2013), the finite-sample kernel estimation of Py (g0r) (v) is ( (27(), where À is hyper-parameter.
Proposition 2.2 (Estimation of (mi.)) when y PyIZ=ue) Given (Ii,, {)); P P
8b
Vo,
the finite- sample kernel estimation of (l(r)))) when y Priz =2; Kxy (Kz +XII
K2
Kxy + 1I) Kz - u(2z 2:) e(033))) with w(2), 21) = [Kz -1) K2 "r
Proof. For any 7 = 2, along with Definition 2.1, we estimate ob (gey (v) when y ~ Priz==
By-Priz-. (00r (w)) ( + AI) TT (2). Then, we plug in the result for the data pair (z,, estimate e (=),))) when y Pr 7=
(6 (3xxx (m.)): (K2 + AT) (2:)) = tr( (0x (oi) (0xx)(0x(0))
DJ(K2
+ -T) =
[Kxy]a(K2 AI) "K) = [Kxy]e- [K2+ A [xx AI) K2
[KxY (K2 AI) - K2lw i is the kernel estimation of e (((.))))) when (2), Pxz, , Priz=z, It defines the similarity between the data pair sampled from Pxiz=a; Priz= Hence, (K - can be seen as a transformation applied on KXY (KxY defines the similarity between X and Y). converting unconditional to conditional similarities between X and Y (conditioning on Z). Proposition 2.2 also re- re-writes this estimation as a weighted sum over (f 15=1 with the weights w(zj; 2,) [(Kz + AI) >I)- KZl,,
ys' We provide an illustration to compare (Kz + >I)-KKz and Kz in Figure 2, showing that (K2 + AI)- Kz can be seen as smoothed version of K suggesting the weight [KK + A1)-1 K2 jú captures the similarity between (2,, zi). To conclude, we use the Kernel Conditional Embedding Operator (Song et al., 2013) to avoid explicitly sampling y PYiZ= which alleviates the limitation of having insufficient data from that are associated with z. It is worth noting that our method neither generates raw data directly nor includes additional training.
Cosine similarity is proper kernel, and the exponential of proper kernel also proper kernel.
5

---

Published as a conference paper at ICLR 2022
In terms of computational complexity, calculating the inverse (K2 + XI) costs O(b²) where b is the batch size, or ((b2-376) using more efficient inverse algorithms like Coppersmith and Winograd (1987). We use the inverse approach with O(b²) computational cost, which will not be an issue for our method. This is because we consider a mini-batch training to constrain the size of b, and the inverse ( (Kz AT)- does not contain gradients. The computational bottlenecks are gradients computation and their updates.
Step IV- Converting Existing Contrastive Learning Objectives. We short hand Kxy +
Kz as KxLYIZZ following notation from prior work (Fukumizu et al., 2007) that short- hands Pxiz Pyiz as PxuYIe. Now, we plug in the estimation of (x(()))) using Proposition 2.2 into WeaklySup-InfoNCC (equation 3), Fair-InfoNCE (equation 4), and HardNeg-InfoNCE (equation 5) objectives, coverting them into Conditional Contrastive learning with Kernel (CCL-K) objectives:
WeaklySupocLu
@b ~P XY2
[log [Kxarizla EsedKKyle
[&xx vIZ.a.
FairecLK [1e (1 2
[1ox [KxYlu + (b- 1][Kxxxxxxxxxxxxx[x uriahe
(Kxyla
HardNegcoLK 2 ((
- on
[1og [Kxylu +i (b DJR DIKx
(kxrla
RELATED WORK
The majority of the literature on contrastive learning focuses on self-supervised learning tasks (Oord et al., 2018), which leverages unlabeled samples for pretraining representations and then uses the learned representations for downstream tasks. Its applications span various domains, including computer vision (Hjelm et al., 2018), natural language processing (Kong et al., 2019), speech process- ing (Baevski et al., 2020), and even interdisciplinary (vision and language) across domains (Radford et al., 2021). Besides the empirical success, Arora et al. (2019); Lee et al. (2020); Tsai et al. (2021d) provide theoretical guarantees, showing that contrastively learning can reduce the sample complexity on downstream tasks. The standard self-supervised contrastive frameworks consider the objective that requires only data's pairing information: it Iearns similar representations between different views of data (augmented variants of the same image (Chen et al., 2020) or an image-caption pair (Radford al., 2021)) and dissimilar representations between two random data. We refer to these frameworks as unconditional contrastive learning, in contrast to our paper's 's focus conditional contrastive learning, which considers contrastive objectives that take additional conditioning variables into account. Such conditioning variables can be sensitive information from data (Tsai et al., 2021c), auxiliary information from data (Tsai al., 2021a), downstream labels (Khosla et al., 2020), or data's embedded features (Robinson et al., 2020; Wu et al., 2020; Kalantidis et al., 2020). It is worth noting that, with additional conditioning variables, the conditional contrastive frameworks extend the self-supervised learning settings to the weakly supervised learning (Tsai et al., 2021a) or the supervised learning setting (Khosla et al., 2020).
Our paper also relates to the literature on few-shot conditional generation (Sinha et al., 2021). which aims to model the conditional generative probability (generating instances according to a conditioning variable) given only limited amount of paired data (paired between an instance and its corresponding conditioning variable). Its applications span conditional mutual information estimation (Mondal et al., 2020), noisy signals recovery (Candes et al., 2006), image manipulation (Park et al., 2020; Sinha et al., 2021), etc. These applications require generating authentic data, which is notoriously challenging (Goodfellow et al., 2014; Arjovsky et al., 2017). On the contrary, our method models the conditional generative probability via Kernel Conditional Embedding Operator (Song et al., 2013), which generates kernel embeddings but not raw data. Ton et al. (2021) relates to our work and also uses conditional mean embedding to perform estimation regarding the conditional distribution. The differences that Ton et al. (2021) tries to improve conditional density estimation while this paper aims resolve the challenge of insufficient samples of the conditional variable. Also, both Ton et al. (2021) and this work consider noise contrastive method, more specifically, Ton et al. (2021) discusses noise contrastive estimation (NCE) (Gutmann and Hyvärinen, 2010), while this work discusses InfoNCE (Oord et al., 2018 8) objective which is inspired from NCE.
Our proposed method can also connect to domain generalization (Blanchard et al., 2017), if we treat each 2; as a domain or a task indicator (Tsai et al., 2021c). In specific, Tsai et al. (2021c) considers conditional contrastive learning setup, and one task of it performs contrastive learning from data

---

Published as a conference paper at ICLR 2022
across multiple domains, by conditioning on domain indicators reduce domain-specifit information for better generalization. This paper further extends this idea from Tsai et al. (2021c), by using conditional mean embedding to address the challenge of insufficient data in certain conditioning variables (in this case, domains).
4 EXPERIMENTS
We conduct experiments on various conditional contrastive learning frameworks that are discussed in Section 2.2: Section 4.1 for the weakly supervised contrastive learning, Section 4.2 for the fair contrastive learning; and Section 4.3 for the hard-negatives contrastive learning.
Experimental Protocal. We consider the setup from the recent contrastive learning literature (Chen et al., 2020; Robinson et al., 2020; Wu et al., 2020), which contains stages of pre-training, fine-ture mie-ttning, and evaluation. In the pre-training stage, on data's 's training split, we update the parameters in the feature encoder (i.e., go. (-)s in equation 2) using the contrastive learning objectives (e.g., InfoNCE (equation 1), WeaklySuPman.ce (equation 3), or WeaklySupocI.K (equation 7)). In the fine-tuning stage, we fix the parameters of the feature encoder and add a small fine-tuning network on top of it. On the data's training split, we fine-tune this small network with the downstream labels. In the evaluation stage, we evaluate the fine-tuned representations on the data's 's test split. We adopt ResNet-50 Hc et al. (2016) or LeNet-5 LeCun et al. (1998) as the feature encoder and a linear layer as the fine-tuning network. All experiments are performed using LARS optimizer You et al. (2017). More details can be found in Appendix and our released code.
4.1 WEAKLY SUPERVISED CONTRASTIVE LEARNING
In this subsection, we perform experiments within the weakly supervised contrastive learning frame- work (Tsai et al., 2021a) which considers auxiliary information the conditioning variable Z. It aims to learn similar representations for pair of data with similar outcomes from the conditioning variable (i.e., similar auxiliary information). and vice versa.
Datasets and Metrics. We consider three visual datasets in this set of experiments. Data X
and Y represent images after applying arbitrary image augmentations. 1) UT-Zappos (Yu and Grauman, 2014): It contains 50, 025 shoe images over 21 shoe categories. Each image is annotated with 7 sinomially-distributee attributes as auxiliary information, and we convert them into 126 binary attributes (Bernouli-ddstributed) 2) CUB (Wah et al., 2011): It contains 11, 788 bird images spanning 200 fine-grain bird species, meanwhile 312 binary attributes are attached to each image. 3) ImageNet-100 (0000Russe (Russakovsky et al., 2015): It is subset of ImageNet- 1 Russakovsky et al. (2015) dataset, containing 0.12 million images spanning 100 categories. This dataset does not come with auxiliary information, and hence we extract the 512-dim. visual features from the CLIP (Radford et al., 2021) model (a large pre-trained visual model with natural language supervision) to be its auxiliary information. Note that we consider different types of auxiliary information. For UT-Zappos and CUB, we consider discrete and continuous and pre-trained language-enricher features. We report the top- I accuracy as the metric for the downstream classification task.
attributes. For ImageNet-100 we consider
Methodology. We consider the WeaklySupccIL objective (equation 7) as the main method. For WeaklySupecI.x we perform the sampling process {(I;, Vs. =_) Pxyz by first sampling an image im; along with its auxiliary information z; and then applying different image augmentations on im; to obtain (Ii; y.). We also study different types of kernels for Kz. On the other hand, we select two baseline methods. The first one is the unconditional contrastive learning method: the InfoNCE objective (Oord al., 2018: Chen al., 2020) (equation 1). The second one the conditional contrastive learning baseline: the WeaklySSupinoNNcE objective (cquation 3). The difference between WeaklySupecLk and WeaklySuppnaoooes is that the latter requires sampling pair of data with the same outcome from the conditioning variable (i.e., (x, ~ Px suggested in Section 2.3, directly performing conditional sampling is challenging if there is not enough data to support the conditioning sampling procedure. Such limitation exists in our datasets: CUB has on average 1.001 data points per Z's configuration, and ImageNet-1109 has only 1 data point per Z's configuration since its conditioning variable is continuous and each instance from the dataset has a unique Z. To avoid this limitation, in WeaklySupnnnoNces clusters the data to ensure that data within the same cluster are abundant and have similar auxiliary information, and then treating
However, as
7

---

Published as a conference paper at ICLR 2022
UT-Zappos
CUB
ImogeNee-100
Kemels
UT-Zappos
CUB
ImageNet-100
Unconditional Courrasiv Learning Mechods
RBF
86.5 0.5
32.3 : 0.5
81.8 :: 0.4
InfoNCR
77.8
14.1 0.7
76.2
Laplacian
86.8 0.3
32.1 1 0.5
80.2 4 0.3
Conditional Contrastive Loarnnng Menhods
Lincar 86.5 = 0.4
29.4 = 0.8
77.5 + 0.3
Wiskkt-Spanieenss
84.6=04
20.6
WeaklySupema (ours)
86.6
29,9 10.3 42.40:55
Cusime 86.6 0.7
29.9 1 0.3
82.4 0.5
Table 2: Object classification accuracy (%) under the weakly supervised contrastive learning setup. Left: results of the proposed method and the baselines. Right: different types of kernel choice in WeaklySuPcoux.
the clustering information as the new conditioning variable. The result of WoaallySuPinnonce is reported by selecting the optimal number of clusters via cross-validation.
Results. We show the results in Table First, WeaklySSPcoLLK shows consistent improvements over unconditional baseline InfoNCE, with absolute improvements of 8.8%. 15.8%, and 6. 6.2% on UT-Zappos, CUB and. ImageNet- 100 respectively. This because the conditional method utilizes the additional auxiliary information (Tsai et al., 2021a). Second, WeaklySupceLA performs better than Sup with absolute improvements of 2%. 9.3%, and 1%. We attribute better perfor- first
mance of WeaklySupoeLK over WeaklySupintoNce to the following fact: WeaklySupiutooe performs clustering on the auxiliary information and considers the new clusters as the conditioning variable, while WeaklySupceLk directly considers the auxiliary information the conditioning variable. The clustering in WeaklySSup may lose precision of the auxiliary information and may negatively affect the quality of the auxiliary information incorporated in the representation. Ablation study on the choice of kernels has shown the consistent performance of WeaklySupocL. across different kernels on the UT-ZZappss, CUB and ImageNet- 100 dataset, where consider the following kernel functions: RBF, Laplacian, Linear and Cosine. Most kernels have similar performances, except that linear kernel is worse than others on ImageNet-100 (by least 2.7%).
4.2 FAIR CONTRASTIVE LEARNING
In this subsection, we perform experiments within the fair contrastive learning framework (Tsai et al., 2021c), which considers sensitive information as the conditioning variable Z. It fixes the outcome of the sensitive variable for both the positively-paine and negatively-paiie samples in the contrastive learning process, which leads the representations to ignore the effect from the sensitive variable.
Datasets and Metrics. Our experiments focus on continuous sensitive information, to echo with the limitation of the conditional sampling procedure in Section 2.3. Nonetheless, existing datasets mostly consider discrete sensitive variables, such as gender or race. Therefore, we synthetically create ColorMNIST dataset, which randomly assigns a continuous RBG color value for the background in each handwritten digit image in the MNIST dataset (LeCun et al., 1998). We consider the background color as sensitive information. For statistics, ColorMNIST has 60, 000 colored digit images across 10 digit labels. Similar to Section 4. data X and Y represent images after applying arbitrary image augmentations. Our goal is two-fold: we want to see how well the learned representations 1) perform on the downstream classification task and 2) ignore the effect from the sensitive variable. For the former one, we report the top-1 I accuracy as the metric; for the latter one, we report the Mean Square Error (MSE) when trying to predict the color information. Note that the MSE score is higher the better since we would like the learned representations to contain less color information.
Methodology. We consider the FairςOIKK objective (equation 8) as the main method. For FairCCLKK, we per- form the sampling process {(x;, Vs. }))b=_ ~ Px 86 Pxyz by first sampling a digit image im, along with its sensitive information 2; and then applying different image augmen- tations on im, to obtain (x,, v.). We select the uncondi- tional contrastive learning method the InfoNCE objec- tive (equation 1) as our baseline. We also consider the PaIrMONNNC objective (equation 4) baseline, by clus- tering the continuous conditioning variable Z into one of the following: 3, 5, 10, 15, or 20 clusters using K-means.
Top-1 Accuracy (1)
MSE (f)
Unconditional Contraztive Learning Menhods
InfoNCE
84.1 + 1.8
48.8 + 4.5
Conditional Contrastive Learning Methods
FairaANCE Fairgrix (ours)
85.9 1 0.4
64.9 1. 5.1
86.4 1 0.9
64.7 1 3.9
Table 3:
Classification accuracy (%) under the fair contrastive learning setup, and the MSE (higher the better) between color in an image and color prediction by the image's 's representation. A higher MSE indicates less color information from the original image is contained in the learned representation.
Results. We show the results in Table 3. FaireCLK is consistently better than the InfoNCE where the absolute accuracy improvement is 2.3% and the relative improvement of MSE (higher the better) is 32.6% over InfoNCE. We report the result of using the Cosine kernel and provide the ablation study of different kernel choices in the Appendix. This result suggests that the proposed FairoeLK can
8

---

Published as a conference paper at ICLR 2022
achieve better downstream classification accuracy while ignoring more sensitive information (color information) compared to the unconditional baseline, suggesting that our method can achieve a better level of faimess (by excluding color bias) without sacrificing performance. Next we compare to FairinoNNCE: baseline, and we report the result using the 10-cluster partition of Z as it achieves the best top-1 accuracy. Compared
FaiMI FairCCLK is better in downstream accuracy, while slightly worse in MSE (a difference of 0.2). For FaCEE if the number of discretized value of Z increases, the MSE in general grows, but the accuracy peaks at 10 clusters and then declines. This suggests that FairinfoNCE can remove more sensitive information as the granularity of Z increases, but may hurt the downstream task performance. Overall, the FaircCLK performs slightly better than Fai, and do not need clustering to discretize Z.
4.3 HARD-NEGATIVES CONTRASTIVE LEARNING
In this subsection, we perform experiments within the hard negative contrastive learning frame- work (Robinson et al., 2020), which considers the embedded features as the conditioning variable Z. Different from conventional contrastive learning methods (Chen et al., 2020; He et al., 2020) that considers learning dissimilar representations for pair of random data, the hard negative contrastive learning methods learn dissimilar representations for pair of random data when they have similar embedded features (i.e.. similar outcomes from the conditioning variable).
Datasets and Metries. We consider two visual datasets in this set of experiments. Data X and Y represent images after applying arbitrary image augmentations. 1) CIFARIO (Krizhevsky et al., 2009): It contains 60, 000 images spanning 10 classes, e.g. automobile, plane, or dog. 2) ImageNet- 100 (Russakovsky et al., 2015): It is the same dataset that we used in Section 2. We report the top-1 accuracy as the metric for the downstream classification task.
Methodology. We consider the HardNegccLK objective (equation 9) as the main method. For PxYz by first sampling an image im, then applying different image augmentations on im; to obtain (æt, y:), and last defining zi = yox (. We select two baseline methods. The first one is the unconditional contrastive learning method: the InfoNCE objective (Oord et al., 2018; Chen et al., 2020) (equation 1). The second one is the conditional contrastive learning baseline: the HardNeggufoNNC- objective (Robinson et al., 2020), and we report its result directly using the released code from the author (Robinson et al., 2020).
HardNegccL.K we perform the sampling process {( yr: 2:) K))
Results. From Table 4, first, HardNegccLK consis-
tently shows better performances over the InfoNCE base- line, with absolute improvements of 1.8% and 3.1% on CIFAR- 10 and ImageNett 100 respectively. This sug- gests that the hard negative sampling effectively im- proves the downstream performance, which is in accor- dance with the observation by Robinson et al. (2020), Next, HardNegccLK also performs better than the HardNegintoNCe baseline, with absolute improvements of 0.3% and 2.0% on CIFAR-10 and ImageNet- 100 re- spectively. Both methods construct hard negatives by assigning a higher weight to a random paired data that are close in the embedding space and a lower weight to a random paired data that are far in the embedding space. The implementation by Robinson et al. (2020) uses Fuclidean distances to measure the similarity and HardNegccLK uses the smoothed kernel similarity (i.c., (Kz + AII -Kz in Proposition 2.2) measure the similarity. Empirically our approach performs better.
CIFAR-10
ImageNet-110
Unconditional Contrustive Learning Methods
InfoNCE
89.9 = 0.2
77.8 + 0.4
Conditional Contrastive Leaminn Methods
HardNegtus.oce
91.4 + 0.2
79.2 4 0.5
HardNegnotu (ours)
91.7
81.2 + 0.2
Table 4: Classification accuracy (%) under the hard negatives contrastive learning setup.
CONCLUSION
In this paper, we present CCL-K, the Conditional Contrastive Learning objectives with Kernel ex- pressions. CCL-K avoids the need to perform explicit conditional sampling in conditional contrastive learning frameworks, alleviating the insufficient data problem of the conditioning variable. CCL-K uses the Kernel Conditional Embedding Operator, which first defines the kernel similarities between the conditioning variable's values and then samples data that have similar values of the conditioning variable. CCL-K is can directly work with continuous conditioning variables, while prior work requires binning or clustering to ensure sufficient data for each bin or cluster. Empirically, CCL-K contrastive learning,
also outperforms conditional contrastive baselines tailored for fair contrastive learning, and hard negatives contrastive learning. An interesting future direction is to add more flexibility CCL.K, by relaxing the kernel similarity arbitrary similarity measurements.
9

---

Published as a conference paper at ICLR 2022
6 ETHICS STATEMENT
Because our method can improve removing sensitive information in contrastive learning representa- tions, our contribution can have positive impact on fairness and privacy. where biases user-specific information should be excluded from the representation. However, the conditioning variable must be predefined, so our method cannot directly remove any biases that are implicit and are not captured by variable in the dataset.
REPRODUCIBILITY STATEMENT
We provide an anonymous source code link for reproducing our result the supplementary material and include complete files which can reproduce the data processing steps for each dataset we use in the supplementary material.
ACKNOWLEDGEMENTS
The authors would like to thank the anonymous reviewers for helpful comments and suggestions. This work is partially supported by the National Science Foundation 1631763562, IARPA D17PC00340, ONR Grant N00014181288
Facebook PhD Fellowship, BMW, National Science Foundation awards
1722822 and 1750439, and National Institutes of Health awards RO1 MO1MH125740 RO1 MH096951 and UOIMHI 16925. KZ would like acknowledge the support by the National Institutes of Health (NIH) under Contract ROIHLL59805, by the NSF-Convergence Accelerator Track-D award 2134901, and by the United States Air Force under Contract No. FA8650-17-C77155 HZ would like to thank support from a Facebook research award. Any opinions, findings, conclusions, or recommendations expressed in this material those of the author(s) and do not necessarily reflect the views of the sponsors, and no official endorsement should be inferred.
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pages 214-223. PMLR, 2017.
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saun- shi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint 1X02:09902.0922 2019.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprint arXiv: 1906: 00900, 2019.
Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vee 2.0: A framework for self-supervised learning of speech representations. arXiv preprint arXiv:2006. 1477, 2020.
Gilles Blanchard, Aniket Anand Deshmukh, Urun Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning. arXiv preprint trXiv://11..79/0 2017.
Emmanuel J Candes, Justin K Romberg, and Terence Tao. Stable signal recovery from incomplete and inaccurate measurements. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences, 59(8):1207-13 2006.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pages 1597-1607. PMLR, 2020.
Don Coppersmith and Shmuel Winograd. Matrix multiplication via arithmetic progressions. In Proceedings of the nineteenth anmual ACM symposium Theory of computing, pages 1-6, 1987.
Kenji Fukumizu. Arthur Gretton, Xiaohai Sun, and Bernhard Schölkopf. Kernel measures of conditional dependence. In NIPS, volume 20, pages 489-496. 2007.
10

---

Published as a conference paper at ICLR 2022
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 297-304. JMLR Workshop and Conference Proceedings. 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738, 2020.
Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchilee Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint ArXiv: 008.066700 2018.
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. arXiv preprint arXiv:2010.01028, 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint arXiv:2004. / 1362, 2020.
Lingpeng Kong, Cyprien de Masson d' Autume, Wang Ling, Lei Yu, Zihang Dai, and Dani Yogatama. A mutual information maximization perspective of language representation learning, arXiv preprint vrXiv:1910.08350 2019.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 (6211))227822324 1998.
Jason D Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo Predicting what you already know helps: Provable self-supervised learning. arXiv preprint wxxx::2008.0/666 2020.
Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical programmingg 45(1):503-528. 1989.
Arnab Mondal, Arnab Bhattacharjee, Sudipto Mukherjee, Himanshu Asnani, Sreeram Kannan, and AP Prathosh. C-mi-gan: Estimation of conditional mutual information using minmax formulation. In Conference on Uncertainty in Artificial Intelligence, pages 849-858. PMLR, 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 107.v:1807.03748, 2018.
Taesung Park, Jun-Yan Zhu, Oliver Wang. Jingwan Lu, Eli Shechtman, Alexei A Efros, and Richard Zhang. Swapping autoencoder for deep image manipulation. arXiv preprint arXiv:2007.000 3,065,,,,,,, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint 103i::20330000 2021.
Joshua Robinson, Ching-Yas Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.0459) 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 11(())221 1-252, 2015.
11

---

Published as a conference paper at ICLR 2022
Bernhard Schölkopf, Alexander Smola, Francis Bach, et al. Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT press, 2002.
Abhishek Sinha, Jiaming Song, Chenlin Meng, and Stefano Ermon. D2c: Diffusion-denoising models for few-shot conditional generation. arXiv preprint arXiv:2106.068/ 2021.
Le Song, Kenji Fukumizu, and Arthur Gretton. Kernel embeddings of conditional distributions: unified kernel framework for nonparametric inference in graphical models. IEEE Signal Processing Magazine, 30(4):98-111 2013.
Jcan-Francois Ton, CHAN Lucian, Yee Whye Tch, and Dino Sejdinovic. Noise contrastive meta- learning for conditional density estimation using kernel mean embeddings. In International Conference on Artificial Intelligence and Statistics, pages 1099-1107. PMLR, 2021.
Yao-Hung Hubert Tsai, Tianqin Li, Weixin Liu, Peiyuan Liao, Ruslan Salakhutdinov, and Louis- Philippe Morency. Integrating auxiliary information in self-supervised learning. arXiv preprint arXiv:2106. 02869, 202
Yao-Hung Hubert Tsai, Martin Q Ma, Muqiao Yang. Han Zhao, Louis-Philippe Morency, and Ruslan Salakhutdinov. Self-supervised representation learning with relative predictive coding. arXiv preprint arXiv:2103.11277 2021b.
Yao-Hung Hubert Tsai, Martin Q Ma, Han Zhao, Kun Zhang, Louis-Philippe Morency, and Ruslan Salakhutdinov. Conditional contrastive learning: Removing undesirable information self- supervised representations. arXiv preprint
2021c
Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Moreney. Self-supervised learning from a multi-view perspective. In ICLR, 2021d.
Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. arXiv preprint arXiv: 13625, 2019.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Mike Wu, Milan Mosse, Chengxu Zhuang. Daniel Yamins, and Noah Goodman. Conditional negative sampling for contrastive learning of visual representations. arXiv preprint arXiv:2010.02037, 2020.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint 1rXiv:1708.088 2017.
Aron Yu and Kristen Grauman. Fine grained visual comparisons with local learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 192-199, 2014.
12

---

Published as a conference paper at ICLR 2022
A CODE
We include our code in a Github ccLK-zelease
link:
htttss///gthhbb.com/rrrayyy/
B ABLATION OF DIFFERENT CHOICES OF KERNELS
We report the kernel choice ablation study for FairçcLK and HardNegocLæ (Table 5). For the FaireeLK, we consider the synthetically created ColorMNIST dataset, where the background color for each image digit image randomly assigned. We report the accuracy of the downstream classification task, as well as the Mean Square Error (MSE) between the assigned background color and a color predicted by the learned representation. would like the learned representations to contain less color information. In Table 5, we consider the following kernel functions: RBF, Polynomial (degree of 3), Laplacian and Cosine. The performances using different kernels are consistent.
higher MSE score is better because we
For the HardNego we consider the CIFAR-I0 10 dataset the ablation study and use the top-1 accuracy for object classification the metric. In Table 5, we consider the following kernel functions: Linear, RBF, and Polynomial (degree of 3), Laplacian and Cosine. The performances using different kernels consistent.
Lastly, we also provide the ablation of different x² when using the RBF kernel. First, the performance trend does change too much and is sensitive different bandwidths for the RBF kernel. In Table 7 we show the result of using different o² in the RBF kernel. We found that using o² = 1 significantly hurts performance (only 3.0%), using 6² 1000 is also sub-optimal. The performances of o² 10, 100, 500 are close.
C ADDITIONAL RESULTS ON CIFARIO
In Section 4.3 in the main text, we report the results of HardNegcoLK as well as baseline methods on CIFARIO dataset by training with 400 epochs using 256 batch size. Here we also include the results with larger batch size (512 batch size) and longer training time (1000 epochs). The results are summarized in Table For reference, we name the training procedure with 256 batch size and 400 epochs setting 1 and the one with 512 batch size and 1000 epochs as setting 2. From Table 6 observe that HardNegocLK still has better performance comparing to HardNegnotaNC,, suggesting that our method is solid. However, the performance differences shrink between the vallina InfoNCE method and the Hard Negative mining approaches in general because the benefit hard negative mining mainly lies in the training efficiency, i.e., spend less training iteration discriminating the negative samples that have been already pushed away.
D FAIRITIONCE ON COLORMNIST
Here we include the results of performing FairifoNNC on the ColorMNIST dataset as baseline. We implemented FairmfoNNC based on Tsai et al. (2021c), where the idea remove the information of the conditional variable by sampling positive and negative pairs from the same outcome of the conditional variable each iteration. Same our previous setup in Section 4.2, we evaluate the results by the accuracy of the downstream classification task as well as the MSE value, both of which are deemed higher the better (as we want the learned representation contain less color information, so larger MSE is desirable because it means the representation contains less color information for reconstruction.) To perform FairmIoNCC and condition on the continuous color information, need to discretize the color variable using clustering method such K-means. We use K-means to cluster samples based on their color variable, and we use the following numbers of clusters: (3. 5, 10, 15, 20).
As shown in Table 8, with the number of clusters increasing, the downstream accuracy first increases then drops, peaking number of clusters being 10. The MSE values continue increase number of clusters increase. This suggests that FairmOONNE can remove more sensitive information as the granularity of Z increases, but the downstream task performance may decrease.
13

---

Published as a conference paper at ICLR 2022
accuarucy
85.5
85.0
84.5
84.0
WeaklySup
WeaklySup CCLK
10
20
30
40
50
Number Cluster
Illustration of the problem of insufficient samples in conditional contrastive learning. When the average number samples per outcome (cluster) of the conditional variable small (towards the left of x-axis in the figure), the previous conditional contrastive learning framework WeaklySupinfoNNE (blue) suffers, while the proposed WcaklySupecLk (black) outperforms WcaklySuptnsooce significantly and is very stable regardless of whether the samples are sufficient (towards the right of the x-axis) or insufficient (towards the left of the x-axis).
Figure 3:
E PERFORMANCE UNDER INSUFFICIENT NUMBER OF SAMPLES
We would illustrate why the insufficient sample would be a problem for conditional contrastive learning. We provide comparison of performances under different number of conditioning data. To be specific, we provide Figure 3, where the x-axis is the averaged number of data samples per discretized conditioning variable (cluster) for conditional contrastive learning, if like the WeaklySupinnfoNEE. The dataset is UT-ZZappss and the conditioning variable is the annotative attributes. The discretization is done by grouping instances that share the same annotative attributes to the same cluster. The blue line is WeaklySSptanoNNc. which requires discretized conditioning variables. The black line represents the proposed WeaklySupcer.e which does not require dis- cretization. As we can see from the figure, the performance of WeaklySupintoNcz suffers when the number data per cluster (conditioning variable) is small, and Weakly Suaacly outperforms WeaklySupmnioNce in all cases. From this example, (towards the origin in this figure), the proposed WeaklySupecLk outperforms WeaklySupinsoNc significantly.
framework
that when the data is very insufficient
F DATASET DETAILS
We provide the training details of experiments conducted on the following datasets: UT-Zappos50 (Yu and Grauman, 2014), CUB-200-2011 (Wah et al., 2011), CIFAR-10 (Krizhevsky et al., 2009), ColorMNIST (our creation), and ImageNet-100 (Russakovsky al., 2015).
F.1 UT-ZAPPOS50K
The following section describes the experiments we performed on UT-Zaapos50KK dataset.
The dataset is attributed to (Yu and Grauman, 2014) and available at the link: http::/visiii cs. utexas. ectp/proojeccss/inegraeeeedddd /ttpee/dddzzzz.... The dataset is for non-commercial use only.
Accessiblity
Fairecus
Top-1 Accuracy (f)
MSE (t)
HardNogocLL
CIFAR-10
kemee
86.2
57.6 10.6
Linear kernel
91.5 1: 0.2
Polynomial kemel
86.7 0.5
61.3 9.4
RBF kerne
91.7 0.1
Laplacian kemnel
85.0 0.9
72.8 +: 13.2
Polynomial kerne
90.3 = 0.4
Cosine kernel
86.4
64.7
Table Ablation study of different types of kernel choices. Left: digit classification accuracy of FairсCıKK in ColorMNIST, and MSE (higher the better) between the color in the original image and the color predicted based on the learned representation from that image. Higher MSE is better because intend to remove color information in the representation. Right: classification accuracy of HardNegocLK on CIFAR- 10 object classification. The performances using different kernels in both settings consistent.
14

---

Published as a conference paper at ICLR 2022
CIFAR-10 (setting 1)
CIEAR-I0 (setting
Unconditional Contrustive Learning Methods
InfoNCE
89.9 0.2
93.4
Conditional Contrastive Learning Methods
HardNegunooNC
91.4 0.2
93.6 - 0.2
HardNegoCLK (ours)
91.7 0.1
93.9
Table Results of HardNegocu. on CIFARIO dataset with two different training settings.
RBF d²
10
100
500
1000
Accuracy (%)
3.0 1 1.5
30.9
0.3
32.2 0.4
32.0
0.2
24.4 1: 0.9
Table 7: Result of WeaklySupocLK under different CUB dataset.
o² using the RBF kernel in the
Number of Clusters
Top-1 Accuracy (t)
MSE (?)
3
82.12 + 0.3
56.27 + 4.9
84.55 +
58.67 + 4.8
85,90 1
64.91 :
15
85.22 + 0.4
65.02 + 5.0
20
84.23 + 0.3
65.11 + 4.9
Table 8: Results of FaiR on the colored MNIST dataset with different numbers of clusters.
Data Processing The dataset contains images shoe from Zappos.com We downsamples the images to 32 x 32. The official dataset has 4 large categories following 21 sub-categories. We utilize 21 subcategories for all our classification tasks. The dataset comes with attributes as auxiliary information. We binarize the 7 discrete attributes into 126 binary attributes. We consider our conditional variable Z is this 128 dimensional variable.
Training and Test Split: We randomly split train-validattio images by 7 : ratio, resulting in 35, 017 train data and 15, 008 validation dataset.
Network Design and Optimization We use ResNet-50 architecture to serve as a backbone for the encoder. To compensate the 32x32 image size, change the first 7x7 2D convolution 3x3 2D convolution and remove the first max pooling layer in the normal ResNet-50 (See code for details). This allows a finer grain information processing. After using the modified ResNet-50 as the encoder, we include a 2048-2048 1194--18 Multi-Layer Perceptron (MLP) as the projection head. Batch normalization used after each 2048 activation layers. During the evaluation, we discard the projection head and train linear layer on top of the encoder's ' output. We train 1000 epochs for all experiments with LARS optimizer (base learning rate 1.5 and scale learning rate based on our batch size divided by 256) with batch size 152 on 4 NVIDIA 1080ti GPUs. It takes about 16 hours to finish 1000 epochs training.
F.2 CUB-200-2011
The following section describes the experiments we performed on CUB-200-2011 dataset.
Accessiblity 200-200-2011 created by Wah et al. (2011) and is a fine-grained dataset for bird species. It can be downloaded from the link: http://www vision. caltech. edu/ visipedia//e / 2000-200-2011..1 html. The usage is restricted to noncommercial research and educational purposes.
15

---

Published as a conference paper at ICLR 2022
0
0
0
0
0
0
O 2
/
1
/
(
2
2
2-
2.
8
2
2
2
2
2
8
2
2
2
2
2
8
3
3
3
3
3
3
3
4
4
4
4
4
4
4
4 6
4 5
6
5
5
5
5
5
5
5
5
5
5
5
6
6
6
6
6
6
6
6
6
6
6
6 6 7 8 K
7
7
T
1
8
8
8
8
P
6
9
Figure 4: Creation of ColorMNIST for experiments on Fair-InfoNCE validation.
Data Processing The original dataset contains 200 birds categories over 11, 788 images with 312 binary attributes attached to each image. The image is rescaled to 224 x 224.
Train Test Split: We follow the original train-validation split, resulting in 5, 994 train images and 5, 794 validation images. We combine the original training and validation set as our training set and the original test set validation set. The resulting training set contains 6, 871 images and the validation set contains 6, 918 images.
Network Design and Optimization We use ResNet-50 architecture encoder. We choose 2048-2048- 128 MLP as the projection head. Batch normalization is used after each 2048 activation Differeif than UT-Zappos dataset, we directly employ the original design of ResNet-50 since we are training on 224x224 images. Similarly, LARS is used for optimization during the contrastive learning pretraining and we fine tune a linear layer and use Limited Limited-memory BFGS (L-BFGS (Liu and Nocedal, 1989)) optimizer after pretraining. All experiments are run with 1000 pretraining iterations and 500 L-BFGS fine tuning steps. We use 128 batch size and train on 1080ti NVIDIA GPUs. It takes about 13 hours to finish 1000 epochs training.
F.3 CIFAR-10
The following section describes the experiments we performed on CIFAR-10.
Accessibility CIFAR-10 (Krizhevsky et al., 2009) is an object detection dataset with 60, 000 32 32 images in 10 classes. The test set includes 10, 000 images. The dataset can be downloaded at wttpt:pp/// /www. cs toronto. wwt.eeedddddeeeeeeeeeeeeeea
html.
Data Processing and Train and Test split We use the training and test split from the original dataset.
Network Design and Optimization We employ ResNet-50 backbone architecture, but we change the first 7x7 2D convolution to 3x3 2D convolution and remove the first max pooling layer in the normal ResNet-50 (See code for details). This allows better results on CIFARIO as this dataset consists of 32x32 resolution images. 2048-2048-11 128 projection head is employed during contrastive learning. Batch normalization is used after each 2048 activation layers. There are two CIFAR training settings we consider. The first setting, which is reported in the main text, trains contrastive learning with 256 batch size for 400 epochs. It takes a GPU 1080ti Machine hours to finish the pretraining. For the second setting where train with 512 batch size for 1000 epochs, it takes an DGX-1 machine 48 hours to finish training. We use LARS optimizer for all CCL-K related experiments with base 1r=1.5 and base batch size equals 256.
F.4 CREATION OF COLORMNIST
Accessiblity We create ColorMNIST dataset for experiments in Section 4.2 in the main text. The train and test split images can be accessed from our anonymous Github link (Section A). We allow any non-commerical usage of our dataset.
16

---

Published as a conference paper at ICLR 2022
Data Processing As discussed in Section 4.2 in the main text, create the ColorMNIST dataset by assigning a random sampled color as MNIST's background. Images are converted into 32x32 resolution and only the background is augmented with the sampled color while the digit stroke pixel remains black. Examples of the ColorMNIST images are shown in Figure 4.
Training and Test Split We follow the original MNIST train/test split, resulting in 60,000 training images and 10,000 testing images spanning 10 digit categories.
Network Design and Optimization To train our model, we use LeNet-5 (LeCun al., 1998) as backbone architecture and use 2 layer linear projection head to project it to 128 dimension. We use LARS (You et al., 2017) as our optimizer. After our network pretrained using contrastive learning, we discard the head annd fine tune linear layer use Limited-memory BFGS (L-BFGS (Liu and Nocedal, 1989)) as optimizer. All experiments are run with 1175 pretraining iterations and 500 L-BFGS fine tuning steps.
F.5 IMAGENET-100
The following section describes the experiments we performed on ImageNet-100 dataset in Section in the main text.
Accessibility This dataset is subset of ImageNet- K dataset, which comes from the ImageNet Large Scale Visual Recognition Challenge (ILSVVCC 2012-2017 (Russakovsky et al., 2015). ILSVRC is for non-commercial research and educational purposes and we refer the ImageNet official site for more information:
https: / /www. image net org/ / download. php.
Data Processing We select 100 classes from ImageNet-1 K conduct experiments. Selected class names can be accessed from our Github link.
Training and Test Split: The training split contains 128, 783 images and the test split contains 5, 000 images. The images are rescaled to size 224 x 224.
Network Design and Optimization Hyper-parameters We use conventional ResNet-50 as the backbone for the encoder. 2048-2048-15 28 MLP layer and a 12 normalization layer is used after the encoder during training and discarded in the linear evaluation protocol. Batch normalization used after each 2048 activation layers. For optimization, setting and 512 batch size for HardNegccI.k Open AI CLIP model (Radford et al., 2021) is used to extract continuous feature from the image. WeaklySup
choose 128 batch size for WeaklySuPeeux
discretize the Z space using Kmeans clustering with k=100, 200, 500, 2500. The best result of Weakly is produced by k=200. All experiments are trained with 200 epochs and require 53 hours of training DGX machine with Tesla P100 GPUs.
17


Reviews:
[{"review_id": "lnVrIq_hON9", "reviewer": "Reviewer_kZnA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Authors proposes a solution to a very important problem. I will not question neither the motivation nor the importance of the problem. Paper is well written and mostly connects previous art. However, I believe some relations to needs be build with kernel literature (I will give examples below). \nAlthough in general I am positive about the paper, I have some concerns:\n-\tAuthors rely on side information which they assume is a part of the training data. In many practical cases, the cause of any bias/unfairness in data is caused by human annotation. Image annotations (cited by authors) may contain sexist/racist information. How does proposed method deal with such problems?\n-\tAuthors conditions on z_i (please see Step – 1 Problem Setup) and develops a kernel and use it as similarity function. Indeed, one can see each z_i as a domain and K_z as a probability on densities. Please see the following the following paper\n[1] Domain Generalization by Marginal Transfer Learning. JMLR, 2021. Please section 5.1.  \nThe proposed kernel in the paper is a simplified version of kernels described in Section 5.1 of [1]. It is clear that [1] is not considering neural networks. However, a discussion is needed because of the following reason:\n-\tGiven that authors are learning ‘a fair’ representation from data, I would like to understand the reason of fairness. If authors would have explicitly assumed a model than it is clear that ‘fairness’ is coming from the model. However, in the current version, I am not sure the whether the observed performance is because of ‘fairness’. It is very well possible that implicitly modelling problem as a domain generalization problem played a significant role.\nMoreover, the final version of the used kernel has similarities with \n[2] Multi-Task Learning for Contextual Bandits (Neurips 2017, I will use the arxiv version https://arxiv.org/pdf/1705.08618.pdf). Please see the first eq on page 15.\nAgain [2] is in the space of kernel methods however taking into account multiple domains (or multiple tasks) and defining a kernel on these tasks seems improving the baselines.\nWould you please discuss the connection of the proposed method with [1] and [2]? Moreover, would you please elaborate why the proposed methods is limiting the undesired information?\n", "summary_of_the_paper": "Authors proposed to use nonparametric methods for sampling data point for contrastive learning. The main concern of the paper is to limit extraction of any undesired information from the data.", "summary_of_the_review": "Overall, I like the paper. However, I have several concerns or confusions. I am looking forward to author responses and engagement such that I can keep my score or increase it. \n\n(After the rebuttal, I have decided to increase my score)", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "V2xHVsCAYTG", "reviewer": "Reviewer_imU8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Reasons to accept the paper:\n- the paper proposed a novel way to use more contrastive examples by weighting the datapoints using CME.\n- the paper is clearly written with a simple idea applied to conditional contrastive learning\n- the experimental results look very promising on a variety of experiments\n\nReasons to reject the paper:\n- the paper completely dismisses the work done in [1] which uses almost the exact same idea but in the meta learning setting and conditional density estimation. Also [1] uses NCE instead of batch wise representation learning for negative samples. However, it is undeniable that the objective function proposed in this paper is a simple derivation from [1] and adapted to this representation learning setting. Furthermore, the proposition 2 the authors write inner product with respect to H. What is H in this case? As a matter of fact the inner product should equal to the evaluation of the CME. Could the authors elaborate in this?\n\n- What is the computational complexity of the methods ? Given that you have more comparisons does that play a role?\n\n- InfoNCE is hardly the best representation learning algorithm these days. Could the authors please elaborate why methods such as SimCLR?\n\n- the results look. very promising. However I am curious to see whether the authors could also add the table for different kernels from the other 2 datasets as I can't seem to see them in the Appendix. In addition, please correct me if I am wrong, I do not see how you currently select the hyper parameters for the RBF kernel etc. How have these been selected and was the test set used? The reason I am asking this is because if the linear kernel is good enough it is questionable how much kernels even play a role.\n\n- Are the results reported with 1 or 2 deviations?\n\n- How do the competing methods perform as you increase/decrease the conditioning data. It would be very enlightening to see a plot where the x axis has the number of conditioning data and y axis the performance to see how your method performs much better when there is little conditioning data available. Could the authors please comment and report on this? \n\n\n[1]: Ton et al 2021 https://arxiv.org/abs/1906.02236", "summary_of_the_paper": "The paper looks into using more conditional data than standard methods by leveraging conditional mean embeddings and this weighting more datapoint when performing contrastive learning. The methodology hence allows to more comparisons (but weighted) and hence allowing the proposed method to achieve higher representational power. The idea of CME allows them to compare the conditional data (ie usually a smaller set) with the batch and then still perform contrastive learning now will the batch. They show improved performance on a variety of experiments such as fairness, representation learning and weakly supervised learning.", "summary_of_the_review": "I believe that the paper as its merits and really like how they have seemingly incorporated CME into representation learning and obtain good performances. However, there are still important references missing which I believe reduce the novelty by quite a margin given that it s a simple derivation from [1] applied to different setting. In addition, there are some clarification on the experimental side which I would like know. **I am more than happy to increase my score if the authors are able to clarify my concerns and will be looking forward to their rebuttal.**", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RdoGRfAPfII", "reviewer": "Reviewer_oJk7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1.\tThe idea of using the kernel function to address the conditional sampling procedure seems novel to me. Experimental results also verify its effectiveness in weakly supervised contrastive learning and hard-negatives contrastive learning tasks. \n2.\tThe motivation of this paper is clear. The paper is well-written and easy to follow.\n3.\tCodes and data are provided for reproducibility. \n\nWeakness:\n1. Computational complexity of CCL-K is missing. I think it has a higher computational cost compared with baselines due to the computation of gram matrix and its inversion. \n2. I wonder whether Fair_{CCLK} can actually remove sensitive information of conditioning variable in the learned representations. Because the calculation of $K_{X \\perp Y \\mid Z}$ involves $K_{Z}$, i.e., the similarities of the conditioning variable. Can the information of this conditioning variable be actually removed when it is explicitly incorporated in the loss function of Fair_{CCLK}? The experimental result of Fair_{CCLK} is not very convincing because it is not compared to other fair contrastive learning baselines, such as Fair_{InfoNCE}. And the reason why Fair_{InfoNCE} is not used as a baseline is not compelling. Similar to WeaklySup_{InfoNCE} on ImageNet-100 dataset, Fair_{InfoNCE} can still be applied in ColorMNIST dataset by converting the continuous conditioning variable to a discrete one through clustering. Or the easier way is to compare these two fair contrastive learning methods on datasets with discrete conditioning variables.\n3. I notice that the loss functions of Fair_{CCLK} and HardNeg_{CCLK} are exactly the same. However, the goal of Fari_{CCLK} is to remove sensitive information of Z while the goal of HardNeg_{CCLK} is to learn better representations from hard negative samples. So I think it will be better if the paper could elaborate more about how CCLK can achieve both goals with the same loss function and discuss their differences. \n4. The key of CCLK is to use kernel function to measure the similarities between samples, i.e., the definition of $w(z_j, z_i)$. I think it would be better if an ablation study could be conducted to examine the effect of this component. Specifically, we could just fix all $w(z_j, z_i)$ to a constant value, such as $1/b$, to see how the CCLK performs. It is currently unclear whether the performance improvement of CCLK comes from the kernel function or the availability of all the data. This kind of ablation study could help to verify the effective part of CCLK.\n", "summary_of_the_paper": "This paper studies conditional contrastive learning tasks when there is no enough data for some values of the conditioning variable. This scenario can be problematic in the conditional sampling procedure. To mitigate this problem, this paper proposes a CCL-K model. It uses a Kernel Conditional Embedding Operator to sample from all the available data and assign a kernel similarity to each sampled data, which is based on the values of the conditioning variable. The CCL-K also extends conditional contrastive learning to deal with continuous conditioning variables. ", "summary_of_the_review": "The idea of using a kernel function to address the conditional sampling problem in contrastive learning is interesting and novel. The experimental results validate its effectiveness in some contrastive learning tasks. However, my major concern is that whether  Fair_{CCLK} can actually remove sensitive information of conditioning variables (See the second point of the Weakness in Main Review). More elaboration regarding this issue is needed. Also, Fair_{CCLK} should also compare to other fair contrastive learning methods to verify its effectiveness. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "O73pKEv80kK", "reviewer": "Reviewer_F3Tg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper well extends the Kernel Conditional Embedding Operator to the contrastive learning framework. The authors present their method in both clear formulation and extensive experiments in three different tasks. The paper is generally correct but a little bit hard to follow. Some motivations mentioned in the introduction are not well addressed in the method and experiments part (e.g. CCL-K can work with insufficient data, for continuous learning, etc.), thus I am still a little bit confused about the advantage of CCL-K in practice.\n\nFrom the side of technique in the paper, the proposed sampling method is not novel, as the weighted sum form has been discussed by various paper [1-3]. The novelty lies in the weight estimation, which is achieved by the kernel score.  However, the difference between calculating similarity in the original space and the encoder space is not elaborated enough. The results in Table 2 (right table) seems to show this difference is not significant, which makes the usage of kernel a little bit trivial.\n\n\nThe experiments show the effectiveness of CCL-K compared to InfoNCE loss in weekly-supervised CL, fair CL, but in the hard-negative CL case the improvement is a little bit marginal compared to hard-negative InfoNCE on CIFAR10. Moreover, some papers involving different negative sampling strategy are not compared in the experiments, though they are cited.  \n\n[1] Contrastive learning with hard negative samples.\n[2] Conditional negative sampling for contrastive learning of visual representations.\n[3] Contrastive Attraction and Contrastive Repulsion for Representation Learning.\n", "summary_of_the_paper": "This paper presents Conditional Contrastive Learning with Kernel, a kernel sampling method for contrastive learning.  This sampling method leverages the Kernel Conditional Embedding Operator and measures the similarity between the conditioning variables. The authors test the performance of CCL-K in three application scenarios: weakly supervised contrastive learning, fair contrastive learning, and hard negative contrastive learning.", "summary_of_the_review": "This paper proposes an importance weight sampling technique to improve the InfoNCE loss, where the weight is estimated by using kernel embedding operator, but as the evidence of advantages of using the kernel estimation is not sufficiently shown. I am inclined to recommend a reject of this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all acknowledge the importance of the paper as it addressed the challenge of the insufficient data problem in conditional contrastive learning, feeling that the idea was novel, the experiments verified the effectiveness of the model well, and the paper is well written. Reviewers also raised some good questions, such as the computational complexity, comparison with Fair_InfoNCE in the experiments, and kernel ablations. These questions are well addressed in the rebuttal and the revised version. One reviewer raised the issue of similarity to [1]. After taking a close look at this paper and [1], the AC felt that the motivation and focus of this paper are quite different from [1]. The authors should incorporate all the rebuttal info into the final version.

[1] Jean-Francois Ton et al. 2021.
Decision: Accept (Poster)