Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Authors proposed to use nonparametric methods for sampling data point for contrastive learning. The main concern of the paper is to limit extraction of any undesired information from the data.
main_review: Authors proposes a solution to a very important problem. I will not question neither the motivation nor the importance of the problem. Paper is well written and mostly connects previous art. However, I believe some relations to needs be build with kernel literature (I will give examples below). 
Although in general I am positive about the paper, I have some concerns:
-	Authors rely on side information which they assume is a part of the training data. In many practical cases, the cause of any bias/unfairness in data is caused by human annotation. Image annotations (cited by authors) may contain sexist/racist information. How does proposed method deal with such problems?
-	Authors conditions on z_i (please see Step – 1 Problem Setup) and develops a kernel and use it as similarity function. Indeed, one can see each z_i as a domain and K_z as a probability on densities. Please see the following the following paper
[1] Domain Generalization by Marginal Transfer Learning. JMLR, 2021. Please section 5.1.  
The proposed kernel in the paper is a simplified version of kernels described in Section 5.1 of [1]. It is clear that [1] is not considering neural networks. However, a discussion is needed because of the following reason:
-	Given that authors are learning ‘a fair’ representation from data, I would like to understand the reason of fairness. If authors would have explicitly assumed a model than it is clear that ‘fairness’ is coming from the model. However, in the current version, I am not sure the whether the observed performance is because of ‘fairness’. It is very well possible that implicitly modelling problem as a domain generalization problem played a significant role.
Moreover, the final version of the used kernel has similarities with 
[2] Multi-Task Learning for Contextual Bandits (Neurips 2017, I will use the arxiv version https://arxiv.org/pdf/1705.08618.pdf). Please see the first eq on page 15.
Again [2] is in the space of kernel methods however taking into account multiple domains (or multiple tasks) and defining a kernel on these tasks seems improving the baselines.
Would you please discuss the connection of the proposed method with [1] and [2]? Moreover, would you please elaborate why the proposed methods is limiting the undesired information?

summary_of_the_review: Overall, I like the paper. However, I have several concerns or confusions. I am looking forward to author responses and engagement such that I can keep my score or increase it. 

(After the rebuttal, I have decided to increase my score)
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies conditional contrastive learning tasks when there is no enough data for some values of the conditioning variable. This scenario can be problematic in the conditional sampling procedure. To mitigate this problem, this paper proposes a CCL-K model. It uses a Kernel Conditional Embedding Operator to sample from all the available data and assign a kernel similarity to each sampled data, which is based on the values of the conditioning variable. The CCL-K also extends conditional contrastive learning to deal with continuous conditioning variables. 
main_review: Strength:
1.	The idea of using the kernel function to address the conditional sampling procedure seems novel to me. Experimental results also verify its effectiveness in weakly supervised contrastive learning and hard-negatives contrastive learning tasks. 
2.	The motivation of this paper is clear. The paper is well-written and easy to follow.
3.	Codes and data are provided for reproducibility. 

Weakness:
1. Computational complexity of CCL-K is missing. I think it has a higher computational cost compared with baselines due to the computation of gram matrix and its inversion. 
2. I wonder whether Fair_{CCLK} can actually remove sensitive information of conditioning variable in the learned representations. Because the calculation of $K_{X \perp Y \mid Z}$ involves $K_{Z}$, i.e., the similarities of the conditioning variable. Can the information of this conditioning variable be actually removed when it is explicitly incorporated in the loss function of Fair_{CCLK}? The experimental result of Fair_{CCLK} is not very convincing because it is not compared to other fair contrastive learning baselines, such as Fair_{InfoNCE}. And the reason why Fair_{InfoNCE} is not used as a baseline is not compelling. Similar to WeaklySup_{InfoNCE} on ImageNet-100 dataset, Fair_{InfoNCE} can still be applied in ColorMNIST dataset by converting the continuous conditioning variable to a discrete one through clustering. Or the easier way is to compare these two fair contrastive learning methods on datasets with discrete conditioning variables.
3. I notice that the loss functions of Fair_{CCLK} and HardNeg_{CCLK} are exactly the same. However, the goal of Fari_{CCLK} is to remove sensitive information of Z while the goal of HardNeg_{CCLK} is to learn better representations from hard negative samples. So I think it will be better if the paper could elaborate more about how CCLK can achieve both goals with the same loss function and discuss their differences. 
4. The key of CCLK is to use kernel function to measure the similarities between samples, i.e., the definition of $w(z_j, z_i)$. I think it would be better if an ablation study could be conducted to examine the effect of this component. Specifically, we could just fix all $w(z_j, z_i)$ to a constant value, such as $1/b$, to see how the CCLK performs. It is currently unclear whether the performance improvement of CCLK comes from the kernel function or the availability of all the data. This kind of ablation study could help to verify the effective part of CCLK.

summary_of_the_review: The idea of using a kernel function to address the conditional sampling problem in contrastive learning is interesting and novel. The experimental results validate its effectiveness in some contrastive learning tasks. However, my major concern is that whether  Fair_{CCLK} can actually remove sensitive information of conditioning variables (See the second point of the Weakness in Main Review). More elaboration regarding this issue is needed. Also, Fair_{CCLK} should also compare to other fair contrastive learning methods to verify its effectiveness.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper looks into using more conditional data than standard methods by leveraging conditional mean embeddings and this weighting more datapoint when performing contrastive learning. The methodology hence allows to more comparisons (but weighted) and hence allowing the proposed method to achieve higher representational power. The idea of CME allows them to compare the conditional data (ie usually a smaller set) with the batch and then still perform contrastive learning now will the batch. They show improved performance on a variety of experiments such as fairness, representation learning and weakly supervised learning.
main_review: Reasons to accept the paper:
- the paper proposed a novel way to use more contrastive examples by weighting the datapoints using CME.
- the paper is clearly written with a simple idea applied to conditional contrastive learning
- the experimental results look very promising on a variety of experiments

Reasons to reject the paper:
- the paper completely dismisses the work done in [1] which uses almost the exact same idea but in the meta learning setting and conditional density estimation. Also [1] uses NCE instead of batch wise representation learning for negative samples. However, it is undeniable that the objective function proposed in this paper is a simple derivation from [1] and adapted to this representation learning setting. Furthermore, the proposition 2 the authors write inner product with respect to H. What is H in this case? As a matter of fact the inner product should equal to the evaluation of the CME. Could the authors elaborate in this?

- What is the computational complexity of the methods ? Given that you have more comparisons does that play a role?

- InfoNCE is hardly the best representation learning algorithm these days. Could the authors please elaborate why methods such as SimCLR?

- the results look. very promising. However I am curious to see whether the authors could also add the table for different kernels from the other 2 datasets as I can't seem to see them in the Appendix. In addition, please correct me if I am wrong, I do not see how you currently select the hyper parameters for the RBF kernel etc. How have these been selected and was the test set used? The reason I am asking this is because if the linear kernel is good enough it is questionable how much kernels even play a role.

- Are the results reported with 1 or 2 deviations?

- How do the competing methods perform as you increase/decrease the conditioning data. It would be very enlightening to see a plot where the x axis has the number of conditioning data and y axis the performance to see how your method performs much better when there is little conditioning data available. Could the authors please comment and report on this? 


[1]: Ton et al 2021 https://arxiv.org/abs/1906.02236
summary_of_the_review: I believe that the paper as its merits and really like how they have seemingly incorporated CME into representation learning and obtain good performances. However, there are still important references missing which I believe reduce the novelty by quite a margin given that it s a simple derivation from [1] applied to different setting. In addition, there are some clarification on the experimental side which I would like know. **I am more than happy to increase my score if the authors are able to clarify my concerns and will be looking forward to their rebuttal.**
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents Conditional Contrastive Learning with Kernel, a kernel sampling method for contrastive learning.  This sampling method leverages the Kernel Conditional Embedding Operator and measures the similarity between the conditioning variables. The authors test the performance of CCL-K in three application scenarios: weakly supervised contrastive learning, fair contrastive learning, and hard negative contrastive learning.
main_review: This paper well extends the Kernel Conditional Embedding Operator to the contrastive learning framework. The authors present their method in both clear formulation and extensive experiments in three different tasks. The paper is generally correct but a little bit hard to follow. Some motivations mentioned in the introduction are not well addressed in the method and experiments part (e.g. CCL-K can work with insufficient data, for continuous learning, etc.), thus I am still a little bit confused about the advantage of CCL-K in practice.

From the side of technique in the paper, the proposed sampling method is not novel, as the weighted sum form has been discussed by various paper [1-3]. The novelty lies in the weight estimation, which is achieved by the kernel score.  However, the difference between calculating similarity in the original space and the encoder space is not elaborated enough. The results in Table 2 (right table) seems to show this difference is not significant, which makes the usage of kernel a little bit trivial.


The experiments show the effectiveness of CCL-K compared to InfoNCE loss in weekly-supervised CL, fair CL, but in the hard-negative CL case the improvement is a little bit marginal compared to hard-negative InfoNCE on CIFAR10. Moreover, some papers involving different negative sampling strategy are not compared in the experiments, though they are cited.  

[1] Contrastive learning with hard negative samples.
[2] Conditional negative sampling for contrastive learning of visual representations.
[3] Contrastive Attraction and Contrastive Repulsion for Representation Learning.

summary_of_the_review: This paper proposes an importance weight sampling technique to improve the InfoNCE loss, where the weight is estimated by using kernel embedding operator, but as the evidence of advantages of using the kernel estimation is not sufficiently shown. I am inclined to recommend a reject of this paper.
review_title: null

Meta-review: The reviewers all acknowledge the importance of the paper as it addressed the challenge of the insufficient data problem in conditional contrastive learning, feeling that the idea was novel, the experiments verified the effectiveness of the model well, and the paper is well written. Reviewers also raised some good questions, such as the computational complexity, comparison with Fair_InfoNCE in the experiments, and kernel ablations. These questions are well addressed in the rebuttal and the revised version. One reviewer raised the issue of similarity to [1]. After taking a close look at this paper and [1], the AC felt that the motivation and focus of this paper are quite different from [1]. The authors should incorporate all the rebuttal info into the final version.

[1] Jean-Francois Ton et al. 2021.
Decision: Accept (Poster)