Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper tackles the problem of learning representations that are robust to biases in the data. The authors first clarify why existing contrastive losses fail to deal with biased data, and further derive a novel formulation of supervised contrastive loss to provide more accurate control of minimal distance between positive and negative samples. Moreover,the authors propose a new debiasing regularization loss to deal with extremely biased data. 

To evaluation the proposed losses, the authors first provide a benchmark on standard classification datasets CIFAR10, CIFAR100, ImageNet to evaluate different formulations of the proposed loss terms. The authors further conduct experiments on biased datasets Biased-MNIST, Corrupted CIFAR-10, bFFHQ and 9-Class ImageNet and ImageNet-A, which contains color biases or texture biases in the data. The experiments show better results in comparison to other debiasing techniques  
strength_and_weaknesses: + The paper studies an important problem and proposes reformulations upon the contrastive loss to address the biases in the data.

- The motivation that drive the model formulation is not clear. In particular, t is not very clear why contrastive losses should be used as for the formulations to address the biases in the data. Why can't one also consider other metric loss such as the triplet loss, or consider classification loss that also allow to introduce margin or temperature to address the biases in the data? 

- The proposed method lacks intuitive explanations about why the reformulations of contrastive loss can properly address the biases in the data. From the formulations, it turns out the authors introduce an additional margin parameter to ensure the relevant distance of a positive wrt an anchor and the nearest negative. The formulations are quite similar to the idea of triplet loss  [a,b], but it is unclear why and how it resolve biases in data. 

- What kinds of biases do the paper aims to tackle? It seems that the bias in this paper is quite related to domain drift across datasets. How does the proposed technique compared to ones that address the domain drift in data? 

- There is an existing formulation of supervised contrastive loss proposed in [c]. What is the difference between the SupInfoNCE proposed in this paper and the one proposed in [c]? 

[a] FaceNet: A Unified Embedding for Face Recognition and Clustering. CVPR 2015
[b] In Defense of the Triplet Loss for Person Re-Identification. arXiv2017
[c] Supervised Contrastive Learning. NeuRIPS 2020


clarity,_quality,_novelty_and_reproducibility: Clarity: 
The proposed method is interesting, however, it would be better to give more intuitive explanations about why the reformulations of the contrastive losses can properly address the biases in datasets. In particular, Section 3.1 could be improved with some visual illustration of the loss design. 

Quality: 
The paper contains solid formulations and experiments with comparison on different datasets. More efforts could be spent to make the method formulation clearer. 

Novelty: 
The proposed method is related to a couple of existing methods as indicated in Section 3.1.1, but it also has constraints different from other methods. 

Reproducibility:
The authors provide in the supplementary for reproducing their results.


summary_of_the_review: The paper studies an important problem and proposes reformulations upon the contrastive loss. Experiments are conducted to verify the effectiveness of the proposed method. However, the writing about proposed method is suggested to be improved with better clarity as mentioned above in the section of Strength And Weaknesses.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors use a common approximation of the max operator to simply motivate and formulate eps-SupInfoNCE, their proposed loss for contrastive learning, and FairKL, their proposed regularization term for mitigating effects of bias in datasets. Also using this approximation, they formulate prior work such as InfoNCE, InfoL1O, and SupCon, and highlight some of their deficiencies. They demonstrate eps-SupInfoNCE and FairKL across a variety of common vision datasets and their biased variants, showing that eps-SupInfoNCE performs favorably to alternatives on CIFAR-10, CIFAR-100, and a small variant of ImageNet, and showing that the combination of eps-SupInfoNCE + FairKL performs favorably on Biased-MNIST, Corrupted CIFAR-10, bFFHQ (faces of younger females and older males), and small variants of ImageNet with known biases.
strength_and_weaknesses: The paper is very clear, especially when taking into account the derivations in the appendix that would usually be glossed over. I liked that the authors include both a thorough related work section *and* section 3.1.1, which makes direct comparisons with closely related work after introducing the method (for a more in-depth understanding). In addition to formulating eps-SupInfoNCE and eps-SupCon, the discussion around prior work, particularly Equations 6 and 7, is interesting and insightful. Calling Equation 2 a 'theoretical framework' might be a bit much, but still it is insightful, both in the context of prior work and the proposed methods. The experiments do admittedly lie on the toy-ish side, but are for the most part convincing.

I see only one potential weakness: it appears to me that FairKL is applicable to many constrastive learning methods; is this correct or incorrect? If correct, then there is an experimental gap, in that, with respect to handling bias, the comparison is always between eps-SupInfoNCE + FairKL vs. other methods. Why are they intertwined in the experiments? To evaluate FairKL, wouldn't it make sense to include many experiments without eps-SupInfoNCE?
clarity,_quality,_novelty_and_reproducibility: The manuscript is clear and of high quality. I am not an expert in this area, but in trusting the authors that the derivations of related methods + eps-SupInfoNCE are original, then the work is sufficiently novel. The methods are well explained, and I don't see reproducibility as being an issue, but at the same time, the authors seem to not be releasing code, which is unfortunate.
summary_of_the_review: The 'Summary of the Paper' and 'Strengths and Weaknesses' sections speak for themselves: overall, the simple derivation of related work + new methods is easy to follow and leads to convincing results across various datasets.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the problem of learning representation robust to biased data, i.e. data containing easy-to-learn but unrelated features. It first derives \epsilon-SupInfoNCE, which adds control of the minimal distance between positive and negative samples to the traditional contrastive losses. Then it proposes FairKL, a regularization loss minimizing the KL divergence between the distribution of distances to bias-conflicting samples and the distribution of distances to bias-aligned samples. Experiments show \epsilon-SupInfoNCE + FairKL outperforms other debiasing methods on several biased datasets.
strength_and_weaknesses: Pros:
- The topic of this paper is interesting. It discusses whether traditional contrastive losses can deal with biased data, and how to learn a good representation that is robust to biased data. 
- This paper proposed a natural and reasonable generalization of the traditional contrastive losses which has more control of the minimal distance between positive and negative samples. Figure 1 presents a nice illustration of how it can mitigate the biased issue.
- The proposed FairKL regularization addresses the biased issue directly, and shows better performance compared with existing methods on biased datasets, including synthesized and real-world datasets.

Cons:
- The related works section is very hard to follow. It just lists a lot of literature on debiasing without organization. It does not discuss how those works are related or different from this paper either. The literature of contrastive learning is totally missing.
- The empirical validation of \epsilon-SupInfoNCE is not convincing. The SimCLR and Max-Margin results in Table 2 are collected from [Khosla et al. (2020)]. However, the results of CE and SupCon are re-implemented and are worse than that reported in [Khosla et al. (2020)]. (The results of SupCon reported in the original paper is 76.5 for CIFAR-100, which is even slightly better than the proposed \epsilon-SupInfoNCE.) How could those four columns be put together in a table? Also, the results on ImageNet are not included - the performance of the proposed method on large-scale datasets has not been validated.
- The experiments on biased datasets are not enough. It does not investigate how the traditional contrastive losses(e.g. SupCon) performs empirically and how much the proposed method can outperform them on biased datasets, as it discusses analytically in the method section. Moreover, it is not clear how much the improvement on the biased datasets comes from \epsilon-SupInfoNCE and how much comes from FairKL. How well will \epsilon-SupInfoNCE perform without FairKL? How much benefit will FairKL add to methods other than \epsilon-SupInfoNCE? 

Question:
- For the FairKL, it is reasonable that considering the standard deviation in addition to the mean value of the distances is helpful. However, I do not think the argument of the standard deviation in 3.1.1 (the discussion for EnD)  is correct. The centroid of samples is not the mean of distances, right? In Figure 1(b), even if the green samples and other-color samples have the same centroid, their average distance to the anchor is not the same - the average distance of the green ones is smaller.  Also, why can we assume the distance distribution follows a normal distribution? 

clarity,_quality,_novelty_and_reproducibility: The writing of the paper can be improved. For example, as mentioned above, the related works section should be re-organized and include contrastive learning literature, and the explanation in the method section should be checked and be more clear.

The proposed method is different from existing works, but as mentioned above, it is not enough validated empirically how effective it is. It should be reproducible since most of the details are included in the paper.
summary_of_the_review: This paper investigates an interesting problem - how to learn a good representation that is robust to data bias. It proposes a generalized version of contrastive losses and a regularization loss to address the problem. However, it lacks important experiments and the writing should be improved. Therefore, it still needs a lot of work before it is ready to be published.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
This paper considers the problem where certain features are extremely predictive in the training data but not predictive in the test data. They first provide a quantitative intuition of why this is a problem, and then propose a modification of supervised contrastive learning to fix this issue.


Strengths:
1. Clearly written paper on a common and important problem.
2. Good quantitative intuition motivating their approach, ablation studies and experiments
3. Good survey and contrast to existing work
4. Detailed replies and improvements to the paper in the discuss period

Weaknesses:

Better evaluation of existing baselines was mentioned by a couple of the reviewers
justification_for_why_not_higher_score: The paper gives an interesting, and well-justified, but somewhat narrow modification to the existing supervised contrastive learning framework. A spotlight would need more algorithmic novelty.
justification_for_why_not_lower_score: The paper studies a really important (but, relatively under-studied) issue of deceptively over-predictive features in training. It proposed solution is well-justified and widely applicable. The paper deserves to be published.
Decision: Accept: poster