Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper studies the classification group robustness of large-scale image-text pre-trained models. The authors identified that, under the prompt-based zero-shot classification protocol, the worse-group classification accuracy is considerably lower than the average group classification accuracy. The authors proposed to use a metric-learning loss (i.e., contrastive loss), on top of a MLP adapter from previous works, for finetuning on target datasets. Experimental results on a few benchmarks show improvement over baselines on worse-category classification accuracy.
strengths_and_weaknesses: The paper studies a few interesting questions regarding these large-scale pre-trained models: 1) how to adapt these models for a target dataset, and 2) how do these models actually transfer across datasets and groups. These questions are important and they need to be addressed because the practice, and the hope of the community, is to rely more and more on the pre-trained models. The paper shows that neither a naive version of the zero-shot, nor linear probing on the target dataset, is the optimal choice. Benchmarked on standard datasets, The proposed method is effective across model architectures.

On the other hand, I am curious about a few baselines that I feel essential in evaluating the pre-trained models but seem to be missing from the draft. The first one is more prompt-tuning for zero-shot classification. Although impressed by the CLIP zero-shot classification accuracy, one must recognize the necessity of using properly tuned prompts or hints, as pointed out by the authors of the CLIP paper. It can be argued for the downstream evaluation/benchmarks in this paper such as WaterBirds, we can use prompts like "A land bird, on water background" to give more hints to the base model. I didn't find a discussion on how the zero-shot baseline is constructed or if meticulous tuning was conducted, and I believe this procedure is necessary for reliable conclusions.

Secondly, the metric-learning term (i.e., the contrastive loss) that is at the center of the proposed method looks heavily inspired by the hard-negative sampling strategy (see Line 235), acknowledged by the authors as well in Line 227. A potential ablation study on decoupling this hard-negative sampling procedure from the more complicated contrastive loss, in my opinion, is essential for understanding if the sampling procedure and/or the contrastive term is (more) important in the proposed method. If it is the sampling procedure that works, we don't necessarily need the embedding-based loss and it would be simple.

Thirdly, I can imagine a simple two-stage baseline that has the same essence of negative-sampling procedure. In the first stage, one can train a classifier using the naive sampling strategy. In the second stage, a new classifier can focus more on the misclassified samples, like a cascade classification framework. With a large-scale pre-trained model such as the CLIP model, the first stage can be its native zero-shot classifier. If interpreted this way, the sampling strategy proposed by the authors is is akin to the two-stage framework. Upon searching, a paper named *Just Train Twice: Improving Group Robustness without Training Group Information* does this for group robustness and it seems to be a valid baseline that should be studied in the paper.

Lastly, I'd like to raise my concern for using the term "foundation model" in an official and scientific research conference, despite it didn't play any role in my judgement for reviewing the submission. The practice of grouping a set of large-scale pre-trained models, large-scale being the amount of data, and/or the number of parameters in the model, and terming them "foundation" is subjective and without scientific background. Personally, I work on scaling models as well, and I'm a firm believer in the potential of these big models for the future of AI and ML research. It is precisely that the first-hand experience that makes me realize the flaws in this term. These models aren't foundational yet. They make silly mistakes all the time. They don't do grounding well. And they are far from reaching the ideal performance that we hope to get eventually, as evident in the experimental results of the submission as well. "Foundation model" is a hope, an inspiration, for now, not a reality. And a research paper is, by its nature, about the presence. Therefore in my humble opinion discarding the term will make the paper a stronger submission.
questions: In the author response phase I encourage the authors to respond to the suggested baselines in my comments, with evidence of why they might be inferior or unnecessary. 
limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the group robustness of foundation models, and shows that large scale pretrained foundation models can still have a bad robustness on some group shifts. Then the paper proposed a contrastive adapting method that take advantage of supervised contrastive learning to train the adapter to improve the alignment and cosine similarity metrics. Experimental results show that the proposed contrastive adapter is successful in improving the group robustness of the foundation models.


strengths_and_weaknesses: 
S1: The problem this paper tries to tackle is practically important, as the foundation model is used in more applications, the robustness of these models is important to study.

S2: The performance of proposed method seems to be improving over the baseline by a large margin.

W1: Some of the experimental results are not very clearly explained, see Questions.

W2: The supplementary material is not properly anonymized, the `.git` folder in the code submission contains a commit message that have the author's name on it.


Note that most of this review is completed before I unwillingly find out the identity of the author.

questions: 
Q1: In fig 3, we can see that only on CelebA that the learned adapter has a worse group robustness, but as in tab 2, the group shift in CelebA and Waterbirds are the same (Confounder), why would the robustness of learned adapter on waterbirds be better than zero shot while on CelebA it's the other way around? Is something other than the group shift influencing the robustness of the model?

Q2: It is known that supervised contrastive learning (SupCon) outperforms regular supervised training with CE loss. It would be good if the paper could provide an analysis of how does SupCon helps the group robustness of foundation models, could it be that the performance gain is because SupCon improves the performance of the model overall (including improvement in IID performance), and the improvement in group robustness is a side effect? 

Q3: If I understood correctly, the proposed solution only tunes the adapter on the feature from the image encoder of the CLIP model, I am wondering how this method compare to the visual-language adapters like in [R1] or [R2].

Q4: The proposed method is quite similar to the ones proposed in [57], both uses supervised contrastive to help the group robustness, I would like to see a comparison between the proposed method in this paper and the one in [57].


[R1] Learning to Prompt for Vision-Language Models, arXiv 2021.

[R2] Conditional Prompt Learning for Vision-Language Models, CVPR 2022.

limitations: 
1. This work proposed a way to improve the foundation model's performance wrt to group robustness, however, the method still needs to run inference on the entire foundation model which can be computational expensive.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work proposes a new method to improve group (subpopulation) robustness in foundational models like transformers. The problem of group robustness (in terms of worst-group performance) is particularly challenging when some groups are underrepresented in the training data. Classical approaches use under/over-sampling, reweighting, and two-phase training to deal with this issue. The authors propose to use contrastive adapters without retraining from scratch. The idea is to contrast sample embeddings from the same class to improve the embedding structure. The paper provides experimental evidence that such an approach is effective when dealing with different kinds of shifts in the data, obtaining good performance.
strengths_and_weaknesses: strengths

* The paper is well written and easy to follow with a clear goal (improving group robustness in foundational models).
* The problem tackled is relevant for the use and deployment of foundational models.
* The idea of "contrastive adapter" is novel.
* Method easy to implement and to apply to a large variety of scenarios.

weaknesses

* The idea of increasing the adapter contrasting not only class and sample but also samples from the same class is incremental.
* Results do not clearly show that contrastive adapters work better (in terms of robustness) with all the considered distribution shifts in Table 3. In particular, standard adapters seem to work better when increasing the dataset complexity (number of classes in CIFAR10).
questions: * Regarding the contrastive adapter name: isn't the (standard) adapter contrastive too if we follow your argument in 4.1 and Eq. 5? 

* In Table 2) and Table 3) is the (standard) adapter trained by ERM or using the InfoNCE loss in Eq.5?
    * If you use ERM, I would like to see the adapter performance trained explicitly with Eq 5.
    * Additionally, it would be interesting to see the adapter performance training with Eq 5 only, Eq 6 only, and both equations.

* How large is P for Eq.6? Did you try to use different values for P? 

* How do you mix CIFAR10 and CIFAR02?
limitations: * The method seems to work well only with a few classes (binary classification), and performance deteriorates for different shifts and more complex datasets like CIFAR10.02. In particular, increasing the number of classes, the standard adapter and the linear probe work better than the contrastive adapter (BREEDS and CIFAR in Table 3). This is not the behavior I would expect if the argument in 4.1 is correct.

* Eq 5 and Eq 6 are both contrastive losses. You link the ERM adapter with the InfoNCE but did not use Eq 5 to train the standard adapter (as far as I understand). So this connection is not backed by experiments.

* Looking at Table 2, I agree that all the baselines fail to improve robustness consistently. But so your method in Table 3.

* I would compare your approach with JM1 (https://openreview.net/forum?id=HI2ilxFli0W): both methods have access to fine-grained label structure. The "retraining" in JM1 starts from a pre-trained foundational model.
review_title: null

Meta-review: metareview: This paper received unanimous recommendations of acceptance from the reviewer. The authors did a good job addressing concerns from the reviewers, especially with the additional ablation studies to decouple the gains from other techniques such as SupCon. The AC agrees with the reviewers regarding the contribution of this paper and recommends acceptance.
confidence: Certain
Decision: Accept