Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes UniCLIP (Unified Framework for Contrastive Language–Image Pre-training) to unify inter- and intra-modality contrastive learning under a joint pretraining objective. The main difference of UniCLIP from prior work is that UniCLIP considers all inter- and intra-modal positives/negatives in the contrastive loss. Specifically, as an example, when computing loss for a text-image pair, image-image and text-text negative pairs are also considered. A dedicated multi-positive NCE loss is used to highlight the gradient of hard positives, which is in theory and empirically better than other options such as MIL-NCE loss and Supervised Contrastive Loss. Another special design is having a separate augmentation encoder inserting augmentation information in between the image encoder and image projection head so that the image encoder is agnostic to data augmentations while augmentation’s change to image semantics is also taken into consideration in the projected embedding space.

strengths_and_weaknesses: 
Strenghts

+ The paper has done a comprehensive and rigorous discussion of available contrastive loss functions. The explanations are clear in pointing out each specific configuration’s impact in the context of image-text contrastive learning.

+ Zero-shot classification performance seems to be strong and improvement is consistent across most of the 11 tasks.


Weaknesses

- Why not compare finetuning results on image-text retrieval? Zero-shot performance is not entirely reliable for comparing cross-modal retrieval capability. In my experience, sometimes finetuning even with few-shot examples can result in a huge difference.

- I find the idea of learning discriminative features by leveraging positives/negatives both within and across modalities not drastically different from prior works. Beyond vision-language pretraining, the choice of positives/negatives, and what to include/exclude is comprehensively explored in metric learning or earlier data mining literature. It is good to know that such design works well for vision-language pretraining, but it is not an exceptionally novel contribution.

questions: * For the baselines such as SLIP and DeCLIP, did you implement your own versions and train your own models? Sorry if I missed this detail.

* Since the pretraining dataset is different from the original configuration of DeCLIP, the numbers reported in the original paper are not directly comparable to this paper. I find it a bit puzzling that in some recent reports (https://arxiv.org/pdf/2203.05796.pdf), DeCLIP outperforms SLIP by large margins while in this paper, they perform similarly. Any comments on this?

limitations: I do not see any major issues.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper propose a framework called UniCLIP to integrate the contrastive loss of both inter-domain pairs and intra-domain pairs into a single universal space for Vision-Language Contrastive Pre-training. This work is more like a fusion of three optimizations than other work: augmentation12 aware feature embedding, (2) MP-NCE loss, and (3) domain dependent similarity measure.
The results of UniCLIP performs better than previous work.  
strengths_and_weaknesses: Strengths:
Good performance and simple
This paper is will written and organized.
Weaknesses:
Lack of innovation, this paper is more like a combinatorial work, (2) MP-NCE loss, and (3) domain dependent similarity measure These two are not original to this paper, and (1) data augmentation is also a very common technique.
questions: N/A
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes learning a CLIP model in a single representation space to overcome the limitations of previous work on independent learning of each self-supervised objectives. The model, which they call UniCLIP, integrates the contrastive loss of both inter-domain pairs and intra-domain pairs into a single universal space. The authors utilize three methods to address the discrepancy between different domains: augmentatio-aware feature embedding, an MP-NCE loss, and a domain-dependent similarity measure. 


strengths_and_weaknesses: Strengths:
(1) The paper is well-written.
(2) Experimental settings are described in detail, aiding reproducibility.
(3) The proposed method leads to significant improvements on zero-shot, linear probing and retrieval test.
(4) Strong ablation studies.

Weaknesses:
(1) This work builds incrementally on previous works. 
(2) Experiments on a wider range of tasks are encouraged, e.g., image captioning and visual question answering, as the method claims to learn representations in a unified space, improvements on visual-language tasks are expected.
(3) Lack of proper discussion of limitations.
(4) Some of the claims are not backed by experiments, for instance, the claim that UniCLIP can be used in various types of multimodal datasets.
(5) The paper has several typos (e.g., "datsets")
questions: 1) A detailed discussion of limitations is needed.
2) Need experimental validation of the claim that UniCLIP can be used in various types of multimodal datasets. How can your claim be justified with such experiments?

limitations: The authors do not properly address the limitations of their work. I encourage the author to directly address that in their rebuttal.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this work, the authors propose three components for contrastive learning across modalities which aim to remove discrepancies associated with learning representations with data augmentations, "hard" and "easy" positive pairs, and differences in similarity scoring across domains. 

The first contribution is augmentation-aware feature embedding which concatenates an augmentation feature vector to image encodings before being fed through a projection head. This augmentation feature vector is used in order to constrain the image encoder to output representations that are "augmentation-agnostic" while allowing the projection head to make use of augmentation information. 

The second contribution is a modified NCE loss (MP-NCE) which includes multiple positives and also uses a "strong positive" pair of an item with itself. Additionally, a rebalancing parameter is utilized to ensure that losses are equally weighted across all combinations of domains. 

The third contribution is a domain dependent similarity measure which reweights similarly scores by introducing separate thresholds $b_{i,j}$  and temperature parameters $\tau_{i,j}$ for each pair of domains $i,j$.
strengths_and_weaknesses: Strengths:

- The paper is well written and provides clear descriptions of each of the contributions as well as experimental results on well-studied datasets/benchmark models. 

- Each of the contributions is well motivated by shortcomings in existing contrastive models/losses.

- Experimental results show clear improvements over recent works (CLIP, SLIP, DeCLIP)

- The experiment settings and hyperparameters are clearly outlined in Appendix Section C

Weaknesses:

- Exposition and presentation of each of the contributions ability to solve the problems they address is somewhat limited although proposed components are shown to be beneficial through ablation studies.. Some examples of the augmentation aware feature embedding is provided in Appendix Section E but it is unclear how the MP-NCE and the domain dependent similarity measure change the representations learned and their similarities. It would be helpful to understand how distribution of similarities changes with these mechanisms. 

- Ethical considerations and limitations is not discusses, please consider including information on this material

questions: Please see Weakness 1 and 2
limitations: The authors have not adequately addressed the limitations and potential negative societal impact of their work and should do so in a final version of the paper.
review_title: null

Meta-review: metareview: This paper proposed a framework for image-language pretraining which include three techniques.  The techniques are well motivated and developed to achieve better performance on zero-shot, linear probing and retrieval test.  Ablation studies are provided to show the effectiveness of the three techniques.  The writing of the paper is clear.  The authors appropriately answered most of the questions of the reviewers.
confidence: Certain
Decision: Accept