Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper addresses the problem of vision-language pretraining, whose goal is to learn a strong backbone network transferable into downstream tasks such as object detection or visual grounding.
Thus, the paper proposes a framework for aligning between visual and textual tokens during training motivated by Shapley value from game theory.
Specifically, the model computes the Shapley values to determine the interaction between related image patches, which form objects or related image regions and noun phrases corresponding to object types in images.
The Shapley values for patches in image regions and pairs of image regions and noun phrases are used as soft pseudo-labels to train the alignment between textual and visual modalities.
The paper also introduces an approximation mechanism to reduce the complexity of Shapley value computation via sampling.
The paper evaluates the performances of Image-Text Retrieval, Object Detection, and Visual Grounding on MSCOCO and Flickr30k datasets.

------- After rebuttal -----

The rebuttal sufficiently addresses my concern as well as confirming my understanding.
Thus, I keep my rating and recommend the paper for acceptance.
strengths_and_weaknesses: #### Strength:
+ The formulation of semantic generation as well as semantic alignment under Shapley interaction is interesting and novel. Moreover, viewing weakly supervised visual-textual alignment in terms of game theory is a promising direction to explore.
+ Learning to predict Shapley more, as well as its uncertainty, seems to significantly reduce the training complexity.
+ The paper compares with appropriate baselines to demonstrate its effectiveness.

#### Weakness:
+ The technical section could be challenging to follow, especially for computer vision audiences without prior knowledge of Shapley value. The reviewer believes it would be very helpful if the paper could include a table to highlight key differences of the proposed work with various methods used for vision-language pretraining. If the reviewer understands correctly, the main distinction compared to other works would be the proposal of using Shapley interaction as soft pseudo labels for fine-grained image regions instead of using contrastive learning between image-caption pairs. However, this is not clear and emphasized enough in the paper.  
+ The reviewer is unclear how Shapley value could be used as soft pseudo labels. For example, in line 180, the paper mentions that the game score $v_1$, which is used to compute the Shapley interaction, is the similarity between image and text. However, it appears that the paper does not mention any training phase for learning image-textual similarity. Then how would this similarity be computed correctly? Are there any pretraining phases or pretrained models employed for similarity computation? 
+ The paper lacks convincing motivation on why using Shapley value instead of token-wise alignment (FILIP [42]) is beneficial. Although in lines 97-99, the paper mentions that FILIP has quadratic complexity, the proposed method also suffers from combinatorial complexity, which could be worse. 
+ Although the idea of directly learning to approximate the Shapley value is interesting, it is unclear from the paper how the module can learn to approximate Shapley interaction and estimate its uncertainty. Specifically, what is the input of this module? Is it the set of all visual+textual tokens? Is the uncertainty estimation reliable as neural networks are notorious for being very confident in their predictions?
questions: + Please refer to the weakness section.

limitations: + I believe the work doesn't seem to have any potential negative societal impact.
+ It would be interesting to investigate the bias from web data in future directions. Would the model propagate these biases from captioned images toward downstream tasks?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a fine-grained semantically aligned vision-language pre-training framework. Both visual token-level semantics alignment and phrase-region level semantics alignment are studied. To be more specific, the paper measures the shapley interaction of visual regions to text as a supervision signal for region generation module and visual backbone. Then the alignment between regions and phrases is also supervised by Shapley interaction between them. Further, an uncertainty-aware learning module is introduced to predict Shapley interaction to save the time cost of sampling-based estimation. 
strengths_and_weaknesses: Strengths:
1. The methodology is interesting and novel. It's essential to exploit the local correlation between image and text in contrastive pre-training. The idea of introducing Shapley interaction as supervision seems to fit the problem well.
2. LOUPE can outperform recent works trained with a similar scale of data in retrieval tasks. And the ablation is comprehensive.
3. The region generation model can also be used as a zero-shot object detector, which is quite impressive.

Weaknesses:
1. The evaluations on zero-shot image classification and linear probing are missing. Those are important to prove whether the learned visual backbone is strong and robust.  In CLIP, they reported the above two evaluations in 20+ datasets to prove the transferability of their visual backbone. 
2. The training time needed is increased by 65% (1.17->1.93). 
3. The dataset used seems to be a private dataset and it's hard to convince people that the comparison is fair enough. Although the scale of the proposed dataset is smaller, the quality is not compared and it matters a lot in CLIP-like pretraining. Also, it's not indicated whether the dataset will be released or not. If not, it's hard for people to reproduce this work. 

================================
After rebuttal, weakness 1 and 2 have been alleviated. So I change the score.
questions: Please refer to weaknesses.
limitations: It's not guaranteed that the dataset they use doesn't contain unsuitable images, text, or personal information.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper propose a fine-grained semantically aligned vision-language pre-training framework（LOUPE） from game-theoretic interactions. Experiments on image-text retrieval, object detection and visual grounding tasks demonstrate the effectiveness of LOUPE. 

strengths_and_weaknesses: 1. The motivation is clear. LOUPE propose Phrase-Region Semantic Alignment to achieve fine-grained vision-language pre-training.

2. The hybrid Shapley interaction learning strategy is interesting for me.
questions: 1. The experiments are not sufficient. The model structures（both image encoder and text encoder) of different methods in Tabel 1 and Table 2 are inconsistent and the comparison in unfair.

2. Can plain Dual-Encoder be used for zero-shot classification? How is the result on ImageNet dataset?

3. Why are the loss weights of $L_{CMC}$, $L_{TSA}$ and $L_{FSA}$ set to 1:1:1?

4. Are public datasets, such as CC12M, included in the 240M dataset? In addition to the description part in Supplementary Material E, it is hoped that a more detailed introduction to the 240M dataset can be provided.
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper focuses on learning fine-grained vision-language alignment under the widely used contrastive-based pre-training paradigm. The authors first point out that previous pre-training frameworks, *e.g.,* CLIP, only models global image-text alignment while neglecting the fine-grained semantic features. Alignments though can be achieved by interacting between modalities with the help of certain mechanisms like cross-attention, it lacks explicit and strong supervision to encourage meaningful patch-phrase correspondence, which proved to be quite useful for reasoning downstream tasks. The authors address the limitation by explicitly matching phrases in the text with regions of patches in the image. Different from previous approaches, the authors design light modules to achieve patch-to-patch and region-to-phrase alignment by introducing supervisions derived based on the Shapley value in game theory. Practically, Shapley interaction values are used as the criteria to estimate whether two patches belong to the same region and whether an image region corresponds to a certain phrase. In addition, to address the training efficiency problem when computing Shapley values, an additional estimator is adopted to predict the true values. Extensive experiments show the zero-shot ability of the proposed model on retrieval tasks, detection tasks, and visual grounding tasks. Ablative studies on the new losses and Shapley value approximation module are conducted. Limitations and social impacts are also discussed.

Overall, this paper shares an interesting idea by incorporating game theory mechanisms in the process of text/image alignments. This kind of supervision is free of pre-trained detection/segmentation models and does not require sophisticated text/image labels. The proposed model shows significant improvements on the zero-shot transfer ability and outperforms baseline approaches by a large margin.
strengths_and_weaknesses: + Strengths
  + This paper is well organized and easy to follow. The authors first give a brief introduction on the concept of Shapley values and Shapley interaction values and elaborate on how it works under the framework of multi-modal pre-training. The derivation seems to be sound and easy to understand. The main results and ablative studies help to better understand the proposed model.
  + The idea in this paper is quite interesting and seems novel to me. The idea of aligning image regions with text phrases has been explored in previous works, while most of them rely on additional label information and fail to provide strong supervision. Introducing Shapley value in the aligning process sounds reasonable and practically shows promising results.
 
+ Weaknesses
  + Some missing related works. Similar to this paper, there are also works that attempt to address the problem of missing fine-grained information. For example, RegionCLIP[1] also finds regions in the image and aligns them with text phrases. Off-the-shelf language parsers are similarly considered. Also, RegionCLIP is able to perform zero-shot object detection tasks. Another work, X-VLM[2], shares a similar insight. I believe these works are more related to the paper, and a detailed discussion on the connections and differences with these works is suggested.
  + Training efficiency. I understand that the authors use a sub-network to predict the real Shapley value to save excessive computation costs. Nevertheless, the training time in table 3 (comparing line 1 and line 4) shows that the proposed approach still involves 60~70 training time. This can be a great burden, especially for large-scale pre-training.
  + Please see the **Questions** section for some detailed concerns.

[1] Zhong, Yiwu, et al. "Regionclip: Region-based language-image pretraining." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
[2] Zeng, Yan, Xinsong Zhang, and Hang Li. "Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts." arXiv preprint arXiv:2111.08276 (2021).
questions: I have stated my major concerns in the **Weaknesses** section. The following are some detailed questions and suggestions.

+ How is the score function in the game defined? Does it need to be carefully designed?
+ The alignment between image regions and text phrases implicitly involves the concept of object. This may lead to similar limitations to the previous works that relied on pre-trained object detectors, and hard to generalize to object-free inputs. Also, according to line185, all regions are constrained within a scale of K patches. I wonder how does the model deal with objects (concepts) of different scale, *e.g.,* apple and sky?
+ Line 230. How long do we need to train the UNSIL module?
+ I am confused about Eq.(12). The authors claim the loss function is derived from the regression loss function. Could the authors provide a detailed derivation process? From my perspective the second term $\beta_2 \ \sigma$ should be $\beta_2 \ {\rm log}(\sigma)$.
+ I am also concerned about the training time shown in table 3. It seems like the additional cost is still huge even with the approximation module. I wonder if the authors can provide a training cost-performance trade-off compared with other works? Some qualitative analysis would also do the trick.
+ If possible, I would like to see a comparison between the proposed model and some missing related works, like RegionCLIP and X-VLM that were mentioned above. These approaches share similar insight and can also perform zero-shot transfer to several downstream tasks.
limitations: The authors have adequately addressed the limitations and social impacts in the paper.
review_title: null

Meta-review: metareview: This paper addressed the fine-grained visual-language alignment from the perspective of game-theoretic interactions. It received diverse scores with three weak accept and one week reject. The technical novelty is acknowledged by all reviewers. The initial reviews raised concerns about unclear explanations of Shapley, insufficient experiments, and comparison. During the discussion, the authors have addressed most of the concerns by adding tables to compare with existing works,  adding zero-shot classification experiments in 11 datasets, and sufficient discussion with previous works. Reviewer H9ch mentions possible unfair comparison. The authors have reimplemented the CLIP with a fair encoder and the performances have been consistently improved. The meta-reviewers thus suggest accepting this paper. However, as highlighted by most reviewers, the method is too complex and hard to make readers easily follow, especially the Shapley part. It is strongly encouraged to revise the camera-ready paper. And the proposed 240M dataset is better released to facilitate future research to follow and compare, and explained about filter strategy and possible ethical issues, as addressed by two reviewers and promised by the authors. 
confidence: Certain
Decision: Accept