Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a GLIPv2 model trained on vision-language grounding, where localization and vision-language understanding tasks are reframed through the lens of grounding to have a unified model. Additionally, a new inter image-text token loss is introduced which provides some performance gains. 

strengths_and_weaknesses: Strengths
- The losses and unified framework introduced in the paper are sound and an important direction for vision-language work. It is ideal to have a single model capable of performing both grounding and understanding type tasks, and making the vision-language understanding tasks more grounded for interpretability and improved performance.
- The figures are helpful, well made illustrations and there are extensive experiments for the paper’s method to be validated.
- The method does outperform its predecessor GLIP consistently, albeit small gains. But it does have large performance improvements on VL understanding tasks compared to prior work MDETR and others (e.g., PhraseCut, VQA)

Weaknesses
- While there are some performance improvements over GLIP, I don’t see significant method changes other than the inter loss. This loss does not seem to contribute that much (1-2 point improvement compared to the ablations without the loss), and the performance improvement compared to GLIP is quite small.

questions: Questions
- On L82-84 authors write “Many VL models (e.g., BUTD) (2; 58) rely on a pre-trained localization model as their visual encoder; the downside is the pro-longed “localization->VLP” pre-training pipeline (41; 48; 13; 47; 34; 32; 60; 37; 35). In contrast, GLIPv2 simplifies the pre-training pipeline and enables grounded VL understanding for better interpretability." Doesn’t GLIPv2 use image tags/bbox labels from a pre-trained detector too? Again, paper states “ triplet format (Img, Text, T)” data inputs where T are (box-label) annotations, so don’t we still start with localization to some extent?
- L183-185 “We do not propagate positives to grounding-type texts (natural sentences) because phrases in sentences carry contexts that
185 are unique to that image-sentence pair.” Out of curiosity did you try to include these as positives? If so, what happened? Seems like a thoughtful design choice.

Style and writing comments
- The writing grammar is not consistently correct throughout the paper.
- L56 “world” → word
- L78 bold section should probably be on a new line. Same with L93?
- L93: arriving *at* a…
- L128: “classier” → “classifier”
- L179: “easy” → “easily”
- L229: Figure reference is missing.
- L206: One set of weight*s*, L207 task*s*
limitations: The authors did not include limitations or negative societal impact discussion.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proposes a new VL grounding framework called GLIPv2, which unifies several localization and VL understanding tasks in a same unified interface. It shows that doing pretraining on localization data + image-text pairs with this setup improves downstream model performance on all taska and achieves SOTA performance on most of them. It inroduces an inter sample contrastive loss which improves performance. 
strengths_and_weaknesses: ### Strengths
- Proposed work improves SOTA performance on a variety of localization and vision-language understanding tasks
- The paper is well written and structured
- They present both finetuning with individual task specific heads as well as prompt tuning and using same weights for different tasks for zero shot
- The authors do thorough ablation on all the different combinations of the pretraining objectives and dataset combinations



### Weaknesses
- I am concerned about the novelty as the additional loss term is somewhat similar to well known region-word loss applied over full batch in multiple works and same setup as GLIP but showing performance on new vision+language tasks. Although this work shows better performance compared to previous works, I wonder if this is also due to additional data used for pretraining which is the localized data generated by GLIP. 
- It seems there is additional localization data generated using GLIP model that is used for training. If so, are the GLIP and GLIPv2 models trained in a teacher/student setup? In fact, how does a GLIP model perform if retrained on this additional data (kind of like self-training)?  Are the GLIP baselines in Table 1 using this additional localization data? If not, then the comparison is a bit unfair.
- Line 1184-185 is unclear to me what the authors are trying to say. I understand the explanation why Inter-image region-word loss is different from CLIP but the methodology is not very clear.
- Table 2 if there is trian-test overlap, I would suggest the authors to remove those results. Authors should have been careful about this while preparing the training dataset to deduplicate the train set against any downstream test-validation sets.
questions: - Table 4, Row 6 .. is the MLM objective trained alongside other objectives? Or is there another step of pretraining here? Appendix section 4 mentions that Row 6 has additional stage of training without MLM. I think it will be beneficial to add more analysis behind the intuition of a second stage of pretraining.
- How does the training time gets impacted with the newly introduced loss? 
- Since the authors discuss how the loss differs from CLIP, did they also compare on vision classification tasks?
limitations: Yes they have but I would encourage them to discuss in detail any potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a new V&L model, GLIPv2, that builds upon the original GLIP v1 model centered around multimodal pretraining. GLIPv2 aims to unify visual localization tasks (e.g. object detection) and uses V&L understanding tasks like visual question answering. To do so, three distinct pretraining tasks are used 1) phrase grounding, where the model computes the alignment between image regions and tokens 2) and 3) the standard masked language modeling task. Notably, task-specific classification heads are not used for pretraining. In finetuning on downstream tasks, GLIPv2 performs competitively. The model can also be used in zero-shot and prompt-tuned settings.
strengths_and_weaknesses: **Strengths**

- Similar to CLIP, GLIPv2 can perform open-vocabulary tasks because of the classification-to-matching trick that computes the dot product between the fused visual and linguistic representations. This means the model is particularly adaptable compared to many other V&L models because it can handle new and out-of-domain visual classes.
- The zero-shot and prompt-tuning experiments have impressive results and are, in and of themselves, exciting to see as tasks for a large V&L model. The ability to evaluate models on downstream tasks with either little or no parameters updates and remain comparable to full finetuning shows the model has learned a lot during pretraining.

**Weaknesses**

- GLIPv2 uses GLIP (v1) to generate bounding boxes for the unlabeled (image, text) pairs in the pretraining data versus just an off-the-shelf object detector that has not had linguistic supervision. This also clouds the data and pretraining approach a bit given how similar GLIPv2 is to GLIPv1.

- The text transformer appears to use the text transformer from CLIP including its pretrained weights (let me know if I'm misunderstanding). If it does use those weights, this skews these results because CLIP's text transformer already had visual supervision from the contrastive pretraining.

questions: *Questions*

- Mentioned in the previous section as well, is the text transformer from a pretrained CLIP model (L239)?
- What are the authors thoughts about requiring another large, multimodal model to generate pretraining data?

*Suggestions*

- About half of the prior work section provides details about GLIPv2; space can be saved by compressing some of the details of this paper. For instance L75-78, which is about half of the localization models subsection, describe the novel contributions of this work instead of prior work and how GLIPv2 differs.
- This sentence (“See GLIP (36) for more 132 details”) is essentially repeated twice in Section 3.1; one can be removed to save space and for readability.

*Typos*

- L103: output -> outputs
- caption Figure 2: compute -> computes
- L118: extract -> extracts
- L128: classier -> classifier
- L139: tyeps -> types
- L206: one set of weight -> one set of weights
- L207: task -> tasks
- L210: keep -> keeps
- L229: broken link to figure

limitations: - Section 5 is titled “Conclusion and Social Impacts” without a description of social impacts.

- Any biases learned by GLIP (v1) – for instance, having significantly lower detection accuracy for certain demographics of people in images or performing worse for images where people do not fit in the gendered roles of the training data – propagates to the pretraining data for GLIPv2 as well. This is compared to detection datasets that were hand-annotated. It’s a component of using large-scale data, and particularly using large scale data labeled by another ML model, that the authors should address this in their limitations section.
review_title: null

Meta-review: metareview: All three reviewers provided positive reviews and scores for this paper. They were happy to see the strong empirical evaluations and improvements over GLIP, impressed by the zero shot results, and found the new combination of pre-training objectives interesting. A few questions and concerns were brought up by reviewers that had to do with differentiation to the GLIP paper and model. These concerns including novelty in the loss term, tasks accomplished, need for detection boxes at training time, etc were well addressed by the authors. The reviewers also acknowledged that their questions were answered. Given these positive reviews and discussions, I recommend acceptance.

Note to authors: Please address the comments raised by the ethics reviewer in your final manuscript. Thank you.
confidence: Certain
Decision: Accept