Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes the task of Weakly-Supervised Open-World Phrase-Grounding, which seeks to first generate captions for local regions and then ground the caption. Meanwhile, the paper proposes a way to train a mask generator given image and text with only weak supervision (based on CLIP). 
strengths_and_weaknesses: Pros:
+ The paper studies a new task and proposes a sensible baseline to this task.
+ The paper contributes a way to train a mask generator with only weak supervision with decent performance on benchmark (e.g., Flickr30K), which is "somewhat" novel to my knowledge.

Cons:
- The newly proposed task is not particularly novel or challenging. It seems like a direct combination of image-caption (region-based) + localization. I would have liked to see some discussion on the unique challenge this task poses and how solving it might benefit real-world needs or other tasks.
- The method to the newly proposed task is also a simple combination of previously methods: selective search for proposing regions; BLIP for generating the captions given a region; a localization network.  
- The paper also proposes a way to train a mask generator given image and text with only weak supervision (based on CLIP). However, the paper does not come with a discussion on the novelty of the proposed method and how it compares to previous weakly-supervised methods for localization. The idea of using a "classification" model's heatmap for localization is not particularly new so I would appreciate a detailed discussion.
- The paper is a bit confusing on its core contribution: it lacks a full discussion on the uniqueness of the newly proposed task if the task if the core contribution; the method of training the mask generator lacks a discussion to prior work so it is also hard to judge the contribution of the method part.

Minor points:

Providing more backgrounds on BLIP would be helpful.

MDETR should be referenced.

GLIP [42] does not use CLIP initialization.


questions: See cons.
limitations: N/A.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes and studies a new task, which localizes image regions with masks and describes them with natural language. The paper studies the task in a weakly-supervised open-world setting. In order to tackle the task, two recently proposed pre-trained vision-language models, namely, CLIP and BLIP are leveraged to produce multi-modal matching score and generate candidate captions, respectively. Meanwhile, an encoder-decoder network is trained to generate foreground mask for grounding. Superior performances are shown in multiple benchmark datasets, for the three evaluation tasks.
strengths_and_weaknesses: Strengths:

1. A new perspective for vision-language community, and open-world applications. Comprehensive techniques are proposed with extensive supportive experiments.
2. State-of-the-art quantitative results on multiple benchmark datasets for the three tasks (WSOL, WSG, WWbL).

Concerns:

1. Regarding novelty. The novelty might not reach the standard of NeurIPS conference. The proposed framework largely depends on CLIP-like pre-trained models and lacks original innovation. Moreover, the proposal claims to be "purely visual" (without textual input), but actually the textual information comes from pre-trained BLIP model at inference. I expect more discussion regarding the novelty.
2. Regarding empirical results. I am not quite sure whether it is fair to compare with previous methods, considering the proposed method actually leverages the pre-trained information from CLIP and BLIP. And I am also not sure if the datasets (e.g., Flicker, ReferIt) used for comparison have overlapping (or similar) images with the large-scale datasets involved in the pre-trained models (i.e., CLIP and BLIP). The comparison fairness should be discussed.
3. The writing quality and presentation should be improved. The writing and expression highly affect deeper understanding of this work. There are quite many expression issues and typos throughout the whole paper. Some of the typos are listed as follows.
(1) L50, "A two-stage inference time procedure" shoud be "A two-stage inference procedure". (2) L69, "during the inference test" should be "during the inference". (3) L197, "a encoder-decoder architecture" should be "an encoder-decoder architecture". (4) L241, "An SGD optimizer" should be "A SGD optimizer". 
questions: 1. Discuss the novelty and elaborate the contributions (Concern #1).
2. Discuss the empirical fairness (Concern #2).
limitations: The authors have adequately addressed the limitations of their work.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors explore using pre-trained CLIP for three tasks: (1) object localization, (2) phrase grounding and (3) WWbL: generating object masks and corresponding captions for an image. Their model encodes both image and text with VGG and CLIP respectively, and generates a heat map. They use two weakly supervised losses: (1) foreground loss which encourages the masked image with the heat map to match the text input, (2) background loss which encourages the remaining image to not match the text input, (3) relevancy heatmap which measures the difference between the generated heat map and the relevancy heatmap of CLIP, and (4) regularization loss for the generated heat map. Performing inference on (1) and (2) are trivial, whereas for (3) they designed an algorithm which first unsupervisedly propose region, and then get captions for them to convert the problem to a phrase grounding task. 

Object localization results on CUB, and phrase ground results on VG, Flickr and ReferIt show that their model is better than previous methods, and each of the four losses plays a role in improving the results. On their proposed new task, WWbL, the results are also in favor of their proposed model. Combining all of the results, this paper found that CLIP can be useful in various language related object segmentation tasks. 

strengths_and_weaknesses: Strengths:

It is natural to use a pre-trained vision-language model for object detection/localization. Straightforward and simple solutions like [1], have been proposed and could perform reasonably well in a zero-shot manner. This paper is also a neat method, based on the assumption that objects to localize have higher similarity to the text in CLIP embeddings, compared to background. With the four simple and intuitive losses, the authors show their model performs better than the previous methods.
The authors also contribute a new task, WWbL.

[1] https://github.com/shonenkov/CLIP-ODS

Weaknesses:

1. The model may fail for ambiguous captions. For example, given an image of four apples, if the text is “an apple”, the desired output mask should be one of the apples for a deterministic model. Since the foreground loss and background loss maximizes the difference between the image representation along the text embedding dimension, one of the most likely results is that all of the apples are in the foreground (if CLIP “thinks” four apples to no apples is more different than one apple and three in terms of the caption “an apple”). This means that even for a perfect pre-trained model, such loss may still result in undesired output given ambiguous captions. 

2. Related to 1, this paper lacks a qualitative study of the effects of coefficient. It is shown in this paper that all of the losses are important for the metrics, but would different configurations have different tendencies? Can configurations be optimized for different applications? This might be out of scope, but if the authors had provided insights on this, this paper would have been much stronger. 

questions: Please refer to the weakness section above. 
limitations: It is nice of the authors to point out the limitation that their work only applies to single images. It would also be nice if the authors can also talk about the potential ethical impact of their paper, e.g. would the bias in the pre-trained models be reinforced or reduced in this method?
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a new solution for weakly-supervised open-world phrase grounding. The method is built upon the off-the-shelf CLIP model and captioning model, only taking images as the input. The experiments are conducted on several datasets, showing the superior performance of the proposed solution. 
strengths_and_weaknesses: Strengths:
1. The method is simple yet effective. By combining strong pretrained model and off-the-shelf model, it achieves good performance without text inputs. 

2. The empirical results are several benchmarks are promising. 

Weaknesses:
1. The novelty of the paper is quite limited. Most of the components are not developed by the paper. 

2. It is not clear how this method can perform well in such a weakly-supervised setting. Its generalization ability may be contributed by the powerful pretrained model (CLIP). The author should performant some ablation study to change these models to other candidates to validate this. 

3. The paper ignored some work related to pretrained model for zero/few shot settings (with prompt, etc) [a,b]
a. A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models. ACL 2022.
b. CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models
c. SimVLM: Simple Visual Language Model Pretraining with Weak Supervision. ICLR 2022.
questions: 1. Have the authors tried to change the CLIP model and BLIP to other models to see how the new alternative can perform? 

2. Have the authors compare the model with some pre-trained methods like CPT-Colorful Prompt Tuning, Frozen or Flamingo?
limitations: How can the proposed solution perform if there are no strong pre-trained models available?
review_title: null

Meta-review: metareview: The paper presents a new approach, using two pre-trained models (CLIP and BLIP) as supervision to enable three tasks, including the newly proposed task WWbL, which is a joint open vocabulary description and grounding/localization task trained only with weak supervision.

I recommend acceptance based on the revised paper, the reviewer's comments, and the author response. I think the paper sufficiently contributes:
- Overall idea and architecture
- The WWbL task, even if similar to previous task
- Extensive experimental evaluation and comparison to prior work
- Solid ablation study

The paper received mixed review scores with 2 Borderline rejects, 1 Borderline accept, and 1 strong accept.
The authors have in my opinion largely addressed the concerns and revised the paper, one of the remaining concerns of the weak reject reviewers is novelty, which I think is sufficient.

My recommendation for acceptance is under expectation that the authors revise the paper to address any outstanding points made by reviewers, e.g.
- additional alternative models (reviewer MVej) if possible

Additionally, I think it would be great if the authors discuss the relation ship of WWbL to to the task of dense captioning task more clearly in the paper.
confidence: Less certain
Decision: Accept