Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a colorful prompt tuning (CPT) method for tuning pretrained vision-language models. CPT reformulates visual grounding into a fill-in-the-blank problem with color-based coreferential markers in image and text. The grounding to the target image region is achieved by recovering the corresponding color text from the masked token in the query template. Empirical studies show that CPT can outperform existing methods in promoting VL-PTMs for visual grounding in zero-shot, few-shot and fully supervised settings.
main_review: Strength:

1. The idea of connecting images and texts with the concept of color is quite neat. Color is probably one of the most prominent visual features but this seems to be the first time that it is used to connect the two modality.

2. CPT seems to tighten the gap between pretraining and fine-tuning. Previously, pretraining is a mask-language-modeling (MLM) task, while fine-tuning is a classification task that only uses the [CLS] head. Now, CPT also makes fine-tuning (prompt tuning) as a MLM task, so that it looks more similar to pretraining.

3. The studies on how the color can affect the performance is comprehensive.   


Weakness and questions:

1. The method seems to rely on the quality of an image segmentation model to get the image regions. However, there is little detail about the model. How does the segmentation affect the CPT performance?

2. The query template shown in Figure 1 requires the main body of the query text is a noun. What if it is not that case? For example, what if the the query text is sentence “The horse is standing in the yard”?

3. The paper does not talk about how to ensure the MLM label in the cross-modal prompt tuning part (Figure 1(c)) is correct or how to deal with the ambiguity. For example, in Figure 1, how do you decide the answer is red or blue? For me, the woman can be watching either of the horses, so both red and blue can be correct.

4. The empirical evaluation is only on the visual grounding task, but there are quite a few other multimodal tasks that can be evaluated on to make the paper stronger, e.g., visual question answering. Actually, the authors have pointed out that the method can potentially be applied to object detection, predicate classification and scene graph classification in Figure 5. But why don’t you just work on those tasks to show CPT’s superiority? Furthermore, even for the visual grounding task, CPT is only evaluated on RefCOCO and its variants, making the claims less convincing. Minor question: why is there no CPT-Seg result in the last row of Table 1?

summary_of_the_review: While the idea of this paper is neat, there are a few details missing. The empirical studies lack comprehensiveness to support that the proposed method can generalize. I would be more than happy to change my score if the concerns are addressed well.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a colorful prompt tuning for pre-trained vision-language models, with color masked regions and masked word tokens, on three reference tasks (refCOCOs), it shows promising results in zero-shot and few shot settings.
main_review: Strengths:
- An interesting work to try to establish the connections with words and grounding image regions.
- It shows promising results on three refCOCO tasks. 

Weakness:
There are some questions or missing parts -
1. What is the main pretrained model this work build on? VinVL? or ?

2. For these refCOCO datasets, what is the statistics for the language (text part) that covers the image regions? for example, on average, how many words in one sentence that covers the image regions? My understanding is for the high quality COCO captions, on average, there is 1~2 word objects that covers the image regions. This question may relate the motivation of this work, if the coverage is not too high, it might be quite trivial for this method.

3. Another question is - is it possible to verify this approach on a or more different task(s), rather than only refCOCO. For sure, refCOCO is a natural task for this method. 
summary_of_the_review: This is an interesting work to utilize the grounding between image regions and text words, as a prompt tuning way for pre-trained vision language models, and shows promising results on three refCOCO tasks in zero-shot and few-shot settings. One concern or question is to verify the generalization capability of this work in more broad settings or diverse tasks, or it is just task-specific for refCOCO.

Btw, I am not sure whether it is okay for section 8 & 9, which are on the page 10 (beyond 9 page limit).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel paradigm named Cross-Modal Prompting Tuning (CPT) that reformulates visual grounding into a fill-in-the-blank problem. Specifically, CPT applies a unique colorful mask to each visual region in the input image and then utilizes a pre-defined template to wrap the input text, where the modal needs to identify the color of the corresponding region that contains the described object. Experiments are conducted on RefCOCO, RefCOCO+ as well as RefCOCOg and promising results are achieved.
main_review: [Strengths]
1.	The paper leverages the correspondence of colors between visual perception and textual input learned by the pre-trained model to enable visual grounding in downstream tasks, which is novel.
2.	Extensive ablation study and discussions are involved in the paper.
3.	The proposed Cross-Modal Prompt Search mines the most sensitive colors and considers the correspondence between visual and textual semantics to determine the color set used in the prompt template, which provides some insights of prompt search.

[Weaknesses]
1.	As discussed in Section 4.5, one limitation is that the added color masks would disturb the original colors in the input images and eventually mislead the model. It is believed that such cases are common in real-world data, such as “man in red shirt” or “white cat on the couch”.
2.	Image Region Batching would bring more cost during inference.
3.	The authors should visualize and compare the results of visual grounding derived by the proposed CPT and zero-shot/fine-tuning/fully supervised models.

summary_of_the_review: Pls see the details in main review.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed CPT, colorful prompt tuning for visual grounding tasks using the pre-trained V+L model. By adding color-based co-referential markers in both image and text, CPT makes visual ground as a fill-in-the-blank problem and mitigates the gap between pre-training and fine-tuning. The experiments are conducted on three visual grounding tasks and demonstrate the effectiveness of CPT. 
main_review: Generally, CPT is an interesting, simple but effective prompt-based approach, which reformulates the problem to fit the downstream task. The experimental results and ablation studies are good and provide many insights. The paper is well written. 

However, there are a few concerns of the paper:
1. I doubt that CPT is a generalizable approach that can fit other V+L tasks. Most V+L tasks do not need segmentation level information. Maybe the author can try some tasks like VCR. But for general task such as VQA, captioning, text-image retrieval, I do not think CPT can be applied. Therefore, CPT can not be claimed as a framework for VL-PTMs. It is only for visual grounding tasks.

2. The experiments only included a few simple baselines. We know the fully fine-tuning on large V+L models definitely can not generate reasonable results. It is possible to try some lightweight models like VL-T5 [1]. 
[1] Unifying Vision-and-Language Tasks via Text Generation
summary_of_the_review: I think the paper is good and has the opportunity to be in ICLR. I would appreciate it if my concerns can be addressed in the rebuttal session.

-------------------------------------------------------------------------------------------------------------------
After reading the rebuttal, my concerns have been addressed partially. I will keep my original score.
review_title: null

Meta-review: In my opinion, this is a cool idea, but could use a few more test settings to evaluate the general applicability of their method. It would be interesting to see if the method generalizes to a non-reference based task.

Strengths:
Novel method that explores the interaction of color masks for learning to prompt about regions in images by identifying the color region they correspond to
Paper contains extensive ablation studies & discussions

Weaknesses:
Experimental results are run on uncommon benchmarks, making it difficult to compare to SOTA V+L methods
Consequently, it’s not clear that this method would generalize beyond visual grounding to tasks such as VQA or captioning
Decision: Reject