CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models
Authors: Yuan Yao
Ao Zhang
Zhengyan Zhang
Zhiyuan Liu
Tat-Seng Chua
Maosong Sun
Venue: ICLR 2022 Submitted (2022)
Keywords: Pretrained Vision-language Models
Prompt Tuning
Visual Grounding

Abstract: Pre-Trained Vision-Language Models (VL-PTMs) have shown promising capabilities in grounding natural language in image data, facilitating a broad variety of cross-modal tasks. However, we note that there exists a significant gap between the objective forms of model pre-training and fine-tuning, resulting in a need for large amounts of labeled data to stimulate the visual grounding capability of VL-PTMs for downstream tasks. To address the challenge, we present Cross-modal Prompt Tuning (CPT, alternatively, Colorful Prompt Tuning), a novel paradigm for tuning VL-PTMs, which reformulates visual grounding into a fill-in-the-blank problem with color-based co-referential markers in image and text, maximally mitigating the gap. In this way, CPT enables strong few-shot and even zero-shot visual grounding capabilities of VL-PTMs. Comprehensive experimental results show that the prompt-tuned VL-PTMs outperform their fine-tuned counterparts by a large margin (e.g., 17.3% absolute accuracy improvement, and 73.8% relative standard deviation reduction on average with one shot in RefCOCO evaluation). All the data and codes will be available to facilitate future research.

Under review as a conference paper at ICLR 2022
CPT: COLORFUL PROMPT TUNING FOR PRE-TRAINED VISION-LANGUAGE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Tre-Trained Vision-Langure Models (VL-PTMs) have shown promising capa- bilities in grounding natural language in image data, facilitating a broad variety of cross-modal tasks. However, we note that there exists significant gap between the objective forms of model pre-training and fine-tuning, resulting need for large amounts of labeled data to stimulate the visual grounding capability of VL- PTMs for downstream tasks. To address the challenge, we present Cross-modal Prompt Tuning (CPT, alternatively, Colorful Prompt Tuning), novel paradign ga for tuning VL-PTMs, which reformulates visual grounding into a fill-in-the- blank problem with color-based co-referential markers in image and text, maximally mitigating the gap. In this way, CPT enables strong few-shot and even zero-shot visual grounding capabilities of VL-PTMs. Comprehensive experimental results show that the prompt- tundd VL-PTMs outperform their fine-tuned counterparts by large margin (e.g.. 17.3% absolute accuracy improvement, and 73.8% relative standard deviation reduction on average with one shot in RefCOCO evaluation). All the data and codes will be available facilitate future research.
INTRODUCTION
Grounding natural language in fine-grained image regions is essential for broad variety of vision- language tasks, such as robotic navigation (Tellex et al. 2011 Anderson et al., (0188)) visual ques- tion answering (Antol al. 2015] et al. 20) visual dialogue (Das et al. (017). and vi- sual commonsense reasoning (Zellers et al. 2019). ). Recently Pre-Trained Vision-Languuae Models (VI.-PTMs) have shown promising capabilities in visual grounding. Typically, generic cross-modal representations are first pre-trained on large-scale image-caption data in a self-supervised fashion, and then fine-tuned to adapt to downstream tasks (Lu et al. 2019; Su et al. 2019 Li et al. 2020 Radford et al., 2021). This pre-training siriiinggtten--iee -tee--fin--tuiing paradigm of VL-PTMS Ms has greatly pushed forward the state-of oftte-o--the-asd hettartoo of many cross- cross-moda modal tasks.
Despite the success, note that there exists significant gap between the objective forms of pre- training and fine-tuning of VL-PTMs. As illustrated in Figure during pre-training, most VL-PTMs optimized based the masked language modeling objective, trying to recover the masked token from the cross-modal context. However, during fine-tuning, downstream tasks are usually conducted by classifying unmasked token representations into semantic labels, where task-specific parameters are typically introduced. The gap hinders the effective adaptation of VL-PTMs to downstream tasks. As result, large amount of labeled data is typically required stimulate the visual grounding capabilities of VL-PTMs for downstream tasks.
In this work, inspired by recent progress in pre- pre-trained language models in natural language pro- cessing (Brown et al. 2020, [Shickk & Schütze 2021a: Liu et al.) 202Ip, we present Cross-modal Prompt Tuning (CPT, alternatively, Colorful Prompt Tuning). novel paradigm for tuning VL-- PTMs. The key insight is that by adding color-based co-referential markers in both image and text, visual grounding can be reformulated into a fill-in-the-bee -te--blank problem, maximally mitigating the gap between pre-training and fine-tuning, As shown in Figure sions in image data, CPT consists of two components: (1) a visual sub-prompe that uniquely marks image regions with colored blocks or segmentation masks, and (2) textual sub-prompt that puts the query text into a color- based query template. Explicit grounding to the target image region can then be achieved by recovering the corresponding color text from the masked token in the query
ground natural language expres-

---

Under review as a conference paper at ICLR 2022
watching rasting
LIVER
tousl
(al Pre-training
CIS
licel presitin
label: negative
Label idoace
DVV
(CLS
The
horee
sutched
woman
Besad
(b) Fine teeetttunngg
blue
Head
Inc
ICLSi
Tacraa
nomen
opler
Image Rugions
Query
Quary Temp
Quwry Twect
Turing agproach)
The house watched by the woman
Figure Illustration of (a) pre-training for VL-PTMs with masked language modeling (MLM) head, (b) vanilla fine-tuning with new classification (CLS) head, and (c) our colorful cross-modal prompt tuning (CPT) framework that reformulates visual grounding into a fill-in-the- All-in-the-bbank problem with reused MLM head. Only square parts of relevant image regions are shown for illustration.
template. In addition, we present a principled method to search for high-quality cross-modal prompt configurations (i.e., visual appearances and texts of colors) for CPT.
By mitigating the gap from pre-training. CPT enables strong few-shot and even zero-shot visual grounding capabilities of VL-PTMs. Experimental results show that the prompt-tuned VL-PTMs outperform their fine-tuned counterparts by a large margin. For example, using colored blocks as visual sub-prompts, CPT achieves 17.3% absolute accuracy improvement, and 73.8% relative standard deviation reduction average with one shot in RefCOCO evaluation. In the same setting, when equipped with colored segmentation masks as visual sub-prompts, CPT can further achieve 20.0% absolute accuracy improvement, and 76.2% relative standard deviation reduction than the vanilla fine-tuning approach.
Our contributions are summarized as threefold: (I) We present novel cross-modal prompt tuning paradigm for VL-PTMs. To best of our knowledge, this the first attempt in both cross-modal prompt tuning for VL-PTMs, and zero- and few-shot visual grounding independent of object types. (2) We present a principled approach to search for high-quality cross-modal prompt configurations for CPT. (3) We conduct comprehensive experiments which demonstrate the effectiveness of CPT.
PRELIMINARY
In the literature, visual grounding is typically formulated as a referring expression comprehension (REC) problem (Plummer et al.) 2015 Mao et al. 2016). Given an image / and query text of referring expression g. REC aims to locate the target region in that corresponds to In this section, we introduce the vanilla fine-tuning approach for VL-PTMs.
A common practice for REC is to first detect set of region proposals {V1, 0....... } via object de- tectors, and then classify or rank the proposals select the target region (Lu et 2019: Chen al. 2020). Specifically, visual and textual inputs are first transformed into sequence of input tokens [MG], f1, U/, Un, [CL.), W1, W2, W'm [SEP]} where {w1, W2, ........ Wm) are textual tokens of and [IMG], [CL.S] and [SEP] are special tokens. To obtain input representations, the feature of image regions is extracted by visual encoders, and the embeddings of textual and special tokens are obtained by a lookup table. Then input representations are fed into the pre-trained transformers to produce the hidden representations { [h h h...., ...... h [cis] h h?.... h brsee] }- ".......h, Finally the hidden representation of the target region is optimized against negative fication or ranking loss, where new task-specific parameters are introduced. As a result, fine-tuned VL-PTMMs need large mount of labeled instances to stimulate the visual grounding capability.
classi-
2

---

Under review as a conference paper at ICLR 2022
3 CROSS-MODAL PROMPT TUNING (CPT)
In this section, we introduce the framework of CPT, and how to apply CPT to zero-shot, few-shot and fully supervised visual grounding.
3.1 OVERVIEW
The key visual grounding to establish fine-grained connections between image regions and textual expressions. Therefore, good rross-modal prompt tuning framework should take full ad- vantage of co-referential signals from both image and text, and maximally mitigate the gap between pre-training and tuning. To this end, CPT reformulates visual grounding into a fill-in-the-blank prob- lem, as shown Figure Specifically, the CPT framework consists of two components: (1) a visual sub-promph that uniquely marks the image regions with colored blocks or segmentation masks, and (2) a textual sub-prompt that puts the query text into a color-bast based query template. Equipped with CPT, it is then straightforward for VL-PTMs to ground the query text by filling the masked token with the color text of the target image region, where the objective form identical to pre-training
3.2 VISUAL SUB-PROMPT
Given an image and its region proposals R = {U1, ........ ), visual sub-prompt aims to uniquely mark the image regions with natural visual makers. Interestingly, we note that colored bounding boxes are widely used to uniquely mark objects in images for visualization in the literature. Inspired by this, bridge the image regions and query text through set of colors C, where each color (c), cg) E is defined by its visual appearance c, (e.g., RGB (255, O, 0)) and color text c red). redg. Then we mark each region proposal o in the image with a unique color c,, for grounding, resulting in set of colored image proposals & (R;C), C), where (-) denotes visual sub- sub-prompt.
As for the shape of the visual sub-prompt, in principle, there are multiple plausible choices mark the regions with colors, including colored bounding boxes, solid blocks, or solid object segmentation masks. In our experiments, we find that coloring the object with solid blocks and segmentation masks yields better results than bounding boxes, since solid colors that fit the outlines of objects are more common real-world images (e.g.. red shirl and blue car). Note that the addition of visual sub-prompt to the raw image does not change the architecture or parameters of VL-PTMs.
3.3 TEXTUAL
Textual sub-prompt aims to prompt VL-PTMs to establish the connections between the query text and image regions marked by visual sub-prompt. Specifically, the query text q (e.g., "the horse watched by the woman") is transformed into a fill-in-hhe-bland query using template (-)) as:
(g) = [CLS] g is in [MASK] color [SEP]
In this way, VL-PTMs are prompted to decide the color of which region is more appropriate fill in the mask (c.g., red or blue) as follows:
exp [AASK] clu)
- P(u=a
P( [MASK] c ((R;C),
(I)
Ec exp(h [MASK] C
where e is the target region, c is the embedding of c₁ in the pre-trained MLM head. Note that the procedure does not introduce any new parameters, and also mitigates the gap between pre-training and tuning, and therefore improves the data efficiency for tuning VL-PTMs.
3.4 TRAINING AND INFERENCE
Equipped with CPT, VL-PTMs can readily perform zero-shot visual grounding without any labeled data, since the cross- modal representations of colors and their composition with other concepts (e.g., objects, attributes and relations) have been well learned by VL-PTMs during pre-training. When few or full labeled instances are available, VL-PTMs can be further tuned by CPT using the entropy- based objective: L == - JED log ( q). where Drain is the training set.
3

---

Under review as a conference paper at ICLR 2022
MLM
watching ricling
Hiad
Vocatullery
[MOI
(cuB]
The
Vooman
proien
color
[MAK
horrs
rod
color
SSEP]
Figure 2: CPT framework for predicate classification by filling in-the--ttthee blank with reused MLM head.
Although it is appealing bridge the image and text through color-based prompt, we identify two key challenges in its design: (1) how to determine the configurations of the color set C, and (2) how to deal with the large number of image regions with limited pre-trained colors.
Cross-Modal Prompt Search. Previous works in textual prompt tuning show that prompt configu- rations (e.g., textual templates) have a significant influence on the performance (Jiang et al., 2020). In this work, we make the first investigation in searching the cross moossl prompt configuration (i.e., the color set C). Intuitively, C should consist of colors to which VI.-PTMs are the most sensitive. To obtain a color pre-training text as cy,, and its standard RGB as c (c.g., Q. = ((255, 0, 0), red)). However, this solution is sub-optimal, since it determines the color text without considering its visual appearance, and the visual appearance of a color in real-world images often differs from its standard RGB.
(Ce, ( naive approach is to adopt the most frequent color text in the
To address the challenge, present principled cross-modal prompt search (CPS) algorithm for CPT, which jointly considers visual and textual semantics in real-world cross- modal data. Specifi- cally, we first identify candidate set of color texts C.p and visual appearances C.. For each visual appearance candidate 6, E C., we feed into VL-PTMs a pseudo-data instance consisting of a pure colored block of & and text: "[CLS] a photo in [MASK] color [SEP] Then we compute the decoding score s(c,, G.) for each color text candidate Gw C.. as in Equation [ where a larger decoding score indicates higher correlation between ég and cu. To select the color texts that are sensitive by VL-PTMs, we retain the color texts that achieve the largest decoding scores for visual appearance candidates: Cu. = {ca for = arg max ed, s(2;, (g)), ét E C.).. Similarly, we can obtain the visual appearances according to the largest decoding score, resulting the color set: C {Ce, lc, arg max Eč, s(C,, e,)), cl E C. We refer readers to Section B for the pseudo-code of the algorithm. In experiments, we find that the resultant colors yield better results than the naive ones. To make the raw content of the colored image regions available to VL.-PTMs, a transparency hyperparameter
(0, 1) is further applied to color visual appearances in practice.
Image Region Batching. In visual grounding, the number of region proposals in an image usually exceeds the size of C (~ 10). Besides, we observe that heavily overlapped colored blocks can hinder visual grounding. Therefore, we divide the image regions into batches, where each batch contains handful of moderately overlapping image regions, and mark each batch with visual sub-prompt respectively. To handle the batches that do not contain the target region, we further introduce new candidate text none in the decoding vocabulary, to indicate that there is no target region in the batch.
3.5
CPT FOR PREDICATE CLASSIFICATION
In the previous sections, we introduced CPT for visual grounding. In fact, CPT can also be easily adapted to other cross-modal tasks, such as predicate classification. Given an object pair (including the categories and bounding boxes) in image, predicate classification aims to classify the relation into a relation set PP, Prroviding structured image representations that can facilitate many cross-modal tasks (Johnson al., 2015 Hudson & Manning 2019; Shi et al. 2019). In the literature, since the ground-truth relations cannot be exhaustively annotated during evaluation, to avoid false negatives, previous works typically score the triplets and evaluate the recall of top-N triplets (Xu et al. 2017
Zellers et al. 2018 Chen et al. 2019 Tang et al. 2019).
Visual and Textual Sub-prompts. As shown in Figure 2 to perform predicate classification, CPT first marks the image regions with visual sub-prompt as in Section 3.2 and puts the object pair in the query template as follows:
os) [CLS] The s. in di color is [MASK] the og in cl, color [SEP]
where s., is the subject text, og is the object text, and ci and ci. are the corresponding color texts. Then VL-PTMS are prompted recover the relation texts from masked tokens in the template. To
4

---

Under review as a conference paper at ICLR 2022
accommodate the varied number of tokens in relation texts (e.g., wearing, walking on, typically 1~3 tokens), we introduce a variable indicating the number of tokens in a relation text (c.g., - 2 for walking on). The template T:: will have I consecutive masked tokens for relation prediction. For each template T(: l), we introduce a special NA relation consisting of tokens, which indicates that there is no relation between the entity pair under T(:: Specifically, in our experiments, the NA relation is irrelevant, no relation, no relation with for / = 1, 2, respectively.
Training. Given relational triplet (s, r; o), after decorating the input image regions and the object pair with visual and textual sub-prompts, VL-PTMs are optimized with the MLM loss to recover the relational tokens. Specifically, denote the number of tokens in r as |r]. (1) For templates where == Ir]. models are asked to reconstruct the ith masked token in ((s, o; 1) with the ith relational token rt using the MLM head. (2) For templates where * |r), since there is no relation between (s, o) under (ss, 1), models are asked to reconstruct the NA relation. For (s, o) that do not have any relation in the image, models are asked to reconstruct the NA relation for all T(s,
Inference. During inference, given an object pair (s, o), we score the relations based their fitness to the prompt context. Specifically, the score of each relation r E P U {NA} ob- tained by the aggregated MLM scores of its composing tokens under the corresponding template: r:|T(s, o; ()). where ( = [r). Intuitively, larger s(r) indicates that the relation r better fits the prompt context. Finally, the triplets (s, r, o) are ranked according to the relation score sr)), where r P.
s(r) E log P( [MASK]
Compared with visual grounding that aims to locate image regions for ungrounded texts, predicate classification represents a different series of cross-modal tasks that aim perform semantic recog- nition based on grounded inputs, such as object classification (Zhao et al., (017) and scene graph classification (Xu et al. (017). In addition to better data efficiency, a crucial advantage of using CPT is that the semantic labels can be produced from open- open-world vocabularies, instead of fixed label sets.
EXPERIMENTS
In this section, we empirically evaluate CPT in prompting VL-PTMs for visual grounding in dif- ferent settings, including zero-shot, few-shot and fully supervised settings. We refer readers to Section C for the implementation details.
4.1 EXPERIMENTAL. SETTINGS
We first introduce the experimental settings of the visual grounding task, including datasets, training settings, evaluation protocols and baseline models in our experiments.
Datasets. Following previous works (Rohrbach et al.) 2016 Zhang et al. 2018). we adopt three
widely used visual grounding datasets collected from MSCOCO images (Lin al. 2014), including RefCOCO (Yu et al. 2016), RefCOCO+ (Yu et al. 2016) and RefCOCOg (Mao et al. 2016). We refer readers Section D.2 for more dataset details.
Training Settings. We report experimental results of different training settings, including (1) zero- shot setting, where no training data is available, (2) few-shot setting, where K training instances are available (K = 1, 2, 4, 8, 16), and (3) fully supervised setting, where the full training set is available.
Evaluation Protocols. (1) Evaluation metrics. Following Zhang et al. (0188;; Lu al. (2019)
we adopt accuracy of the grounding results as the evaluation metrics. An expression is considered correctly grounded if the loU of the top predicted region and the ground truth is greater than 0.5. (2) Model validation. To better approximate the few-shot scenario where only few labeled in- stances are available, inspired by [ao al.) (202T), we use a few-shot validation set (consisting of 16 instances) for few-shot and zero-shot experiments, and use full validation set for fully supervised experiments. (3) Robust evaluation. Previous works have shown that model training on limited data suffer from instability (Dodge et al.) 2020 Gao et al., 2021). For robust and comprehensive evaluation, we report mean results over 5 random training set splits, as well as the standard deviation. For fair comparisons, the training and validation sets are identical for our baselines and CPT.
Baselines. We evaluate two variants of CPT. including CPT using colored blocks (CPT-BIk) and colored segmentation masks (CPT-Seg). We adopt the widely used VInVI. (Zhang al. 2021)
5

---

Under review as a conference paper at ICLR 2022
Table Main results, Accuracies (%) of grounding referring expressions in zero-shot, few-shot and fully supervised settings. We report mean and standard deviation performance over 5 random splits. ZS: shoo-shot. Blk: colored block, Seg: colored segmentation mask.
RefCOCO
RefCOCO+
RefCOCOp
Shot
Model
val
lestA
lestB
val
lestA
lestB
val
Random
15.9 0.2
19.4 + 0.6
13.4 0.4
16.1+0..1
13.3 0.6
20.0 0.2
18.8 0.4
19.2 + 0.3
1
CPT Bik
26.9
27.5
27.4
25.4
25.0
27.0
32.1
32.3
CPT-Seg
32.2
36.1
30.3
31.9
35.2
28.8
36.7
36.5
Fine-tuning
23.5 5.7
20.6 9.3
+:
4:
8.4
26.9
CPT-BIK
31.1 1.3
37.7 1.7
32.2 + 1.5
35.9
10.4
32.2
39. +
3.4
39.9 + 3.0
CPT-Seg
37.2
41.5
42.3 5.9
43.1
2.9
Fine tuning
22.5 4..55:4..5
21.0 7.2
25.9 4.7
27.0 3.1
27.8 8: 4.2
27.0 2:0
28.4 12.0
28.1
11.3
CPT-BIK
30.9
3.6
97.3
Fow- Shot
CPT-Seg
39.8 1.7
45.6
33.9
0.4
38.6 3.6
44.5 = 4.5
32.8 1: 3.8
44.7 =
5.1
44.3
4.8
tuning
20.1
7.8
29.8
37.7
34.0
CPT-BIK
38.3
43.6 8.3
34.0 1.6
38.8 3.8
44.4 6.4 6.4
33.5 1.5
40.6 7.9
40.9 7.9
CPT Seg
47.4
Fine-tuning CPT-BIK
31.6+
37.8
6.5
31.4 5.1
36.2 3.6
10.1
1.6
2..7
10.6 11.2
40.1 11.7
41.0
43.9 1.7
35.8
39.3
46.1
33.2
43.4+
6.5
43.6
CPT-Sog
2.6
48.2
4.6
35.7
2.5
2,9 42,6
49.3 4.7
35,4
47.4
3.5
47.4
3.5
Fine luning
3.2
45.5
5.0
34.9 1. 3.0
41.8
47.3 3.1
36.2 2.3
47.5
4.1
47.8
16
CPT-BIK
11.8
51.4 4.1
38.2 2.3
41.5
18.2 2.1
31.7 0.9
47.8
2.1
48.2
CPT-Seg
45.3
53.3 = 3.0
37.5
1.3
$2.5 1.2
36.6
51.
2.6
51.4
MAttNet
76.7
81.1
70.0
65.3
71.6
52.0
66.6
67.3
Folly Supervised
VI.-5
71.2
71.3
VILBERT
72.3
78.5
62.6
VLBERT
71.6
77.7
61.0
[Pini
ERNIE ViL
74.0
80.3
64.7
UNITER
81.2
86.5
73.9
75.3
81.3
65.6
74.3
74.5
Fine-tume tunettning
81.8
87.2
74.3
74.5
80.8
64.3
74.6
75.7
CPT Blk
81.9
87.1
73.8
74.4
80.4
64.1
75.3
75.3
CPT-Seg
81,9
86.4
74.4
73.9
79.5
64.4
74.4
73.2
as the CPT backhone. We compare CPT with series of strong baselines that utilize detected proposals, including vanilla fine-tuning tnn-tuning of VinVL and other VL-PTMs (see Section [.] for more baseline details). For fair comparisons, we adopt the base size for all VL-PTMs. We refer readers to Section A.I for the results large size VL-PTMs.
4.2
MAIN RESULTS
The main results are reported in Table [ from which observe that: (1) CPT outperforms the random baseline and the strong fine-tuning baseline by a large margin in zero-shot and few-shot settings. For example, using colored blocks as visual sub- sob-prompts. CPT achieves 17.3% absolute accuracy improvement average with one shot in RefCOCO evaluation. This indicates that CPT can effectively improve sample efficiency tuning VL-PTMs. (2) Coloring objects with segmen- tation masks in visual nub-prompts (CPT-Seg) achieves even better results than blocks (CPT-BIk). The reason is that solid colors that the outlines objects are more common in real-world im- ages, making CPT-Seg more natural visual sub-prompts (despite requiring stronger annotation to train the segmentation tools). (3) Notably, CPT achieves significantly smaller standard deviation than fine-tuning. For example, CPT-BIk achieves 73.8% relative standard deviation reduction average with one shot in RefCOCO evaluation. This shows that a coherent tuning approach from pre-training lead to substantially more stable few-shot training, which is crucial factor for eval- uating few-shot learning models (Gao et al. (00T). (4) We note that CPT-BIk slightly underperforms fine-tuning with 16 shots ReiCOCO+ evaluation. The reason is that RefCOCO+ has more color- based expressions (c.g., the person in red shirt and blue hat), which can disturb our color- based CPT. However, this problem can be alleviated with more tuning instances in the fully supervised scenario, where models can learn to better distinguish colors in the query text and prompt template. (5) CPT models achieve comparable performance to strong fine-tuned VL-PTMs the fully super- vised settings. This shows that CPT is competitive tuning approach for VL-PTMs even in the fully supervised scenario. We note that CPT-BIL -ll slightly outperforms CPT-Seg in the fully supervised setting, and refer readers to Section A.3 for detailed analysis. In summary, compared to the vanilla fine-tuning approach. CPT achieves superior/comparable, and more stable performance in zero-shot, fhw-shot and fully supervised visual grounding.

---

Under review as a conference paper at ICLR 2022
4.3 INFLUENCE OF COLORS IN CPT's VISUAL GROUNDING
In our analysis, we first investigate the influence of colors-the key ingredients-in the visual grounding performance of CPT. Specifically, we compare colors obtained from the frequency-based baseline (Freq) (See Section (.44 and our cross-modal prompt search method CPS (Ours) in two dimensions, including an overall evaluation of top-N colors and zoom-in study of individual col- ors. Unless otherwise specified, all the following experiments are conducted based on CPT-BIk on the validation set of RefCOCO in 0, 2, shot settings.
Table Top-6 colors from the frequency- baseline and our CPS. Visual appearances and color texts are reported. Best viewed in color.
Model
Color
Color
Color #3
Color
Freq
(255,0,0),
(0,0.0), black
(0,0,255), blue
(0,255, O), green
(255,255,0g) yellow yellow
(165,42 42,4442,2), brown
Ours
(240,0,30),re (155,50,210), purple
(255,255,25) 25), yellow
(0, 10, 10,25,, blue
23,,0 pink
(0, (0225000) green
Overall Evaluation of Top-N Colors. We first show the top-6 colors recommended by each ap- proach Table 2 To evaluate the overall performance of the top colors from different models, evaluate CPT equipped with each color from the top-6 colors respectively, and report the mean accuracy and standard deviation over different colors. From the experimental results in Figure Ba observe that the top colors produced by CPS achieve both higher mean accuracy and lower stan- dard deviation than the baseline method in different shot-settings. The reason is that CPS jointly considers visual and textual semantics in searching cross-modal prompts, and therefore is able to effectively adjust and rank the colors for more accurate and stable visual grounding.
40
201
40
98
20
90
Freq
22
Freq
26
20
20
20
0 Shot
2 Shot Shot
0 Shot
rod
D00 groon low
plnk purple
rbd
0800 groan loW
pink purpl
rbd
groon pink
purpl
0-Shat
2-Shot
8-Shol
(a) Overall performance.
(b) Zoom-in study of individual colors in different shots.
Figure 3: Results of utilizing different colors for visual grounding, including (a) an overall evaluation of top-6 colors from different models, and (b) zoom-in study of aligned individual colors.
Zoom-In Study of Individual Colors. To investigate the fine tne-grained influence of specific colors in CPT's visual grounding, we further perform a zoom-in study of individual colors. To align the colors for comparison, we merge the top-6 colors from the baseline and CPS, and remove the colors that are not included in the models' complete color sets (e.g., black & CPS). We report the accuracies in Figure 3b from which observe that: (1) The performance of different colors varies greatly in prompting VL-PTMs in the same shot-settings, and the optimal colors are different in different shot- settings. The results indicate the large influence of cross modal prompt configurations consistent with the findings from recent studies in textual prompt tuning (Jiang et al., 22020; Gao et al. 2021) (2) Colors produced by CPS achieve comparable superior performance compared to the baseline in individual colors. The results show that given the color texts, CPS can properly adjust the color visual appearance (i.e., RGB) to improve the visual grounding performance. (3) We note that in some cases, colors produced by CPS slightly underperform the baseline. We hypothesize the reason that, CPS uses a single textual template to compute the decoding scores for color adjustment, which can be biased. The problem can potentially be addressed by ensembling templates as in Qin Fisner (02T), which we leave for future work.
4.4
CASE STUDY
To provide more intuitive understanding of CPT, conduct study the validation set of RefCOCO in 8-shot setting. From the results in Figure 4 we have the following observations: (1) CPT enables VL-PTMMs distinguish target objects distracted by the same of type objects using

---

Under review as a conference paper at ICLR 2022
Query Text: right elephant in water
Query Text: apple on the bottom the right of the orange middle
Query Text: food red bowl
(a) Correctly predicted
(b) Disturbed by objects of the same type, but still reasonable
(c) Disturbed by colors in raw image regions and text
Figure 4: Case study. The bounding boxes given by image region proposals (olive), ground-truth annotation (pink), CPT (green), and fine-tuning baseline (yellow) are highlighted accordingly.
zero-shot, FS: fully supervised. We report the mean and standard deviation performance over random splits.
Table 3: Predicate classification results on Visual Genome. ZS:
Shot
Model
Val
Test
R@
R@
mR
100
mR
100
Random
1.2
1.8 0.2
1.1 + 0.2 14.8
1.3 0.1
1.5 = 0.0
1.1
1.2 0.1
1.6
CPT Bik
33.6
34.7
15.5
29.3
30.5
13.0
14.5
Fine-tuning
3.8
0.1
7.8 + 0.9
8.7
4.1 0.1
0.0
6.7 0.3
0.6
CPT-Bik
16.3 +: 2.0
17.5 - 2.3
25.2 = 0.7
27.4 0.8
18.0 2.8
20.0 10.0:::0 3.0
23,9 = 0.3
26.3 0.3
Few- Pho-Shot
tuming
7.1
7.6.1.2..
11.8
4
CPT-BB Bll
14.4 + 0.4
15.4 + 0.4
30.4 + 1.5
32.8 1.6
17.7 + 0.6
19.3 0.6
28.5 + 1.5
32.1 1.0
Fine-tuning CPT-Blk
8.1
0.3
20.7 0.6
21.7 0.6
10:4 0.7
11.2 0.8
19.7 0.1
21.7
16
15.0 0.6
16.0 0.8
33.0 0.2
35.4
0.6
1844 1.0
20.0 1.1
32,5 - 0.5
36.1 0.6
32
Fine-tuming
9.7111
1.12::1111
21.9 0.6
22.9 0.:0002 022
11.7 0.2
12.4 0.3
22.0 0.1
24.1
CPT Blk
17.2 12:.04
18.2 0.4
34.6 1: 0.2
37.9
0.1
20.8 :: 0.1
22.3 0.1
34.0 :: 0.1
37.7
Neural Motif
65.2
67.0
14.8
16.1
[Daiel
BONN
32.9
PCPL
50.8
52.6
35.2
37.8
DT2-ACRS
39.7
only few training instances, while the fine-tunin tine-tuning method struggles succeed (Figure (2) CPT can be distracted by hard candidates (e.g., objects of the same type the target that requires complex reasoning to identify), but will typically produce reasonable predictions.
example, in Figure CPT predicts nearby apple while the fine-tuning baseline predicts a bowl. The reason is that CPT maximally reuses the pre-trained parameters of VL-PTMs, which can help prevent outragcous predictions that typically happen in few-shot fine-tuning.
find that CPT can be disturbed by colors in raw image regions and text. For example, can be difficult for the model to identify red bowl when the candidate regions colored by red blocks (Figure
(3) However,
4.5 EXPERIMENTS ON PREDICATE CLASSIFICATION
To investigate the generalization capability of CPT, we evaluate CPT on predicate classification task.
Experimental Settings. (1) Datasets. We adopt the popular Visual Genome dataset (Krishna et al.,
2017). which contains 50 visual relations. We refer readers to Section D.2 for the dataset details.
(2) Evaluation protocols. Following previous works (Xu et al. 2017 Chen et al.) 2019). we use @ @R@N) and mean @ecall@N (mR @N) as the evaluation metrics. During training, K la- beled instances are provided for each relation. (3) Baselines. We adopt fine-tuning of Vin as our most direct baseline model. Specifically, we feed the image regions and their categories into the model, and concatenate the visual hidden representations of the subject and object. Then the object pair representation is fed into a softmax classifier. All VL-PTMs are in base size. We also report the results of strong baselines that are tailored for the task, and are fully supervised with 315, 642 labeled triplets, including Neural Motif (Zellers et al. 2018), BGNN (Li et al. 2021a). PCPL Yan
al. 2020) and DT2-ACBS (Desai al. 2021).
8

---

Under review as a conference paper at ICLR 2022
Results. From the results in Table we observe that: (1) CPT significantly outperforms the ran- dom baseline and the strong fine-tuning baseline in zero-shot and few-shot settings. For example, using 32 shots, CPT achieves strong mR @ 100 of 37.7%, outperforming fine-tuning by 13.6% ab- solute points, and closely approaching state-of-the-ar fully supervised DT2-ACBS. This indicates that CPT can improve sample efficiency in tuning VI.-PTMs. (2) We note that while the macro per- formance of CPT monotonically increases as the shot number grows, the micro performance drops first in 1- and 4-shot settings. This is due the distribution gap between the balanced training set (i.e., K shot for each relation) and the long-tal -til test set. Since the relations in the pre-traini training corpora also follow a long-tail distribution, CPT can achieve a high starting point for miero performance.
5 RELATED WORK
Pre-trained Mlsion-language Models. Existing VL-PTMs can be roughly divided into three cate-
gories according to their pre-training objectives and architectures: (1) Masked language modeling based VL-PTMs are mainly pre-traine trained recover the masked tokens (Lu et al., 2019) [Su et al. 2
Tan & Bansal 2019; Li et al.) 2020 Yu et al. 2021): (2) Auto-regressive language modeling based PTMS model image and text tokens with Transformer decoders auto-regressively (Ramesh et al.)
202
Wang et al. 2021); (3) Contrastive learning based VL-PTMs are pre-trained to holistically
match image-text pairs (Radford et al. 2021 Li et al. 2021b). Note that our Cross-modas Prompt Tuning (CPT) framework is orthogonal to VL-PTM design. In this work, without loss of generality, we focus on prompting masked language modeling based VL-PTMss due to their prevalence and superior performance, while applying CPT to other VL-PTMss is also applicable.
Prompt Tuning for NLP. Prompt tuning for pre-trained language models is a rapidly emerging Originally designed for prob- ing knowledge in pre-trained language models (Petroni et al. 2019), prompt tuning has now been extended to handle a variety of NLP tasks, including language understanding (Schick & Schütze b) and generation (Li & Liang 2021). To facilitate prompt engincering, (Shin et al. (2020) pro- pose to automatically generate prompt templates via gradient-based search. Most related to our work are TTimpoukelli et al. (202T); Zhou et al. (2021): Wang et al. (2021) that present textual prompt tuning for VL-PTMs, achieving promising results on some vision- languae tasks. However, simi- lar to existing works in NLP, they focus prompt engineering in text, keeping images untouched, and therefore can only perform holistic implicit visual grounding.
field in NLP (Raffel et al. 2019 Brown al., 2020 Liu al. 2021).
comparison, to the best of our knowledge. CPT is the first cross cross-modal prompt tuning framework tailored for both image and text, and is capable of explicitly grounding natural language to fine-grained image regions.
Visual Grounding. There is general consensus that visual grounding plays an essential role in solving vasion-language tasks (Karpathy & Fei-Fei 2015: Plummer et al., 2015 Goodfellow et al.] 2016 Krishna al. 2017: Lu al.) 2019). Mao al. (2016) propose the referring expression comprehension task to explicitly evaluate the visual grounding capability. To address the task, most models learn to classify vised fashion (Mao et al.] 2016; Zhang et al.) 2018 Lu et al.] [2119] Chen et al.) (020), requiring large amounts of costly
rank image region candidates based the expressions in a fully super-
data. To alleviate reliance on human annotation, some works have investigated zero-/few-shot grounding of new object types (Sadhu et al. 2019 Blukis jet al. 2020), whereas amounts of training data are still needed for existing object types. com- parison, we prompt general VL-PTMMs for zero- and few-shot visual grounding in a reformulated fill-in-the-blank paradigm independent of specific object types.
6 CONCLUSION AND FUTURE WORK
In this work, we present the first Cross-modal Prompt Tuning (CPT) framework for VL-PTMs. To facilitate prompt engineering, we present a principled approach to search for cross- prompt configurations. Comprehensive experimental results demonstrate the effectiveness of CPT on zero- shot, few-shot and fully supervised visual grounding. In future, we plan to address the color dis- turbance and improve the computation efficiency of CPT, and also investigate the effectiveness of CPT on other vision-language tasks. As the first attempt in cross-modee prompt tuning, we propose a color-based based framework as one of the possible prompt tuning solutions. We leave exploring other plausible prompt tuning approaches of VI.PPMMs for future work.

---

Under review as a conference paper at ICLR 2022
7 ETHICS STATEMENT
In this section, we discuss the main ethical considerations of CPT: (1) Intellectual property pro- tection. The codes and data adopted from previous works are granted for assercc--urppose usage. (2) Privacy. The data adopted in this work (i.e., the pre-training data and tuning data) is created by human annotators for research purposes, and should not cause privacy issues. (3) Potential prob- lems. VL-PTMs may be biased towards some objects and attributes. There are increasing efforts to address the problem the community (Ross et al. 2021 Zhao al. 202T).
8 REPRODUCIBILITY STATEMENT
To maximize the reproducibility, we provide a clear description of the methodology in Section 3 the pseudo-code of the model in Section B implementation details Section [ and detailed data characteristics and evaluation protocols in Section 4.1 All the data and codes will be available to facilitate future research.
REFERENCES
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answer- ing. In Proceedings of CVPR, pp. 6077-6086, 2018a.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. rision-anddlanguaae navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of CVPR, June 2018b.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. VQA: Visual question answering. In Proceedings of ICCV pp. 2425-2433, 2015.
Valts Blukis, Ross A Knepper, and Yoav Artzi. Few-shot object grounding and mapping for natural language robot instruction following. arXiv preprint arXiv:201 1.03884, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 1415,, 2020.
Tianshui Chen, Weihao Yu, Riquan Chen, and Liang Lin. Knowledge-embedded routing network for scene graph generation. In Proceedings of CVPR, pp. 6163-6171, 2019.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: Universal image-text representation learning. In Proceedings of ECCV, pp. 104-120. Springer, 2020.
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying asiion-and-languae tasks via text generation. In Proceedings of ICML. PMLR, pp. 1931-1942, 2021.
Abhishek Das. Satwik Kottur, Khushi Gupta, Avi Singh. Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In Proceedings of CVPR, pp. 326-335, 2017.
Alakh Desai, Tz-Ying Wu, Subarna Tripathi, and Nuno Vasconcelos. Learning of visual relations: The devil is in the tails. In Proceedings of ICCV, pp. 15404-15413, 2021.
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint 0rX1v:2002.06305, 2020.
Tianyu Gao, Adam Fisch, and Dangi Chen. Making pre-trained language models better few-shot learners. In Proceedings of ACI., 2021.
10

---

Under review as a conference paper at ICLR 2022
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of CVPR, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In Proceedings of the ICCV, pp. 2969, 2017.
Drew A. Hudson and Christopher D. Manning. Learning by abstraction: The neural state machine. In Proceedings of NeurIPS, pp. 5901-59144 14, 2019.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can know what language models know? TACL, 8:423-433 2020.
Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. Proceedings of CVPR, pp. 3668-3678, 2015.
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descrip- tions. In Proceedings CVPR, pp. 3128-3131 2015.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. ReferitGame: Referring to objects in photographs of natural scenes. In Proceedings of EMNLP, pp. 787-798, 2014.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJC,, 123(1):32-73, 2017.
Rongjie Li, Songyang Zhang, Bo Wan, and Xuming He. Bipartite graph network with adaptive message passing for unbiased scene graph generation. In Proceedings of CVPR, pp. 11109- 11119, 2021a.
Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. UNIMO: Towards unified-modal understanding and generation via cross-modal contrastive learn- ing. In Proceedings of ACI., pp. 2592-2607. Association for Computational Linguistics, 2021b.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00// 2021.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Sheent-semantics aligned pre-training for vision- tasks. In Proceedings of ECCV, pp. 121-137. Springer, 2020.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In Proceedings of ECCV, pp. 740-755. Springer, 2014.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. arXiv preprint m 2021.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. ViLBERT: Pretraining task-agnostic visiolin- guistic representations for iisio-andd-anguual tasks. Proceedings of NeurIPS, 32:13-23, 2019.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan I. Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of CVPR, pp- 11-20, 2016.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models knowledge bases? In Proceedings of EMNIP-IJCNIPP pp. 2463-2473, 2019.
Bryan A Plummer. Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting region- toooontt--phae to-phrase correspondences for richer image- to-sentence models. In Proceedings of ICCV, pp. 2641-2644 2015.

---

Under review as a conference paper at ICLR 2022
Guanghui Qin and Jason Eisner. Learning how to ask: Querying LMs with mixtures of soft prompts. In Proceedings of NAACL, pp. 2003-221,220 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprini 10X.v:2103.00020, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint urXiv:1910 10683, 2019.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2102.1209 12092, 2021.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. Proceedings NeurIPS, 28:91-99, 2015.
Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, and Bernt Schiele. Grounding of textual phrases in images by reconstruction. In Proceedings of ECCV, pp. 817-834. Springer, 2016.
Candace Ross, Boris Katz, and Andrei Barbu. Measuring social biases in grounded vision and language embeddings. In Proceedings of NAACL, pp. 998-1008, 2021.
Arka Sadhu, Kan Chen, and Ram Nevatia. Zero-shot grounding of objects from natural language queries. In Proceedings of ICCV, pp. 4703, 2019.
Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. In Proceedings of NAACL, pp. 2339-2352. Association for Computational Linguisties, 2021a.
Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of EACL. pp. 255-269, 2021b.
Jiaxin Shi, Hanwang Zhang, and Juanzi Li. Explainable and explicit visual reasoning scene graphs. Proceedings of CVPR, pp. 8376-8384, 2019.
Taylor Shin, Yasaman Razeghi, Robert I. Logan IV, Eric Wallace, and Sameer Singh. Eliciting knowledge from language models using automatically generated prompts. In Proceedings of EMNLP, pp. 4222 4235, 2020.
Weijie Su. Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VL-BERT: Pre- training of generic visual- assua-linguistic representations. In Proceedings of ICLR, 2019.
Hao Tan and Mohit Bansal. LXMERT: Learning cross-modality encoder representations from trans- formers. In Proceedings of EMNLP-IJCNLPP, pp. 5100-5111, 2019.
Kaihua Tang. Hanwang Zhang, Baoyuan Wu, Wenhan Luo, and Wei Liu. Learning compose dynamic tree structures for visual contexts. In Proceedings of CVPR, pp. 6619-6628, 2019.
Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew Walter, Ashis Banerjee, Seth Teller, and Nicholas Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of AAAI, volume 25, 2011.
Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multi- modal few-shot learning with frozen language models. arXiv preprint arXiv: 106. 13884, 2021.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. VIM Sim- ple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108. 10904, 2021.
Danfei Xu, Yuke Zhu, Christopher Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In Proceedings of CVPR, pp. 5410-5419, 2017.
12

---

Under review as a conference paper at ICLR 2022
Shaotian Yan, Chen Shen, Zhongming Jin, Jianqiang Huang, Rongxin Jiang, Yaowu Chen, and Xian- Predicate-correlation correlation perception learning for unbiased scene graph generation.
Sheng Hua. Pepl: Predicate In Proceedings of ACM Multimedia, pp. 265-273 273, 2020.
Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. ERNIE-ViL: Knowledge enhanced vision-language representations through scene graphs. In Proceedings of AAAI, volume 35, pp. 3208- 3216, 2021.
Licheng Yu, Patrick Poirson, Shan Yang. Alexander C Berg. and Tamara Berg. Modeling context in referring expressions. In Proceedings of ECCV, pp. 69-85. Springer, 2016.
Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang. Xin Lu, Mohit Bansal, and Tamara L Berg- MAttNet: Modular attention network for referring expression comprehension. In Proceedings of CVPR, pp. 1307-1315, 2018.
Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. In Proceedings of CVPR, pp. 5831-5840, 2018.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. Proceedings of CVPR, pp. 6720-6731, 2019.
Hanwang Zhang, Yulei Niu, and Shih-Fu Chang. Grounding referring expressions images by variational context. In Proceedings of CVPR, pp. 4158 4166, 2018.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. VinVL: Revisiting visual representations in vision-language models. In Proceed- ings of CVPR, pp. 1579-5588, 2021.
Bo Zhao, Jiashi Feng, Xiao Wu, and Shuicheng Yan. A survey on deep learning-base fine-grained object classification and semantic segmentation. IJAC, 14(2):119-135, 2017.
Tony Z Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before Improv- ing few-shot performance of language models. In Proceedings of the ICML, 2021.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. arXiv preprint 3rXiv:200900113, 2021.
13

---

Under review as a conference paper at ICLR 2022
A SUPPLEMENTARY EXPERIMENTS
A.1 RESULTS OF LARGE SIZE VL-PTMS
In this section, we report the experimental results of large size VL-PTMs, including vanilla fine- tuning of baseline VL-PTMs, CPT-BIK and CPT-Seg with large size backbone (i.e., 024 dimen- sional hidden representations and 24 layers). From the experimental results in Table we observe that compared with vanilla fine-tuming CPT achieves significantly better and more stable perfor- mance in zero-shot and few-shot settings, and comparable results in the fully supervised settings, which is consistent with the conclusions of main experiments Section 4.2 In summary, the results show that CPT can generalize to VL-PTMMs of different sizes.
Table Results of large size VL-PTMs. Accuracies (%) of grounding referring expressions in zero- shot, few-shot and fully supervised settings. We report mean and standard deviation performance over random splits. ZS: zero-shot. Blk: colored block, Seg: colored segmentation mask.
RefCOCO
RefCOCO+
RefCOCOp
Shot
Model
testB
Random
15.9 1. 0.2
19.4 1 0.6
13.4 0.4
16.1 0.1
13.3 1. 0.6
20.0 1. 0.2
18.8 1. 0.4
19.2 0.3
CPT-BIk
27.0
25.8
25.7
32.6
CPT-Seg
29.5
30.6
28.7
28.8
30.3
27.4
34,6
34.8
Fine-tuning
18.5
13.7 4.8
25.0 3.7
23.0 6.5
22.8 8.2
23.6
23.6--4.1
30.6 = 7.3
31.5 7.4
CPT-BL
36.4 + 3.5
39.1
34.3
2.7
34.4 + 3.8
38.7 5.4
31.2
2.5
38.7 + 4.8
38.7 4.6
CPT-Seg
39.3 = 4.2
43.2 + 5.6
35.5
2.4
38.9 +
41.0
5.0
31.2
2.8
40.9 - 6.0
41.0 6.1
Fine-tuning
23.4 3.5
26.7
28.3
30.1
26.4
33.
33
2
CPT Bik
38.3 2.9
40.5 1: 4.2
1.2 35.3
36.2 1 5.5
41.1 11:18:77.6
31.9
3.3 33.5
40.6 1: 5.9
41.3 6.1
CPT-Seg
36.6
3.8
5.2
Fine-tuning
27.8 + 4.8
26.0 +
30.1
3.4
33.4 +
36.8 + 5.1
28.3 2.1
36.9 + 8.9
37.2 8.7
Few
CPT-BIk
40.9 1. 1.8
2.0 45.0
36.6
1.6
37.2
3.6
42.4
5.4
33.6
2.3
42.2 1 6.5
42.7 6.9
CPT-Seg
41.3 +-3-+50 5.2
45.9
36.5
3.7
39.8
3.8
45.7
34.1 1.8
45.7
7.3
45.8
Fine tuning
33.3 1: 4.2
35.6
7.4
31.2
2.7
38. 1: 3.7
43.5 8 3.9
31.2 1.222.135 3.8
41.9 1 8.0
42.5 7.9
CPT-BIk
42.7 1:
37.3
CPT Seg
45.2 1: 3.6
51.4
4.9
38.7
2.4
42.4 3.8
49.0
4,9
35.7
1.8
48.1
5.4
48.6
5.8
Fine tuning
38.4
2.4
42.8
33.4
2.5
40.7
3.2
45.6
34.7
1. 2.8
48.7
1. 3.5
49
16
CPT-BIk
45.7 = 2.5
53.0 +
37.9
1.5
41.8
:: 2.0
48.8
2.6
35.7
47.7
=: 2.4
18.6
CPT Seg
48.6 3,1
55.9
40.3
2.0
43.8
2.0
50.9
36.5
1.3
50.8
3.6
$1.6 3.7
MARNet
70.0
67.3
Fully Supervised
ViLBERT
72.3
78.5
62.6
ERNIE VIL.
76.0
82.1
66.9
UNITER
74.2
66.7
75.8
Finc-tuning
81.8
87.5
73.7
71.8
81.0
61.1
71.7
75.8
CPT-BIK
81.5
87.0
74.3
73.6
S0.1
64.1
74.
75.2
CPT CPg-Seg
81.8
87.3
74.1
74.1
79.5
63.8
73.6
74.7
A.2 EFFECT OF COLOR TRANSPARENCY
In practice, the color transparency is crucial hyperpa- rameter CPT. Essentially, the choice of transparency trade-off between two factors: small transparency can establish strong connections between color texts and visual appearances, but will undermine the visibility of the raw image region contents, and vice versa. To inves-
Accuracy
35
30
tigate the effect of color transparency, with different transparency of the default color (i.e., (240, 30), red), with step size 0.1 in grid search. From the results in Figure [5] we observe that: (1) The per- formance peaks at moderate transparencies
evaluate CPT
0-Shot 1-Shot
2-Shot
8-Shot
15
...
4-Shot
16-Shot
0.4
different
Transparency
shot-settings, which is consistent with our analysis of the trade-off. (2) Interestingly, the optimal transparency in- creases the number of training shots grows. The son is that the bottleneck of visual grounding in low shot
Figure Experimental results with dif- ferent color transparencies.
settings is learn to utilize obvious colors in CPT establish coarse-grained connections between
14

---

Under review as a conference paper at ICLR 2022
Query middle tail hanging down
Query Text: first
Query back chair from left
sandwich to the left
(a) CPT: V
CPT: v
(c) CPT:
Query Text: blue and yellow bus first
Query Text: whole bear next to empty spot on center
Query Text: chef glancing over at restaurant blurry
(d) CPT: FT:
CPT: FT:
(1) CPT: x, FT:
Figure 7: Visualization of grounding results. First row: zero-shot setting. Second row: fully su- pervised setting. FT: fine-tuning. The bounding boxes given by image region proposals (olive), ground-truth annotation (pink), CPT (green), and fine-tuning baseline (yellow) are highlighted ac- cordingly. Some images cropped for better visual effects.
images and text. In comparison, in many shot settings, with a better mastery of colors, fine-grained reading and understanding of image regions become more important to handle hard instances that require complex reasoning (e.g., composition of attributes and relations).
A.3 ANALYSIS OF VISUAL SUB-PROMPT SHAPE
In the main experimental results in Table we note that although CPT-Seg significantly outperforms CPT-BIk in the zero-shot and few-shot settings, it slightly underper- forms CPT-BIk in the fully supervised setting. To in- vestigate the reason, we divide target objects in the val- idation set of RefCOCOg into disjoint bins according to the area of the bounding boxes, where each bin contains equal numbers of target objects (thus contributes equally to the overall result), and report the average performance of each bin in the fully supervised setting. From the re- sults in Figure 6 find that CPT-Seg outperforms CPT- Bik on large objects, but is inferior in grounding small ob- jects. We hypothesize the reason is that CPT-Seg changes the object outlines with imperfect colored segmentation masks, hindering the understanding and reasoning ob- jects to some extent. The problem is exacerbated in small objects, since compared with large objects, the segmentation error of small objects is essentially en- larged when the object feature maps are pooled into input features of the same size for Transformers.
90
CPT-Seg CPT-BIA
Accuracy
78
60
(7.20)
[20,
[31. 49)
[49,
[85, 372 K10
Box
Figure 6: Performance of CPT-BIk and CPT-Seg with base size different box areas in the fully supervised setting.
A.4 VISUALIZZTTON
The few-shot grounding results are visualized in Figure 4 In this section, we further visualize the grounding results in zero-shot and fully supervised settings, shown in Figure 7 We find that
15

---

Under review as a conference paper at ICLR 2022
CPT can make reasonable zero-shot predictions. Moreover, observe that the color disturbance problem is largely alleviated in the fully supervised setting, i.e, CPT is less disturbed by colors in raw image and text, shown in Figure The reason is that capable VL-PTM can learn to largely distinguish the colors of varying objects and pre-defined maker blocks.
B PSEUDO-CODE OF CROSS-MODAL PROMPT SEARCH
Here we provide the pseudo-code of cross-modal prompt search. The algorithm aims jointly consider visual and textual semantics in real-world cross-modal data to search for the color set C CPT. The algorithm is simple its design, and leave exploring more advanced cross modal prompt search methods for future work.
Algorithm 1 Cross-modal Prompt Search
Require: P(s-):: VL-PTM with image regions R and query
text g as input
Require: C.: candidate color text set
Require: C.: candidate color RGB set
for ét C2 do
2: R pure color block of {t)
3: g q="[CSS] photo of [MASK] color [SEP]"
4:
for ci in C. do
s(C,, (w) P( (MASK) R)
end for
7: end for
8: Discard color candidates with low decoding scores
9: // Select sensitive color
10: C., = (cwlew arg max ed., s(t,, e,), E C.}
11: // Select corresponding sensitive color RGB
12: C ([c....)) arg max,t cd. s(ct, d,), de E C..)
C IMPLEMENTATION DETAILS
In this section, provide the implementation details about model training and inference, object detection and segmentation, as well as cross-modal prompt search.
Backbone. We adopt the widely used VLn (Zhang al. 2021 the backbone, which achieves strong performance on many vision- languan tasks. We use the VinVLaase model in the main experiments, with 768 dimensional hidden representations and encoding layers.
Object Detection and Segmentation. During training and inference, we use the region proposals predicted by the Faster-RCNN (Ren et al., 2015) and object segmentation masks predicted by the Mask-RCNN (He et al. (017)), which are provided by MAttNet (Yu et al.) 2018). Both Faster-RCNi RCNN and MCNN are based on ResNet101 (He et al. 2016) with a region proposal network and fully connected classifier for object detection. For MasK-RCNN, an additional mask branch is added to conduct multi-task learning. The RCSN and Mask-RCNN provided by MAttNee (Yu et al., 2018) achieve 34.1 and 30.7 average precision on the COCO test set respectively.
Visual Grounding. During training, an image region is considered as the target its intersection- over-union (IoU) with the ground-truth region is greater than 0.5. During inference, we select the target region with the largest. decoding score. All the hyperparameters and models are selected by grid search based on the performance on the few-shot/full validation set. The learning rate is 6e-5, and decreases linearly towards 0, which will be achieved the end of the training (500 and 20, 000 steps for few-shot and fully supervised training respectively). The batch size is 32 and identical to shot size in fully supervised and few-shot settings respectively. The size of image region batch is 1.
Predicate Classification. The hyperparameters and models selected by grid search the vali- dation set. The learning rate 3c-5, and decreases linearly towards 0, which will be achieved at the end of the training (200 steps). The batch size is identical shot size.
16

---

Under review as a conference paper at ICLR 2022
Table 5: Relations in Visual Genome dataset (Krishna et al. 2017). Some relations are renamed
(shown in parentheses) to better fit the query template.
at says (saying)
in over
to
on riding watching
and wears (wearing)
has (having) above
near carrying growing on standing on
under holding made of walking on helonging to
using
along against parked on
behind playing flying in mounted
cating covering
wearing
between
part of walking in
lying on sitting on in front
laying on painted
looking at
covered on back
attached to
hanging from
Cross-modal Prompt Search. The color text candidate set C.. is obtained from Wikipedia at Ltwprlissss__oo___ee
The color appearance candidate set C2 is obtained by grid searching RGB candidates around the standard RGBs of color texts in C.. In grid searching RGB candidates, the range =30 around standard RGBs with step size in each channel. Colors candidates are discarded if the decoding score is less than 0.8. The specific color choice from is determined based on the performance on the few-shot/full validation set. The optimal color used in our experiments are ((240, 0, 30), red) with transparency value 0.5 in zero-shot and few-shot settings, and c = ((255, 170, 230), pink) with transparency value 0.45 in the fully supervised setting.
en.
D EXPERIMENT DETAILS
D.1 BASELINE DETAILS
We provide baseline details for visual grounding. (1) Vanilla fine-tuning for VLn (Zhang et al. 2021]. This model adopts the same backbone as CPT, and serves as the most direct baseline few- shot and fully supervised experiments. Following/Chen et al. (2020), the logits for all regions are fed into a softmax layer, and the score of the target region is optimized using cross-entropy objective. (2) Vanilla fine- tine-tuming for other VL-PTMs. For fully supervised experiments,
also report previ- ous results of fine- tuning other VL-PTMs, including ViLBERT (Lu et al. 2019). VLBERT (Su al.)
2019), UNITER (Chen et al. 2020), ERNIE-VIR ViL. (Yu et al. 2021) and VL-T5 (Cho et al. 2021). (3) Visual grounding model. MAtNNet Yu et al.] 2018) is a strong model tailored for visual grounding, and is compared in fully supervised setting. (4) Random baseline. For zero-shot experiments, we
compare with a random baseline that randomly guesses the target region. For fair comparisons, we use the object proposals detected by MAttNet (Yu al. 2018) for all baselines and CPT-BIk.
D.2 DATASET DETAILS
Visual Grounding Datasets. (1) RefCOCO (Yu et al., 2016) is collected through two-player ref- erential game (Kazemzadeh et al., 2014), and contains 142,210 referential expressions for 50,000 object instances in 19.994 images. The dataset is split into train, validation, testA and testB sets, with 120,624, 10,834, 5,657 and 5,095 expression-object pairs respectively. TestA set only contains peo- ple as target objects, while testB set contains all other types of objects as targets. (2) RefCOCO+ (Yu al. 2016) is also collected in an interactive way, and contains 141,564 referential expressions for 49,856 object instances in 19,992 images. The difference from RefCOCO is that RefCOCO+ fo- cuses on distinguishing objects using appearance-bel bassed expressions, and excludes location-based expressions. The dataset is split into train, validation, testA and testB sets, with 120,191, 10, 758, 5,726 and 4,889 expression-object pairs respectively.
(3) RefCOCOg (Mao et al. (016)) is collected in non-interactive way, and contains 95,010 referential expressions for 49,822 object instances in 25,799 images. The referential expressions in ReiCOCOg are typically longer and more complex. The train, validation and test sets contain 80.512, 4,896 and 9,602 expression-object pairs.
Predicate Classification Datasets. We provide the relations of Visual Genome dataset in Table The dataset contains 65, 651, 5, 000 and 32, 422 images in training, validation and test set respec- tively, where each image contains an average of 10.3 objects and 4.8 labeled relation instances. There 150 distinct object categories and 50 relation categories in the dataset.
17

---

Under review as a conference paper at ICLR 2022
E DISCUSSION AND OUTLOOK
In this section, we discuss the limitations CPT and promising directions for future research.
Limitations. Despite its promising performance visual grounding, note that there several limitations in CPT: (1) Color disturbance. CPT takes advantage of colors to bridge visual and textual semantics, by adding color-based sub-prompts in both images and text. As shown in Section 4.4 the color-based prompt can be disturbed by colors in raw images and text. (2) Computation efficiency. In our experiments, color candidates, we adopt small image region batch sizes. This means that data instance needs fed into the model multiple times in order obtain the result. We believe addressing these challenges are promising directions for improving CPT.
maximally avoid color disturbance and account for the limited number of
Outlook. In this work, we take visual grounding as representative example to demonstrate the effectiveness of CPT. In fact, CPT can be easily adapted to other vision-language tasks. Here discuss the promising directions, as illustrated in Figure 8 The visual and textual sub-prompts in CPT can well capture fine-graide fine-grained object-level semantics for object-level tasks, such as: (1) Object classification. By coloring object proposals with visual sub-prompt, VL-PTMs can be prompted to produce object labels for object classification. (2) Scene graph classification. Moreover, by further decomposing textual sub-prompts, complex tasks involving different sub-tasks can be solved in a unified cross-modal prompt tuning framework. For example, VL-PTMs can be prompted to jointly produce object and predicate labels for challenging scene graph classification. In addition to data efficiency, a crucial advantage of using CPT is that the object/predicate labels be produced from open-world vocabularies, instead of fixed label sets.
MLM
horse worrn Vocaburary
Head
[MG
The
cljject
Tec
cutor
[MAASKI
ISEPS
Object Classification
woman
watching riding
horse cat Vonsatabery
man
Vaccabotary
Vacuabodaary
[MQ]
[MAX
grwen
[MASKK]
the
[UAASK]]
red
catter
(b) Scene Graph
Figure 8: Outlook for adapting cross-modal prompt tuning (CPT) to other tasks.
18


Reviews:
[{"review_id": "VxRqDaKRdVp", "reviewer": "Reviewer_vtFS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Generally, CPT is an interesting, simple but effective prompt-based approach, which reformulates the problem to fit the downstream task. The experimental results and ablation studies are good and provide many insights. The paper is well written. \n\nHowever, there are a few concerns of the paper:\n1. I doubt that CPT is a generalizable approach that can fit other V+L tasks. Most V+L tasks do not need segmentation level information. Maybe the author can try some tasks like VCR. But for general task such as VQA, captioning, text-image retrieval, I do not think CPT can be applied. Therefore, CPT can not be claimed as a framework for VL-PTMs. It is only for visual grounding tasks.\n\n2. The experiments only included a few simple baselines. We know the fully fine-tuning on large V+L models definitely can not generate reasonable results. It is possible to try some lightweight models like VL-T5 [1]. \n[1] Unifying Vision-and-Language Tasks via Text Generation", "summary_of_the_paper": "This paper proposed CPT, colorful prompt tuning for visual grounding tasks using the pre-trained V+L model. By adding color-based co-referential markers in both image and text, CPT makes visual ground as a fill-in-the-blank problem and mitigates the gap between pre-training and fine-tuning. The experiments are conducted on three visual grounding tasks and demonstrate the effectiveness of CPT. ", "summary_of_the_review": "I think the paper is good and has the opportunity to be in ICLR. I would appreciate it if my concerns can be addressed in the rebuttal session.\n\n-------------------------------------------------------------------------------------------------------------------\nAfter reading the rebuttal, my concerns have been addressed partially. I will keep my original score. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "T37DQTMc3bf", "reviewer": "Reviewer_KcLt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- An interesting work to try to establish the connections with words and grounding image regions.\n- It shows promising results on three refCOCO tasks. \n\nWeakness:\nThere are some questions or missing parts -\n1. What is the main pretrained model this work build on? VinVL? or ?\n\n2. For these refCOCO datasets, what is the statistics for the language (text part) that covers the image regions? for example, on average, how many words in one sentence that covers the image regions? My understanding is for the high quality COCO captions, on average, there is 1~2 word objects that covers the image regions. This question may relate the motivation of this work, if the coverage is not too high, it might be quite trivial for this method.\n\n3. Another question is - is it possible to verify this approach on a or more different task(s), rather than only refCOCO. For sure, refCOCO is a natural task for this method. ", "summary_of_the_paper": "This paper proposes a colorful prompt tuning for pre-trained vision-language models, with color masked regions and masked word tokens, on three reference tasks (refCOCOs), it shows promising results in zero-shot and few shot settings.", "summary_of_the_review": "This is an interesting work to utilize the grounding between image regions and text words, as a prompt tuning way for pre-trained vision language models, and shows promising results on three refCOCO tasks in zero-shot and few-shot settings. One concern or question is to verify the generalization capability of this work in more broad settings or diverse tasks, or it is just task-specific for refCOCO.\n\nBtw, I am not sure whether it is okay for section 8 & 9, which are on the page 10 (beyond 9 page limit).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MA7cmzl_U3s", "reviewer": "Reviewer_iZcJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n1. The idea of connecting images and texts with the concept of color is quite neat. Color is probably one of the most prominent visual features but this seems to be the first time that it is used to connect the two modality.\n\n2. CPT seems to tighten the gap between pretraining and fine-tuning. Previously, pretraining is a mask-language-modeling (MLM) task, while fine-tuning is a classification task that only uses the [CLS] head. Now, CPT also makes fine-tuning (prompt tuning) as a MLM task, so that it looks more similar to pretraining.\n\n3. The studies on how the color can affect the performance is comprehensive.   \n\n\nWeakness and questions:\n\n1. The method seems to rely on the quality of an image segmentation model to get the image regions. \u0010However, there is little detail about the model. How does the segmentation affect the CPT performance?\n\n2. The query template shown in Figure 1 requires the main body of the query text is a noun. What if it is not that case? For example, what if the the query text is sentence “The horse is standing in the yard”?\n\n3. The paper does not talk about how to ensure the MLM label in the cross-modal prompt tuning part (Figure 1(c)) is correct or how to deal with the ambiguity. For example, in Figure 1, how do you decide the answer is red or blue? For me, the woman can be watching either of the horses, so both red and blue can be correct.\n\n4. The empirical evaluation is only on the visual grounding task, but there are quite a few other multimodal tasks that can be evaluated on to make the paper stronger, e.g., visual question answering. Actually, the authors have pointed out that the method can potentially be applied to object detection, predicate classification and scene graph classification in Figure 5. But why don’t you just work on those tasks to show CPT’s superiority? Furthermore, even for the visual grounding task, CPT is only evaluated on RefCOCO and its variants, making the claims less convincing. Minor question: why is there no CPT-Seg result in the last row of Table 1?\n", "summary_of_the_paper": "This paper proposes a colorful prompt tuning (CPT) method for tuning pretrained vision-language models. CPT reformulates visual grounding into a fill-in-the-blank problem with color-based coreferential markers in image and text. The grounding to the target image region is achieved by recovering the corresponding color text from the masked token in the query template. Empirical studies show that CPT can outperform existing methods in promoting VL-PTMs for visual grounding in zero-shot, few-shot and fully supervised settings.", "summary_of_the_review": "While the idea of this paper is neat, there are a few details missing. The empirical studies lack comprehensiveness to support that the proposed method can generalize. I would be more than happy to change my score if the concerns are addressed well.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6qKpkIfp1VA", "reviewer": "Reviewer_WQNe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strengths]\n1.\tThe paper leverages the correspondence of colors between visual perception and textual input learned by the pre-trained model to enable visual grounding in downstream tasks, which is novel.\n2.\tExtensive ablation study and discussions are involved in the paper.\n3.\tThe proposed Cross-Modal Prompt Search mines the most sensitive colors and considers the correspondence between visual and textual semantics to determine the color set used in the prompt template, which provides some insights of prompt search.\n\n[Weaknesses]\n1.\tAs discussed in Section 4.5, one limitation is that the added color masks would disturb the original colors in the input images and eventually mislead the model. It is believed that such cases are common in real-world data, such as “man in red shirt” or “white cat on the couch”.\n2.\tImage Region Batching would bring more cost during inference.\n3.\tThe authors should visualize and compare the results of visual grounding derived by the proposed CPT and zero-shot/fine-tuning/fully supervised models.\n", "summary_of_the_paper": "This paper proposes a novel paradigm named Cross-Modal Prompting Tuning (CPT) that reformulates visual grounding into a fill-in-the-blank problem. Specifically, CPT applies a unique colorful mask to each visual region in the input image and then utilizes a pre-defined template to wrap the input text, where the modal needs to identify the color of the corresponding region that contains the described object. Experiments are conducted on RefCOCO, RefCOCO+ as well as RefCOCOg and promising results are achieved.", "summary_of_the_review": "Pls see the details in main review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In my opinion, this is a cool idea, but could use a few more test settings to evaluate the general applicability of their method. It would be interesting to see if the method generalizes to a non-reference based task.

Strengths:
Novel method that explores the interaction of color masks for learning to prompt about regions in images by identifying the color region they correspond to
Paper contains extensive ablation studies & discussions

Weaknesses:
Experimental results are run on uncommon benchmarks, making it difficult to compare to SOTA V+L methods
Consequently, it’s not clear that this method would generalize beyond visual grounding to tasks such as VQA or captioning
Decision: Reject