CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models

1 INTRODUCTION . Grounding natural language in fine-grained image regions is essential for a broad variety of visionlanguage tasks , such as robotic navigation ( Tellex et al. , 2011 ; Anderson et al. , 2018b ) , visual question answering ( Antol et al. , 2015 ; Anderson et al. , 2018a ) , visual dialogue ( Das et al. , 2017 ) , and visual commonsense reasoning ( Zellers et al. , 2019 ) . Recently Pre-Trained Vision-Language Models ( VL-PTMs ) have shown promising capabilities in visual grounding . Typically , generic cross-modal representations are first pre-trained on large-scale image-caption data in a self-supervised fashion , and then fine-tuned to adapt to downstream tasks ( Lu et al. , 2019 ; Su et al. , 2019 ; Li et al. , 2020 ; Radford et al. , 2021 ) . This pre-training-then-fine-tuning paradigm of VL-PTMs has greatly pushed forward the state-of-the-art of many cross-modal tasks . Despite the success , we note that there exists a significant gap between the objective forms of pretraining and fine-tuning of VL-PTMs . As illustrated in Figure 1 , during pre-training , most VL-PTMs are optimized based on the masked language modeling objective , trying to recover the masked token from the cross-modal context . However , during fine-tuning , downstream tasks are usually conducted by classifying unmasked token representations into semantic labels , where task-specific parameters are typically introduced . The gap hinders the effective adaptation of VL-PTMs to downstream tasks . As a result , a large amount of labeled data is typically required to stimulate the visual grounding capabilities of VL-PTMs for downstream tasks . In this work , inspired by recent progress in pre-trained language models in natural language processing ( Brown et al. , 2020 ; Schick & Schütze , 2021a ; Liu et al. , 2021 ) , we present Cross-modal Prompt Tuning ( CPT , alternatively , Colorful Prompt Tuning ) , a novel paradigm for tuning VLPTMs . The key insight is that by adding color-based co-referential markers in both image and text , visual grounding can be reformulated into a fill-in-the-blank problem , maximally mitigating the gap between pre-training and fine-tuning . As shown in Figure 1 , to ground natural language expressions in image data , CPT consists of two components : ( 1 ) a visual sub-prompt that uniquely marks image regions with colored blocks or segmentation masks , and ( 2 ) a textual sub-prompt that puts the query text into a color-based query template . Explicit grounding to the target image region can then be achieved by recovering the corresponding color text from the masked token in the query template . In addition , we present a principled method to search for high-quality cross-modal prompt configurations ( i.e. , visual appearances and texts of colors ) for CPT . By mitigating the gap from pre-training , CPT enables strong few-shot and even zero-shot visual grounding capabilities of VL-PTMs . Experimental results show that the prompt-tuned VL-PTMs outperform their fine-tuned counterparts by a large margin . For example , using colored blocks as visual sub-prompts , CPT achieves 17.3 % absolute accuracy improvement , and 73.8 % relative standard deviation reduction on average with one shot in RefCOCO evaluation . In the same setting , when equipped with colored segmentation masks as visual sub-prompts , CPT can further achieve 20.0 % absolute accuracy improvement , and 76.2 % relative standard deviation reduction than the vanilla fine-tuning approach . Our contributions are summarized as threefold : ( 1 ) We present a novel cross-modal prompt tuning paradigm for VL-PTMs . To the best of our knowledge , this is the first attempt in both cross-modal prompt tuning for VL-PTMs , and zero- and few-shot visual grounding independent of object types . ( 2 ) We present a principled approach to search for high-quality cross-modal prompt configurations for CPT . ( 3 ) We conduct comprehensive experiments which demonstrate the effectiveness of CPT . 2 PRELIMINARY . In the literature , visual grounding is typically formulated as a referring expression comprehension ( REC ) problem ( Plummer et al. , 2015 ; Mao et al. , 2016 ) . Given an image I and a query text of referring expression q , REC aims to locate the target region in I that corresponds to q . In this section , we introduce the vanilla fine-tuning approach for VL-PTMs . A common practice for REC is to first detect a set of region proposals { v1 , v2 , . . . , vn } via object detectors , and then classify or rank the proposals to select the target region ( Lu et al. , 2019 ; Chen et al. , 2020 ) . Specifically , visual and textual inputs are first transformed into a sequence of input tokens { [ IMG ] , v1 , v2 , . . . , vn , [ CLS ] , w1 , w2 , . . . , wm , [ SEP ] } , where { w1 , w2 , . . . , wm } are textual tokens of q , and [ IMG ] , [ CLS ] and [ SEP ] are special tokens . To obtain input representations , the feature of image regions is extracted by visual encoders , and the embeddings of textual and special tokens are obtained by a lookup table . Then input representations are fed into the pre-trained transformers to produce the hidden representations { h [ IMG ] , h1v , h2v , . . . , hnv , h [ CLS ] , h1w , h2w , . . . , hmw , h [ SEP ] } . Finally the hidden representation of the target region is optimized against negative ones via classification or ranking loss , where new task-specific parameters are introduced . As a result , fine-tuned VL-PTMs need a large mount of labeled instances to stimulate the visual grounding capability . 3 CROSS-MODAL PROMPT TUNING ( CPT ) . In this section , we introduce the framework of CPT , and how to apply CPT to zero-shot , few-shot and fully supervised visual grounding . 3.1 OVERVIEW . The key to visual grounding is to establish fine-grained connections between image regions and textual expressions . Therefore , a good cross-modal prompt tuning framework should take full advantage of co-referential signals from both image and text , and maximally mitigate the gap between pre-training and tuning . To this end , CPT reformulates visual grounding into a fill-in-the-blank problem , as shown in Figure 1 . Specifically , the CPT framework consists of two components : ( 1 ) a visual sub-prompt that uniquely marks the image regions with colored blocks or segmentation masks , and ( 2 ) a textual sub-prompt that puts the query text into a color-based query template . Equipped with CPT , it is then straightforward for VL-PTMs to ground the query text by filling the masked token with the color text of the target image region , where the objective form is identical to pre-training . 3.2 VISUAL SUB-PROMPT . Given an image I and its region proposalsR = { v1 , v2 , . . . , vn } , visual sub-prompt aims to uniquely mark the image regions with natural visual makers . Interestingly , we note that colored bounding boxes are widely used to uniquely mark objects in images for visualization in the literature . Inspired by this , we bridge the image regions and query text through a set of colors C , where each color ci = ( c i v , c i w ) ∈ C is defined by its visual appearance civ ( e.g. , RGB ( 255 , 0 , 0 ) ) and color text ciw ( e.g. , red ) . Then we mark each region proposal vi in the image with a unique color civ for grounding , resulting in a set of colored image proposals Ψ ( R ; C ) , where Ψ ( · ) denotes visual sub-prompt . As for the shape of the visual sub-prompt , in principle , there are multiple plausible choices to mark the regions with colors , including colored bounding boxes , solid blocks , or solid object segmentation masks . In our experiments , we find that coloring the object with solid blocks and segmentation masks yields better results than bounding boxes , since solid colors that fit the outlines of objects are more common in real-world images ( e.g. , red shirt and blue car ) . Note that the addition of visual sub-prompt to the raw image does not change the architecture or parameters of VL-PTMs . 3.3 TEXTUAL SUB-PROMPT . Textual sub-prompt aims to prompt VL-PTMs to establish the connections between the query text and image regions marked by visual sub-prompt . Specifically , the query text q ( e.g. , “ the horse watched by the woman ” ) is transformed into a fill-in-the-blank query using a template Tg ( · ) as : Tg ( q ) = [ CLS ] q is in [ MASK ] color [ SEP ] In this way , VL-PTMs are prompted to decide the color of which region is more appropriate to fill in the mask ( e.g. , red or blue ) as follows : P ( v = vi|R , q ) = P ( [ MASK ] = ciw|Ψ ( R ; C ) , Tg ( q ) ) = exp ( h > [ MASK ] c i w ) ∑ cj∈C exp ( h > [ MASK ] c j w ) , ( 1 ) where v is the target region , ciw is the embedding of c i w in the pre-trained MLM head . Note that the procedure does not introduce any new parameters , and also mitigates the gap between pre-training and tuning , and therefore improves the data efficiency for tuning VL-PTMs . 3.4 TRAINING AND INFERENCE . Equipped with CPT , VL-PTMs can readily perform zero-shot visual grounding without any labeled data , since the cross-modal representations of colors and their composition with other concepts ( e.g. , objects , attributes and relations ) have been well learned by VL-PTMs during pre-training . When a few or full labeled instances are available , VL-PTMs can be further tuned by CPT using the entropybased objective : L = − ∑ ( R , q , v ? ) ∈Dtrain logP ( v ? |R , q ) , where Dtrain is the training set . Although it is appealing to bridge the image and text through a color-based prompt , we identify two key challenges in its design : ( 1 ) how to determine the configurations of the color set C , and ( 2 ) how to deal with the large number of image regions with limited pre-trained colors . Cross-Modal Prompt Search . Previous works in textual prompt tuning show that prompt configurations ( e.g. , textual templates ) have a significant influence on the performance ( Jiang et al. , 2020 ) . In this work , we make the first investigation in searching the cross-modal prompt configuration ( i.e. , the color set C ) . Intuitively , C should consist of colors to which VL-PTMs are the most sensitive . To obtain a color ci = ( civ , c i w ) , a naive approach is to adopt the most frequent color text in the pre-training text as ciw , and its standard RGB as c i v ( e.g. , ci = ( ( 255 , 0 , 0 ) , red ) ) . However , this solution is sub-optimal , since it determines the color text without considering its visual appearance , and the visual appearance of a color in real-world images often differs from its standard RGB . To address the challenge , we present a principled cross-modal prompt search ( CPS ) algorithm for CPT , which jointly considers visual and textual semantics in real-world cross-modal data . Specifically , we first identify a candidate set of color texts Ĉw and visual appearances Ĉv . For each visual appearance candidate ĉv ∈ Ĉv , we feed into VL-PTMs a pseudo-data instance consisting of a pure colored block of ĉv and a text : “ [ CLS ] a photo in [ MASK ] color [ SEP ] ” . Then we compute the decoding score s ( ĉv , ĉw ) for each color text candidate ĉw ∈ Ĉw as in Equation 1 , where a larger decoding score indicates higher correlation between ĉv and ĉw . To select the color texts that are sensitive by VL-PTMs , we retain the color texts that achieve the largest decoding scores for visual appearance candidates : Cw = { cw|cw = arg maxĉjw∈Ĉw s ( ĉ i v , ĉ j w ) , ĉ i v ∈ Ĉv } . Similarly , we can obtain the visual appearances according to the largest decoding score , resulting in the color set : C = { ( cv , cw ) |cv = arg maxĉiv∈Ĉv s ( ĉ i v , c j w ) , c j w ∈ Cw } . We refer readers to Section B for the pseudo-code of the algorithm . In experiments , we find that the resultant colors yield better results than the naive ones . To make the raw content of the colored image regions available to VL-PTMs , a transparency hyperparameter α ∈ ( 0 , 1 ) is further applied to color visual appearances in practice . Image Region Batching . In visual grounding , the number of region proposals in an image usually exceeds the size of C ( ∼ 10 ) . Besides , we observe that heavily overlapped colored blocks can hinder visual grounding . Therefore , we divide the image regions into batches , where each batch contains a handful of moderately overlapping image regions , and mark each batch with a visual sub-prompt respectively . To handle the batches that do not contain the target region , we further introduce a new candidate text none in the decoding vocabulary , to indicate that there is no target region in the batch .

Summary: This paper proposes a colorful prompt tuning (CPT) method for tuning pretrained vision-language models. CPT reformulates visual grounding into a fill-in-the-blank problem with color-based coreferential markers in image and text. The grounding to the target image region is achieved by recovering the corresponding color text from the masked token in the query template. Empirical studies show that CPT can outperform existing methods in promoting VL-PTMs for visual grounding in zero-shot, few-shot and fully supervised settings.