CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models

1 INTRODUCTION . Grounding natural language in fine-grained image regions is essential for a broad variety of visionlanguage tasks , such as robotic navigation ( Tellex et al. , 2011 ; Anderson et al. , 2018b ) , visual question answering ( Antol et al. , 2015 ; Anderson et al. , 2018a ) , visual dialogue ( Das et al. , 2017 ) , and visual commonsense reasoning ( Zellers et al. , 2019 ) . Recently Pre-Trained Vision-Language Models ( VL-PTMs ) have shown promising capabilities in visual grounding . Typically , generic cross-modal representations are first pre-trained on large-scale image-caption data in a self-supervised fashion , and then fine-tuned to adapt to downstream tasks ( Lu et al. , 2019 ; Su et al. , 2019 ; Li et al. , 2020 ; Radford et al. , 2021 ) . This pre-training-then-fine-tuning paradigm of VL-PTMs has greatly pushed forward the state-of-the-art of many cross-modal tasks . Despite the success , we note that there exists a significant gap between the objective forms of pretraining and fine-tuning of VL-PTMs . As illustrated in Figure 1 , during pre-training , most VL-PTMs are optimized based on the masked language modeling objective , trying to recover the masked token from the cross-modal context . However , during fine-tuning , downstream tasks are usually conducted by classifying unmasked token representations into semantic labels , where task-specific parameters are typically introduced . The gap hinders the effective adaptation of VL-PTMs to downstream tasks . As a result , a large amount of labeled data is typically required to stimulate the visual grounding capabilities of VL-PTMs for downstream tasks . In this work , inspired by recent progress in pre-trained language models in natural language processing ( Brown et al. , 2020 ; Schick & Schütze , 2021a ; Liu et al. , 2021 ) , we present Cross-modal Prompt Tuning ( CPT , alternatively , Colorful Prompt Tuning ) , a novel paradigm for tuning VLPTMs . The key insight is that by adding color-based co-referential markers in both image and text , visual grounding can be reformulated into a fill-in-the-blank problem , maximally mitigating the gap between pre-training and fine-tuning . As shown in Figure 1 , to ground natural language expressions in image data , CPT consists of two components : ( 1 ) a visual sub-prompt that uniquely marks image regions with colored blocks or segmentation masks , and ( 2 ) a textual sub-prompt that puts the query text into a color-based query template . Explicit grounding to the target image region can then be achieved by recovering the corresponding color text from the masked token in the query template . In addition , we present a principled method to search for high-quality cross-modal prompt configurations ( i.e. , visual appearances and texts of colors ) for CPT . By mitigating the gap from pre-training , CPT enables strong few-shot and even zero-shot visual grounding capabilities of VL-PTMs . Experimental results show that the prompt-tuned VL-PTMs outperform their fine-tuned counterparts by a large margin . For example , using colored blocks as visual sub-prompts , CPT achieves 17.3 % absolute accuracy improvement , and 73.8 % relative standard deviation reduction on average with one shot in RefCOCO evaluation . In the same setting , when equipped with colored segmentation masks as visual sub-prompts , CPT can further achieve 20.0 % absolute accuracy improvement , and 76.2 % relative standard deviation reduction than the vanilla fine-tuning approach . Our contributions are summarized as threefold : ( 1 ) We present a novel cross-modal prompt tuning paradigm for VL-PTMs . To the best of our knowledge , this is the first attempt in both cross-modal prompt tuning for VL-PTMs , and zero- and few-shot visual grounding independent of object types . ( 2 ) We present a principled approach to search for high-quality cross-modal prompt configurations for CPT . ( 3 ) We conduct comprehensive experiments which demonstrate the effectiveness of CPT . 2 PRELIMINARY . In the literature , visual grounding is typically formulated as a referring expression comprehension ( REC ) problem ( Plummer et al. , 2015 ; Mao et al. , 2016 ) . Given an image I and a query text of referring expression q , REC aims to locate the target region in I that corresponds to q . In this section , we introduce the vanilla fine-tuning approach for VL-PTMs . A common practice for REC is to first detect a set of region proposals { v1 , v2 , . . . , vn } via object detectors , and then classify or rank the proposals to select the target region ( Lu et al. , 2019 ; Chen et al. , 2020 ) . Specifically , visual and textual inputs are first transformed into a sequence of input tokens { [ IMG ] , v1 , v2 , . . . , vn , [ CLS ] , w1 , w2 , . . . , wm , [ SEP ] } , where { w1 , w2 , . . . , wm } are textual tokens of q , and [ IMG ] , [ CLS ] and [ SEP ] are special tokens . To obtain input representations , the feature of image regions is extracted by visual encoders , and the embeddings of textual and special tokens are obtained by a lookup table . Then input representations are fed into the pre-trained transformers to produce the hidden representations { h [ IMG ] , h1v , h2v , . . . , hnv , h [ CLS ] , h1w , h2w , . . . , hmw , h [ SEP ] } . Finally the hidden representation of the target region is optimized against negative ones via classification or ranking loss , where new task-specific parameters are introduced . As a result , fine-tuned VL-PTMs need a large mount of labeled instances to stimulate the visual grounding capability . 3 CROSS-MODAL PROMPT TUNING ( CPT ) . In this section , we introduce the framework of CPT , and how to apply CPT to zero-shot , few-shot and fully supervised visual grounding . 3.1 OVERVIEW . The key to visual grounding is to establish fine-grained connections between image regions and textual expressions . Therefore , a good cross-modal prompt tuning framework should take full advantage of co-referential signals from both image and text , and maximally mitigate the gap between pre-training and tuning . To this end , CPT reformulates visual grounding into a fill-in-the-blank problem , as shown in Figure 1 . Specifically , the CPT framework consists of two components : ( 1 ) a visual sub-prompt that uniquely marks the image regions with colored blocks or segmentation masks , and ( 2 ) a textual sub-prompt that puts the query text into a color-based query template . Equipped with CPT , it is then straightforward for VL-PTMs to ground the query text by filling the masked token with the color text of the target image region , where the objective form is identical to pre-training . 3.2 VISUAL SUB-PROMPT . Given an image I and its region proposalsR = { v1 , v2 , . . . , vn } , visual sub-prompt aims to uniquely mark the image regions with natural visual makers . Interestingly , we note that colored bounding boxes are widely used to uniquely mark objects in images for visualization in the literature . Inspired by this , we bridge the image regions and query text through a set of colors C , where each color ci = ( c i v , c i w ) ∈ C is defined by its visual appearance civ ( e.g. , RGB ( 255 , 0 , 0 ) ) and color text ciw ( e.g. , red ) . Then we mark each region proposal vi in the image with a unique color civ for grounding , resulting in a set of colored image proposals Ψ ( R ; C ) , where Ψ ( · ) denotes visual sub-prompt . As for the shape of the visual sub-prompt , in principle , there are multiple plausible choices to mark the regions with colors , including colored bounding boxes , solid blocks , or solid object segmentation masks . In our experiments , we find that coloring the object with solid blocks and segmentation masks yields better results than bounding boxes , since solid colors that fit the outlines of objects are more common in real-world images ( e.g. , red shirt and blue car ) . Note that the addition of visual sub-prompt to the raw image does not change the architecture or parameters of VL-PTMs . 3.3 TEXTUAL SUB-PROMPT . Textual sub-prompt aims to prompt VL-PTMs to establish the connections between the query text and image regions marked by visual sub-prompt . Specifically , the query text q ( e.g. , “ the horse watched by the woman ” ) is transformed into a fill-in-the-blank query using a template Tg ( · ) as : Tg ( q ) = [ CLS ] q is in [ MASK ] color [ SEP ] In this way , VL-PTMs are prompted to decide the color of which region is more appropriate to fill in the mask ( e.g. , red or blue ) as follows : P ( v = vi|R , q ) = P ( [ MASK ] = ciw|Ψ ( R ; C ) , Tg ( q ) ) = exp ( h > [ MASK ] c i w ) ∑ cj∈C exp ( h > [ MASK ] c j w ) , ( 1 ) where v is the target region , ciw is the embedding of c i w in the pre-trained MLM head . Note that the procedure does not introduce any new parameters , and also mitigates the gap between pre-training and tuning , and therefore improves the data efficiency for tuning VL-PTMs . 3.4 TRAINING AND INFERENCE . Equipped with CPT , VL-PTMs can readily perform zero-shot visual grounding without any labeled data , since the cross-modal representations of colors and their composition with other concepts ( e.g. , objects , attributes and relations ) have been well learned by VL-PTMs during pre-training . When a few or full labeled instances are available , VL-PTMs can be further tuned by CPT using the entropybased objective : L = − ∑ ( R , q , v ? ) ∈Dtrain logP ( v ? |R , q ) , where Dtrain is the training set . Although it is appealing to bridge the image and text through a color-based prompt , we identify two key challenges in its design : ( 1 ) how to determine the configurations of the color set C , and ( 2 ) how to deal with the large number of image regions with limited pre-trained colors . Cross-Modal Prompt Search . Previous works in textual prompt tuning show that prompt configurations ( e.g. , textual templates ) have a significant influence on the performance ( Jiang et al. , 2020 ) . In this work , we make the first investigation in searching the cross-modal prompt configuration ( i.e. , the color set C ) . Intuitively , C should consist of colors to which VL-PTMs are the most sensitive . To obtain a color ci = ( civ , c i w ) , a naive approach is to adopt the most frequent color text in the pre-training text as ciw , and its standard RGB as c i v ( e.g. , ci = ( ( 255 , 0 , 0 ) , red ) ) . However , this solution is sub-optimal , since it determines the color text without considering its visual appearance , and the visual appearance of a color in real-world images often differs from its standard RGB . To address the challenge , we present a principled cross-modal prompt search ( CPS ) algorithm for CPT , which jointly considers visual and textual semantics in real-world cross-modal data . Specifically , we first identify a candidate set of color texts Ĉw and visual appearances Ĉv . For each visual appearance candidate ĉv ∈ Ĉv , we feed into VL-PTMs a pseudo-data instance consisting of a pure colored block of ĉv and a text : “ [ CLS ] a photo in [ MASK ] color [ SEP ] ” . Then we compute the decoding score s ( ĉv , ĉw ) for each color text candidate ĉw ∈ Ĉw as in Equation 1 , where a larger decoding score indicates higher correlation between ĉv and ĉw . To select the color texts that are sensitive by VL-PTMs , we retain the color texts that achieve the largest decoding scores for visual appearance candidates : Cw = { cw|cw = arg maxĉjw∈Ĉw s ( ĉ i v , ĉ j w ) , ĉ i v ∈ Ĉv } . Similarly , we can obtain the visual appearances according to the largest decoding score , resulting in the color set : C = { ( cv , cw ) |cv = arg maxĉiv∈Ĉv s ( ĉ i v , c j w ) , c j w ∈ Cw } . We refer readers to Section B for the pseudo-code of the algorithm . In experiments , we find that the resultant colors yield better results than the naive ones . To make the raw content of the colored image regions available to VL-PTMs , a transparency hyperparameter α ∈ ( 0 , 1 ) is further applied to color visual appearances in practice . Image Region Batching . In visual grounding , the number of region proposals in an image usually exceeds the size of C ( ∼ 10 ) . Besides , we observe that heavily overlapped colored blocks can hinder visual grounding . Therefore , we divide the image regions into batches , where each batch contains a handful of moderately overlapping image regions , and mark each batch with a visual sub-prompt respectively . To handle the batches that do not contain the target region , we further introduce a new candidate text none in the decoding vocabulary , to indicate that there is no target region in the batch .

Summary: This paper proposes a novel paradigm named Cross-Modal Prompting Tuning (CPT) that reformulates visual grounding into a fill-in-the-blank problem. Specifically, CPT applies a unique colorful mask to each visual region in the input image and then utilizes a pre-defined template to wrap the input text, where the modal needs to identify the color of the corresponding region that contains the described object. Experiments are conducted on RefCOCO, RefCOCO+ as well as RefCOCOg and promising results are achieved.