CPCF: A Cross-Prompt Contrastive Framework for Referring Multimodal Large Language Models
Authors: Lanyun Zhu
Deyi Ji
Tianrun Chen
Haiyang Wu
De Wen Soh
Jun Liu
Venue: ICML 2025 poster (2025)
Keywords: Referring multimodal large language models
Primary area: deep_learning->large_language_models

Abstract: Referring MLLMs extend conventional multimodal large language models by allowing them to receive referring visual prompts and generate responses tailored to the indicated regions. However, these models often suffer from suboptimal performance due to incorrect responses tailored to misleading areas adjacent to or similar to the target region. This work introduces CPCF, a novel framework to address this issue and achieve superior results. CPCF contrasts outputs generated from the indicated visual prompt with those from contrastive prompts sampled from misleading regions, effectively suppressing the influence of erroneous information outside the target region on response generation. To further enhance the effectiveness and efficiency of our framework, several novel designs are proposed, including a prompt extraction network to automatically identify suitable contrastive prompts, a self-training method that leverages unlabeled data to improve training quality, and a distillation approach to reduce the additional computational overhead associated with contrastive decoding. Incorporating these novel designs, CPCF achieves state-of-the-art performance, as demonstrated by extensive experiments across multiple benchmarks. Project page: https://lanyunzhu.site/CPCF/

CPCF: A Cross-Prompt Contrastive Framework for Referring Multimodal Large Language Models
Lanyun Zhu'
Deyi Ji
Tianrun Chen
Haiyang Wu?
De Wen Soh
Jun Liu
Abstract
Question: What is the color of [region1]?
Referring MILLMs extend conventional multi- modal large language models by allowing them to receive referring visual prompts and generate responses tailored to the indicated regions. How- ever, these models often suffer from suboptimal performance due to incorrect responses tailored to misleading areas adjacent to or similar to the target region. This work introduces CPCF, novel framework to address this issue and achieve su- perior results. CPCF contrasts outputs generated from the indicated visual prompt with those from contrastive prompts sampled from misleading re- gions, effectively suppressing the influence of er- roneous information outside the target region on response generation. To further enhance the effec- tiveness and efficiency of our framework, several novel designs are proposed, including prompt extraction network to automatically identify suit- able contrastive prompts, self-training method that leverages unlabeled data improve train- ing quality, and distillation approach to reduce the additional computational overhead associated with contrastive decoding. Incorporating these novel designs, CPCF achieves state-of-the-an per- formance, demonstrated by extensive experi- ments across multiple benchmarks. Project page: https///annynnzuusiecCCCC sttC/CCP//ttt/CC/////cc
Answer: The colors are orange and green and red.
Reeiona
Question: What in the plate [region2]?
Answer: A fruit salad and cake are in the plate.
Figure example of errors existing referring MLLM mod- cls caused by mislcading regions. The incorrect contents in the responses are highlighted in red
and struggle with fine-graine visual tasks, such as re- gional description. To address this limitation, several re- ferring MLLMs (You et al., 2023; Peng et al., 2023) have been proposed. These methods receive various types of vi- sual prompts input, such as points, bounding boxes, and masks, allowing the MLLM to focus on specific regions and generate responses tailored to the indicated areas.
While these methods have achieved some success, as shown in Figure 1, their performance often remains suboptimal, fre- quently generating incorrect responses related misleading areas adjacent or resembling the target region, rather than being fully tailored to the input visual prompt. For example, in the case shown in Figure 1, the MLLM incorrectly infers the color of region 1 as "green and red",, corresponding to the surrounding kiwifruits and strawberries. Additionally, it mistakenly concludes that there is cake on the plate in region 2. influenced by the presence of another plate in the upper-right corner of the image that contains To investigate further, we use image editing tools modify these misleading image regions (e.g.. replacing kiwifruits and strawberries on the plate with blueberries, replacing the cake on the top-right plate with a steak). We observe that the erroneous content in the MLLM's responses changes accordingly (c.g., from "green and red" "blue". or from "a cake" "a steak"), indicating that these errors are due confusion caused by misleading regions rather than merely being LLM's inherent hallucinations unrelated to the image content. Such errors frequently occur in existing referring MLLM methods, underscoring their limitations robust- ness and effectiveness, highlighting the need for developing novel mechanisms to further enhance response reliability.
1. Introduction
cakes.
Recently, multimodal large language models (MLLMs) (Liu et al., 2023; Li al., 2023b) have established new paradigm for multimodal learning, achieving remarkable success in tasks such as image captioning and visual ques- tion answering. However, conventional MLLMs are typi- cally restricted to only coarse, image-level understanding
"Singapore University of Technology and Design "Tencent "Zhejiang University 'Lancaster University. Correspondence to: Tianrun Chen, Lanyun Zhu <lanyun.zhh sacnyunnzhueee mymail.sutd.cdd edu.sg. hannunnchen www.hhwwnnnuuu......oee eju.edu.en?.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

CPCF: A Cross-Prompt Contrastive Framework for Referring Multimodal Large Language Models
To address the above issues, in this work, we propose novel and effective referring MILM framework with hanced performance, inspired by the success of contrastive decoding (Li et al., 2022; Leng et al., 2023) in reducing hallucinations for LLMs and MILMMs. The core idea of our framework is to contrast outputs generated from different vi- sual
prompts, a self-training method to improve network opti- mization, and the first distillation framework tailored for contrastive decoding techniques to reduce computational costs. (3) Extensive experiments on multiple benchmarks demonstrate that our CPCF achieves SOTA performance.
the original input prompt r and contrastive prompts f extracted from misleading regions. This approach amplifies the accurate information associated with while suppressing erroneous information related to the misleading regions, thereby enhancing response accu- racy and reliability. Building such a framework requires addressing several critical technical challenges. The first challenge is how to identify suitable contrastive prompts f to achieve perforrefore contrastive decoding. A naive and straightforward approach is randomly and manually sample points from regions adjacent to or similar to A. However, such handcrafted and random methods lack robustness and are difficult to yield optimal prompts, since even small variations in the sampled points (e.g., shifts of 5 pixels) may result in significant changes to the final results. To address this limitation, we propose a novel prompt ex- traction network that automatically finds optimal contrastive prompts based the input image and instruction, elimi- nating the uncertainty and instability inherent in manual methods. Furthermore, we introduce self- seff-training method that leverages unlabeled data and self-generated question- answer pairs for DPO training, effectively optimizing the performance of the prompt extraction network and resulting in improved contrastive decoding outcomes.
2. Related Work
Referring Multimodal Large Language Models. Re- cent advancements in multimodal large language models (MLLMs)) (Li et al., 2023b; Liu et al., 2023; Zhu et al., 2023; Li et al., 2023a; Zhu et al., 2024a; 2025b;a; Ji et al.,
2024; Chen et al., 2024a) have expanded the capabilities of conventional LLMs (Touvron et al., 2023; Bai et al., 2023) to the visual domain, achieving remarkable performance on tasks such as image captioning and visual question an- swering. Referring MLLMS (You et al., 2023; Zhang et al., 2024; Wu et al., 2024; Zhang et al., 2023; Peng et al., 2023; Chen et al., 2023; Xuan et al., 2024; Yue et al., 2024; Yuan et al., 2024; Rasheed et al., 2024) further enable models to receive prompts in the form of points, boxes, or masks as input and generate responses for the indicated regions, al- lowing users to interact with models in a more fine-grained manner. However, existing referring MLLM methods of- ten produce incorrect answers by mistakenly focusing on confusing regions adjacent to or similar to the indicated referring regions. To address this limitation, this work intro- duces a novel, task-specific framework that mitigates these errors and significantly imprvve performance.
Contrastive Decoding. To address the hallucination prob- lem in LLMs, contrastive decoding techniques have been proposed to generate more accurate responses by comparing outputs from different models (Li et al., 2022) or different inputs (Kim et al., 2024). Recently, contrastive decoding has been extended to MLLMs (Leng et al., 2023; Zhu et al., 2024b; Wang et al., 2024; Chen et al., 2024b) to improve their reliability in multimodal scenarios. However, most existing contrastive decoding methods are not specifically designed for referring MLLMs. The most closely related work to ours is CRG (Wan et al., 2025), which also employs contrastive decoding for referring tasks, However, CRG relies on perturbing certain regions of the image to gener- ate contrastive inputs, a process that can introduce noise and errors. Moreover, CRG incurs significant additional computational costs, as it requires running the model mul- tiple times for each input. In contrast, our method utilizes different prompts as contrastive counterparts, avoiding the semantic disruption caused by CRG's perturbation based approach. Additionally, a novel mechanism is proposed to automatically generate contrastive prompts, addressing the instability associated with manual strategies. Further- more, we introduce an innovative distillation method that substantially reduces the computational overhead typically associated with most existing contrastive decoding meth-
Another key challenge is the additional computational cost incurred by the multiple executions of the MLLM during contrastive decoding. To address this issue and improve efficiency, we propose a novel distillation method that trans- fers the capabilities of the multi- multi-execution contrastive de- coding framework to a student model requiring only a sin- gle execution, thereby significantly redueing computational cost. To enhance the effectiveness of this distillation pro- cess, diffusion-based inpainting loss function is further introduced, ensuring regional contrastive information to be effectively extracted and utilized by the student model. In- corporating these designs, we develop novel and effective referring MIL.M, CPCF, with extensive experiments across multiple benchmarks and tasks demonstrating its outstand- ing performance and significant advantages.
In conclusion, the main contributions of our work are as fol- lows: (1) We introduce CPCF, an effective referring MLLM framework that contrasts input prompts with contrastive prompts from misleading regions, enabling highly accurate region understanding and high-quality text response gen- eration. (2) We propose several novel designs to enhance model performance and efficiency, including an automatic prompt extraction mechanism to identify optimal contrastive
2

---

CPCF: A Cross-Prompt Contrastive Framework for Referring Multimodal Large Language Models
(A) Prompt Extrac tion Network
(B) Response Generation
Leamable Queries
Input Instruction
Input Image
What is the main color of
Visual P rompt
the [resion??
Final Response
Image Encoder
Text Encoder
Year Yeesk
R Referring MLLM
(1),
White
CLI Encoder
Self Attention
Self Attention
BRx, 3o)
Cross Attention
3-layers Conv
Feed F Forward
Feed Forward
M
Ma
Select & Cluster
Contrast
Contrastive Prompts
Figure Overall pipelines of (A) prompt extraction network and (B) response generation process.
ods, including CRG. Incorporating these novel designs, our framework significantly different from CRG. Results in Table 3 highlight the performance advantages of our method over existing contrastive decoding techniques.
the approach described in Sec.3.3. The combined model, consisting of the MLLM and the prompt extraction network, is denoted as om. (3) Finally, reduce the additional com- putational overhead introduced by contrastive decoding, the trained 6m is distilled into o, (Sec.3.4), which executes the MILM only once. The following sections provide a detailed explanation of the key techniques in our framework.
3. Method
3.1. Preliminaries and Overview
3.2. Generation of Contrastive Referring Prompts
Referring MLLMs extend standard MLLMs by incorporat- ing an additional prompt, r, as imput to help the model focus on specific regions of interest!. The prompt r can take vari- ous forms, such as bounding boxes, masks, or points. The prediction process of referring MILM is formally defined as: p(-18)) Softmax[MLLM// 9, z, V<oi], where rep- resents the input image, x is the instruction, V<i is the first i - - tokens in the response y, and denotes the prediction likelihood. Building on the basic structure of Ferret (You et al., 2023), our method incorporates a contrastive decoding framework to mitigate common issue in existing methods: misfocusing confusing regions. Formally, the prediction process of the proposed approach can be written as:
Prompt Extraction Network. We propose a prompt extrac- tion network to automatically generate the contrastive refer- ring prompts, instead of relying on manual strategies, which are often suboptimal and lack robustness (as discussed in the Introduction). As illustrated Figure (A), the network comprises an image encoder and encoder, each with stages. The input to the image encoder consists of 64 learn able query embeddings = (e: ) while the input to the text encoder is the instruction æ. In each stage, the hidden states of first interact with both themselves and the text tokens from the text encoder via self-attention. They then in- teract with image tokens through cross-attentien where the image tokens are enhanced by prior information (detailed in the next section). The hidden states of each e, from the final stage are passed through linear layer to produce 2D coordinates (h,, w;) and confidence score Notably, the text features of the instruction are integrated into the prompt generation process. This design ensures that the generation of the contrastive prompt instruction-deeendeet,, as differ- ent instructions may cause the model to mistakenly focus different confusing regions. We utilize the first 3 stages of the Q-Former from BLIP2 (Li et al., 2023b) to initialize the network, leveraging its pre-learned, effective image-text feature interaction capabilities.
p(-ly<x) Softmax
N ((1 + 2I,
aMLLM(I, fn.
where Fn denotes the n-th contrastive referring prompt (in the form of points) and a is a hyperparameter. To improve performance and enhance efficiency, we propose shree-step training process for framework: (1) First, the MILM is pretrained using the same method Ferret. (2) Next, the pretrained MILMM is frozen, and prompt extraction network (Sec.3.2) 2) is introduced to generate the contrastive prompts (Ar N.
n= in 1. This network is trained using
Enhancement with Prior Information. As discussed in the Introduction, referring MILMS often mistakenly focus on regions adjacent or similar to r' corresponding region R. To assist the prompt extraction network in generating the
In the main paper, we illustrate our method for the inputs containing only one visual prompt r. Our method can be easily extended multi-r scenarios, as introduced Appendix A.c.A.1.
3

---

CPCF: A Cross-Prompt Contrastive Framework for Referring Multimodal Large Language Models
more appropriate i., we incorporate these empirical observa- tions as prior information enhance the generation process. We begin by creating two masks: a semantic similarity map M, E R ff x W and relative distance map Ma E ""*W, where H and W are the height and width of the input image I. Each pixel M in M, is computed as the L2 distance between the CLIP features of (i.3) and the average features of R. Each M in Ma represents the coordinate Euclidean distance from ($,5) to its nearest pixel within R. We nor- malize M, and Ma to the range [0, IJ. with the pixels inside R. set 1 as a flag indicating region R. Ms and Ma are then processed through a 3-layer CNN, producing outputs fe and fa. which are added to the image tokens. The en- hanced tokens are finally utilized in the image encoder of the prompt extraction network, where they interact with the query embeddings e (see the previous section).
Image D
Generated question a Why does he [region] jump and raise the hand?
Referring MLLM
You are an assistant to ask a question about [region F] in [image FJ. You can refor to the following example
Retrieval
this process: Image: dy, Region. di, Question: q. Please generate questions in similar mannerr You can generate quections complex than the example, but they relevant to the indicated regions and images.
Question q: Why is he Iragion
posture?
Image Description d,; A sunset scene at the besch with vibrant orange and pink foreground, two people are activelly playing beach volleyball, with a net set up on the sand.
the the
Dataset D
Ii
Repion Description d,; man beach valleyball, leaning slightly forward in ready stance he prepares interact with the ball.
playing
Reference Data
Contrastive Prompt Selection. We select appropriate prompts from the 64 outputs (h,, wi, C: 1 generated by the prompt extraction network. To enhance exploratory be- havior and improve robustness, we add a random standard Gaussian noise A to o and compute 8; = Sigmoid +-). Points (he, w;) with 8; > 0.5 are considered valid con- trastive prompts. We observe that many of these prompts are located in adjacent positions. To reduce redundancy, we apply the Mean Shift algorithm to cluster the positions of all valid prompts. The resulting cluster centers are then used as the contrastive prompts {fn [ in Eq.
Figure 3. Illustration of the RAG-based question generation pro- cess in the proposed self-training method.
answers for each image in D to train the prompt extraction network, enabling instruction tuning without the costly and labor-intensice manual or GPT-based data annotation.
Question Generation with RAG. We denote each data D in the Ferret's training set D2 as D' = {I', r" ".1", a²) rep- resenting the image, referring prompt, question, and answer, respectively. The CLIP feature of I is computed as the global representation fg. and the average CLIP features of region r' serve as a local representation fí- For each image I in the new set D, we randomly select a labeled bounding box and randomly transform it into a point or mask, creating the prompt 7. The corresponding global and local repre- sentations fg and ft for and i are then computed. We calculate the cosine similarity sg between F, and each fü in D, as well sj between fi and each fj. The data D with the highest s' + +s} among D is selected, and two text descriptions, d, and dy, are then generated by the MLLM for the full image I and the region rf in the selected D', respectively (see Appendix Sec.A.2 for the prompt). These descriptions, along with the corresponding question q, form an in-context example that prompts the MLLM to generate a question q for the region f on i in a similar fashion. Please refer to Figure 3 for the detailed prompt used in this pro- cess. Notably, we employ a retrieval- augmented generation (RAG) approach with in-context learning for this question generation method. This is because directly prompting the MILM to generate questions often results in errors and hallucinations, producing questions unrelated to I and F. In contrast, providing the MLLM with similar example from D for reference can significantly improve reliability.
3.3. Optimization with Self-Training
Motivation. The next key challenge is how to train the prompt extraction network effectively. We first tried two straightforward approaches: (I) we initialize the MLLM parameters using LLaVA and jointly train both the MLLM and the prompt extraction network on the referring question- answer dataset D. However, this approach results in un- stable training and poor performance. (2) We first train the MLLM on 'D following Ferret's 's method. Afterward, the MILMM is frozen, and the prompt extraction network is added and trained on the same dataset D. While this leads to some improvement, the results remain unsatisfactory. A potential reason for the limited success is that the MILM, having already trained on D and fit well to it, exhibits few hallucinations and errors on the training set. As result, there is very minimal room for further enhancement through contrastive decoding on DD making it difficult to train the prompt extraction network effectively. To address this, we introduce an additional dataset D beyond D - sampled from the Open Images dataset (Kuznetsova et al., 2020)- to train the prompt extraction network, which is not used in MLLM's training and contains rich bounding box labels. To mitigate the issue that D lacks question-ansee answer annotations, we propose a self-training mechanism, in which the MLLM,, after trained on D. is prompted to generate questions and
'The data from the VCR dataset within D is used for RAG because contains diverse and complex questions.
4

---

CPCF: A Cross-Prompt Contrastive Framework for Referring Multimodal Large Language Models
Results in Sec.4.3 and Appendix S demonstrate the effectiveness of the proposed approach.
Adapter Generation Network
Concat
Self Atten lio
S0l Attenttennn flom
Se A.L.Ateeriion
Prompt @
Answer Generation and Preference Optimization. With the image-prone prompt pair {f. } and the generated question q. the MILM is further prompted to generate two answers, a and ā is directly obtained by MLLM(I, F, .), while anot is produced by incorporating a chain-of of-thuught (CoT) prompt Pc. i.e. MLLM(1,FF F, q. Pe). encouraging the MLLM to generate results in more detailed, step-by-step manner (see Appendix S.c.A.3 for details of po)). Results presented in the Appendix reveal that a is more accurate than ā in most cases. Based on this observation, we employ the fol- lowing direct preference optimization (DPO) loss (Rafailov et al., 2024) to train the prompt extraction network:
Extraction
Network
Goncat
W
What is the main color of (region) ?
Adapter Module A
LLM Layers
LLM Lay Lavers
LLM Layers
LLM Layers
Large Language Model
LLM's Input
Figure 4. Model structure of the single-exccution model Φ., that distilled from the multi-execution model.
To aa I,F.
Capo Bloo a log
Tea @0 1, (,
by the attentions in the current layer. To generate A, we first extract regional features f, (with 32 tokens) using Ferret's ' method (see (You et al., 2023) Sec 3.2) for the input prompt r's region R. The same method is employed to generate fr. E [G,,, for each contrastive prompt A. E W The adapter tokens A are then computed as:
To ( [ (),
3 log Tee @ 1, F.
)}
where B is a hyperparameter set to 0.5, Ha is the policy MLLM that is continuously updated during training, and Teet is a reference MILMM updated via a Momentum strat- egy in each iteration, i.c., Tee 0.999 + 0.001 log (ā) F. Te E.il pog F, 7, aes) is the av- erage likelihood of all tokens in à computed by E..
N₂
E.] = AGN (Concat [G,; [G, e)))),, A = W
Σ n=11
E,
where AGN is an adapter generation network composed of
3.4. Distillation for Computation Reduction
3 sequential self-attention layers, e = {e:)]²m is set of learnable embeddings and A refers to their hidden states at AGN's output layer, W denotes a projection layer. To reduce computational and parameter overhead, the adapter modules at different LLM layers share the same AGN but use independent projection layers W.
The aforementioned model structures and train- ing methods enable effective contrastive decoding with en- hanced performance. However, this contrastive decoding method introduces significant computational overhead in inference, as the MIL.M needs to be executed multiple times for different contrastive referring prompts to generate the response for each input (I, r, xr) (see 1). To reduce com- putational complexity, inspired by the success of knowledge distillation in large models (Wan et al., 2023), we propose distilling the results of the original eulti-execution model pm. trained on D. into a single-execution model é., which requires only one pass of the MLLM for each input (I, r,,,, thus reducing computational cost.
Motivation.
Distillation Loss. We use the trained Φm to distill the adapter-based dingle-executi model é. on D. During this process, the parameters of the prompt extraction network and MLLM in 6. are copied from om and frozen, while only the adapter generation network AGN and projection layers W updated. The distillation loss function Ldis is calculated as:
model
la [anot]
Adapter Module for in Figure 4, the single-execution model o, retains the core structure of Φm. incorporating prompt extraction network and an MILM. To facilitate finetuning during the distil- lation process, 4 adapter modules are inserted at evenly distributed intervals throughout the LLM in Φs. As il- lustrated in Figure 4, each adapter module consists of N. tokens A= ( which are concatenated with the hid- den states output by the preceding LLM layer and processed
Model. As shown
Ldis
Σ
KL. (p (-1aze) Po. (-122))
(4)
laoot
where KL refers to Kullback-Leibler divergence. acot denotes each generated answer in dataset D (see Sec 3.3), P (-lag)) refers to the prediction probability dis- tribution in the i-th time step of om obtained by Eq. Pe. (- ((aaee) is directly generated in a single step, i.e., Softmax (MLLmax(MLLM(,,,, r, a,ace))
3 At the 8th, 16th, 24th, and 32nd layers of the 7B model, and the 10th, 20th, 30th, and 40th layers of the 13B model
Inpainting- Based AGN Optimization. In addition to Caiss we also propose an inpainting loss to optimize the adapter
5

---

CPCF: A Cross-Prompt Contrastive Framework for Referring Multimodal Large Language Models
generation network AGN. The motivation stems from the fact that contrastive decoding in 6n enhances performance by comparing the regions associated with the input prompt r against those corresponding to different contrastive prompts (fn J. Therefore, to effectively distill 6m into Φs. the adapter modules integrated into the MLLM of o₂ are opti- mized to also capture and encode these regional differences. For each training image I, let R represent the region cor- responding to the input referring prompt r. We construct region R., with the same shape as R. centered at each contrastive prompt f.. and replace R., in I with R. to gen- erate a perturbed image n In is inpainted back to I using a frozen stable diffusion model SD. The inpainting-base loss function Linp is then calculated as:
4.2. Comparison with Other Methods
Evaluated Tasks and Benchmarks. Following prior work (You et al., 2023; Wu et al., 2024), we evaluate our approach against other methods across four tasks and benchmarks: Referring Object Classification (ROC): This task requires the model to identify the object within the referring re- gion. We use the benchmark provided by (Wu et al., 2024), which generates 1748 questions from the LVIS validation set (Gupta et al., 2019) for evaluation. (2) Referring Text Classification (RTC): In this task, the model selects the text candidate corresponding to the referring object from sev- eral options. The evaluated benchmark is from (Wu et al., 2024), generated from the COCO-Tec dataset (Veit et al., 2016). (3) Referring Description (RD): This task is eval- uated on RefCOCOg (Kazemzadeh et al., 2014), requiring the model to generate textual description of the referring hounding box region. To accelerate testing, 1000 samples are randomly selected from this benchmark for evaluation. (4) Referring Reasoning and Referring Captioning: These tasks are evaluated on Ferret-Benchh as proposed by (You et al., 2023), which includes more challenging description and reasoning problems. Please refer to Appendix Sec.B B for more details, including evaluation metries and input instructions used during testing.
Ppos dR.. Piegg I. (En), I, =SD (Ir: Linp = En (1hn - 1112-11100 1| - -1.11))
Ru, Pposs Pieg
(5)
where dR. refer to a textual description for region R., gen- erated by the referring MLLM. L denotes a linear layer, and its input E, is generated by AGN through Eq.3. Ppos and Preg respectively refers to the positive and negative prompt used in stable diffusion. The idea is straightforward: Limp can be minimized if E., contains differential information that describes what is present in R but absent in R,,. This information can serve as a negative prompt to prevent the inpainting result of R -> R., (during in - T) from preserv- ing the characteristics of R. By leveraging Limp, AGN can be optimized to extract more comprehensive and accurate regional difference information, facilitating the distillation process from o. to Φg. Finally, the overall distillation loss is computed as the sum of Lais (Eq,4) and Limp (Eq.
Comparison Results. Results for the Referring Object Classification (ROC), Referring Text Classification (RTC) and Referring Description (RD) tasks are presented in Table 1, and comparison results for the Referring Reasoning and Referring Captioning tasks are shown in Table 2. Compared to other referring MILM approaches such as Ferret (You et al., 2023), our method consistently achieves the best per- formance across all benchmarks with significant advantages, demonstrating the excellent effectiveness of our approach and its strong generalization capability across diverse tasks.
4. Experiments
Evaluation of Errors Related to Misleading Regions. In Appendix Sec. C.2, we provide a more detailed comparison with other methods, specifically focusing on errors caused by the model incorrectly misled by regions adjacent or similar to the input visual prompt.
4.1. Implementation Details
We follow the basic model structures of Ferret, using CLIP- ViT-L/14 336p as the image encoder and Vicuma as the L.M. As described in 1e, the entire training process consists of three steps. In the first step, initial training is conducted in the same manner as Ferret. In the second step, from the Open Images dataset (Kuznetsova et al., 2020), we sample 50K images that were not used in the first step and are not included in the test set, synthesize corresponding questions and answers to generate dataset D, and train the prompt extraction network for 50K steps (details in Sec.3.3). This step takes about 30/50 hours for the 7B/13B models on 8 A100 GPUs In the third step, the model is distilled into a single-execution model o, by training for 50K steps on D (details in Sec.3.4). This step takes about 25/40 hours for the 7B/13B models. The number N₂ of tokens in each adapter module of model de is set to 32. or in Eq.1 is 0.5.
Comparison with Other Contrastive Decoding Methods. Based on the Ferret model, we further compare our method with other contrastive decoding approaches on the referring object classification task. As shown in Table 3, benefiting from our task-tailored designs and the automatically learned contrastive prompts, our method achieves the best perfor- mance, demonstrating its significant advantage over existing contrastive decoding methods in addressing referring tasks.
4.3. Ablation Study
We conduct ablation study on the referring object classifi- cation (ROC) task. Due to paper length limitation, more
6

---

CPCF: A Cross-Prompt Contrastive Framework for Referring Multimodal Large Language Models
Table Comparison results on the Referring Object Classification (ROC) task, Referring Text Classification (RTC) task, and
Referring Description (RD) task.
ROC
RTC
RD
Models
Box
Mask
Scribble
Point
Mask
B@4
1oamos2-1.68 (Peng al., 2023)
55.17
16.55
38.68
23,90
66.99
19.76
GPT4RoI-7B (Zhang al., 2023)
58.59
54.23
36.12
24.83
69.17
20.05
Shikra-7B (Chen al., 2023)
64.60
56.27
50.07
36.55
26.43
73.80
20.09
CogVLM-17B (Wang et al., 2023)
68 44
52.29
39.05
27.75
75.86
20.51
GLaMM-7B (Rasheed et al., 2024)
69.93
53,92
40.12
28.08
75.80
21.75
Osprey-7B (Yuan ul., 2024)
72.15
74.19
55,98
60.45
40.50
27.33
79.18
21.47
Ferret-BB (You et al., 2023)
71.71
72.39
71.58
68.54
55.47
56.34
41.30
27.18
78.36
21.62
Ferret-133 13B (You et al., 2023)
72.83
73.75
72.33
69.70
56.24
58,90
41.78
28.22
80.24
21.91
CPCF-7B (Ours)
78.37
79.55
77.97
77.14
62.69
63.38
44.23
29.79
82.41
22.67
CPCF-13B (Ours)
79.09
80.72
78.85
78.19
63.90
64.28
45.11
30.55
84.52
23.07
Table 2. Comparison results the Referring Reasoning task and Referring Captioning tasks within the Ferret-Bench benchmark.
Table 4. Ablation study of key components in our framework.
Inference Time (Methed / Ours)
Referring Captioning
Referring Reasoning
Methods
Box
Mask
Models
Ours
78.37
79.55
1.00
Kosmos2-1 1-mmos2-1.6B (Peng et al., 2023)
51.8
33.7
Ours w/o prompt extraction network
74.33
75.08
0.98
Shikra-7B (Chen et al., 2023)
46.0
41.6
Ours wlo self-training
73.19
74.90
1.00
CogVL.M-17B (Wang al., 2023)
67.1
67.6
Ours w/o distillation
79.08
80.15
3.79
Osprey-7B 7B (Yuan et al., 2024)
72.2
67.8
7B (You et al., 2023)
68.7
67.3
Table 5. Ablation study of prompt extraction network.
Ferret-3BB (You al., 2023)
70.6
68.7
Methods
Box
Mask
Scribble
Point
Ferret-v2-7B (Zhang al., 2024)
79.9
Ferret-v2-133 (Zhang et al., 2024)
79.6
79.4
Ours
78.37
79.55
77.97
77.14
Ours w/o initialization from BLIP2
76.93
77.88
76.44
75.90
CPCF-7B (Ours)
83.5
82.9
Ours w/o Mr
76.50
77 77.94
76.13
75.83
CPCF-13B (Ours)
84.0
Ours w/o M4 Ours w/o random noise for selection
75.77
78.02
76.09
75.41
77.51
78.49
77.04
76.06
Table 3. Comparison results with other contrastive decoding meth- on Referring Object Classification (ROC) task.
formance becomes slightly better, the computational cost greatly increases by more than threefold. These results demonstrate the effectiveness of our methods in improving performance and reducing computational overhead.
Models
Box
Mask
Scribble
Point
Femet-7B
71.71
72.39
71.58
69.54
Femet-7B VCD (Leng et al., 2023)
74.90
75.08
73.04
72.40
Fe ICD (Wang al., 2024)
74.55
75.73
73.58
72.05
Fenet-7B CRG (Wan al., 2025)
75.40
76.20
75.24
73.39
Ablation Study of Prompt Extraction Network. We propose several designs enhance the effectiveness of the prompt extraction network (Sec.3.2) in generating con- trastive prompts, including: (1) Initializing weight parame- ters using BLIP2's Q-former. (2) Incorporating the semantic similarity map M, and relative distance map Ma as prior information. (3) Adding random standard Gaussian noise A for more robust prompt selection. As shown in Table removing any of these design components leads signif- icant decline in performance, demonstrating the rationale and effectiveness of our proposed designs in this network.
CPCF-7B (Ours)
78.37
79.55
77.97
77.14
results presented in Appendix.
Effectiveness of Key Components. We conduct exper- iments to verify the key components of our framework, with results presented Table 4. As shown in the table, model's performance significantly decreases under the following conditions: (I) Excluding the prompt extraction network (Sec.3.2) and using a prior-based manual method to generate the contrastive prompts in Eq. (2) Removing the proposed self-training method (Sec.3.3) and directly train- ing the prompt extraction network on the original dataset D. We also evaluate (3) Removing the distillation method (Sec.3.4) and directly using the multi-execution approach described Eq.1 for testing. Although the model's per-
Ablation Study of Self-Training. We further evaluate the designs in the proposed self-training method (Sec.3.3) through the following experiments: (1) Generating ques- tions for the data in D directly, without using the RAG method. (2) Removing either the global similarity s) the local similarity sf during retrieval. (3) Replacing the DPO

---

CPCF: A Cross-Prompt Contrastive Framework for Referring Multimodal Large Language Models
Table 6. Ablation study of self training.
Table 8. Ablation study of distillation.
Methods
Box
Mask
Scribble
Point
Methods
Mask
Scribble
Point
Ours
78.37
79.55
77.97
77.14
Ours
78.37
79.55
77.97
77.14
Ours w/o RAG for question generation
75.98
76.52
75,41
75.03
Ours w/o AGN
74.71
75.83
74.08
73.73
Ours w/o s for retrieval
76.99
78.22
76,68
76.51
Ours who Cass (Fq.4)
72.56
74.10)
71.39
71.45
Ours w/o sf for retrieval
77.71
78.80
77.55
76.70
Ours who Limp (Eq.5)
75.95
76.80
75.52
74.49
Ours w/o DPO w/ CE loss
75.47
76.08
75.22
75.01
Ours w/o distillation w/ direct training
74.26
74.90
73,98
72.10
Question: What the color of hat [raglonj?
Question: What the textinn [raglon]?
Table 7. Effectiveness of different methods to leverage D.
STOP
Methods
Box
Mask
Scrihble
Point
Ferret: The color of the het
Ferret:
The text Stop coution
Buseline Ferret (w/o using D)
71.71
72.39
71.58
68.54
Ours: The color of the hot jehite
Ours: The test the reg ion hitopp
Use D through our method (Scc.3.3)
78.37
79.55
77.97
77.14
Use 1 to finetune Ferret
74.22
74.37
73.97
70.70
Ferret + CRG (wo fine-tuning)
75.40
76.20
75.33
74.89
Ferret + CRG (w/ fine- nae-tuning on D)
75.95
77.06
75.90
75.37
Figure 5. Visualization results of the contrastive prompts generated by the prompt extraction network and the quantitative comparison between our method and Ferret. The contrastive prompts are represented by purple dots. The incorrect contents generated by Ferret and the corresponding correct contents generated by our method are highlighted in red and green respectively.
in Eq.2 2 with conventional cross-entropy loss, using aoot as pseudo-label. As shown in Table 6, each of these modi- fications to the original method can result in performance drop, demonstrating the effectiveness of our designs.
4.4. Discussion of Computation and Parameter
Different Methods to Leverage D. In our self trainini method, we generate question-answers from D to optimize the prompt extraction network, thereby enhancing the ef- fectiveness of contrastive decoding. We also test directly finetuning the Ferret D without utilizing contrastive de- coding and the prompt extraction network. As shown in Table 7, while this approach achieves performance slightly better than the baseline Ferret, it still remains significantly lower than our method for leveraging D. This demonstrates that the improvement of our method is not simply due to the additional training data provided by D. but is primarily at- tributed to our carefully designed approach that leverages D to optimize the prompt extraction network and enhance the effectiveness of contrastive decoding. We also fine-tuning Ferret-7B on D using the contrastive decoding method CRG. As shown in the table below, this method achieves an accu- racy of 75.95% on the ROC-Box scenario, outperforming baseline Ferret (71.71%) %) and naive fine-tuning (74.22%) ) but still falling significantly short of our CPCF method (78.37%). This demonstrates the significant advantage of our proposed automatic contrastive prompt extraction ap- proach over the manually designed strategy in CRG.
Compared to the baseline Ferret, our method introduces additional computational and parameter costs due to the prompt extraction network and adapter generation network. However, both modules are highly lightweight, resulting in only a 2.1% increase in parameter count and a 4.3% in- crease in average inference time compared to Ferret, while achieving significantly improved performance (see Table 1). Moreover, compared to directly using the model without distillation, our model requires only 26.4% of its average inference time as the MLLM needs to be executed only once, while maintaining highly comparable performance (see Table 4 "ours w/o distillation"). This high- lights the high efficiency and effectiveness of our method.
4.5. Visualization Results
In Figure 5, we present two types of visualization results: (1) the contrastive prompts generated by the prompt extraction network; and (2) the quantitative comparison of responses generated by our method and Ferret. In these examples, the generated contrastive prompts are primarily located in mis- leading regions adjacent to the target region (e.g., clothing next to a hat) or similar to it (e.g., another person's hat). Fer- ret frequently generates incorrect answers related these misleading regions rather than completely tailored to the target region (e.g., in the left example, misjudging the hat's color as the color of the adjacent clothing). In contrast, our framework, enhanced by the proposed cross-prompt con- trastive methods, significantly improves performance and reduces errors. These results highlight the advantages of our approach over Ferret. Due to space limitations, more
Ablation Study of Distillation. We conduct the following experiments to evaluate different components of the pro- posed distillation method: (1) Removing the adapter genera- tion network AGN and directly using learnable embeddings as the adapter tokens A. (2) Removing Laie (Eq.4) or Limp (Eq.5) from the loss functions for distillation. (3) Training 6. directly instead of distillation. The performance decline shown in Table demonstrates the critical importance of the proposed AGN, loss functions and distillation method.
8

---

CPCF: A Cross-Prompt Contrastive Framework for Referring Multimodal Large Language Models
visualization results are provided in the Appendix Figure 8.
contrast decoding. arXiv preprint arXiv:2403.00425, 2024b.
5. Conclusion
Gupta, A., Dollar, P., and Girshick, R. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition, pp. 5356-5364, 2019.
This paper introduces CPCF, novel and effective cross- prompt contrastive framework for referring MLLMs. This framework incorporates a prompt extraction network for automatically generating prompts used contrastive de- coding, self-training method that leverages unlabeled data for enhanced training, and distillation approach reduce the additional computational overhead introduced by con- trastive decoding. Extensive evaluation results on multiple benchmarks demonstrate the superiority of our framework over existing methods. We believe this work can provide valuable insights for developing more robust, efficient, and high-performing referring MILLM methods.
Ji, D., Zhu, L., Gao, S., Xu, P., Lu, H., Ye, J., and Zhao, F. Tree-of-table, Unleashing the power of ilms for en- hanced large-scale table understanding. arXiv preprint arXiv:241 1.08516, 2024.
Kazemzadeh, S., Ordonez, V., Matten, M., and Berg, T. Referitgame: Referring to objects in photographs of natu- ral scenes. In Proceedings of the 2014 conference on em- pirical methods natural language processing (EMNLP), pp. 787-798, 2014.
Acknowledgment
T. C. acknowledges support from Dream Set Off Kunpeng & Ascend Seed Program.
Kim, Y., Kim, H. J., Park, C., Park, C., Cho, H., Kim, J., Yoo, K. M., Lee, S.-g., and Kim, T. Adaptive contrastive decoding in retrieval-augmente generation for handling noisy contexts. arXiv preprint arXiv:2408. 2000020080000844 01084, 2024.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., al. The open images dataset v4: Uni- fied image classification, object detection, and visual re- lationship detection scale. International journal of computer vision, 128(7): 1956--991,,,,11 2020.
References
Anderson, P., Fernando, B., Johnson, M., and Gould, S. Spice: Semantic propositional image caption evaluation. In Computer Vision-ECCH 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 11-14,2 2016, Proceedings, Part 14, pp. 382-398. Springer, 2016.
Leng. S., Zhang, H., Chen, G., Li. X., Lu, S., Miao, C., and Bing. L. Mitigating object hallucinations in large vision- language models through visual contrastive decoding. arXiv preprint arXiv:2311. 1..22.1.162.... 2023.
Bai, J., Bai, S., Yang. S.. Wang, S., Tan, S., Wang. P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A frontier large vision- language model with versatile abilities. arXiv preprint arXiv:2308.1 12966. 2023.
Li, B., Zhang, Y., Chen, L., Wang, J., Pu, F., Yang. J., Li, C., and Liu, Z. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306. 05400. 2023a.
Banerjee, S. and Lavie, A. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on in- trinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Boot- strapping language-iman pre-training with frozen im- age encoders and large language models. arXiv preprint arXiv:2301.1259) 2023b.
Li, X. L., Holtzman, A., Fried, D., Liang, P., Eisner, J., Hashimoto, T., Zettlemoyer, L.,, and Lewis, M. Con- trastive decoding: Open-ended text generation as opti- mization. arXiv preprint arXiv:22 20. 15097, 2022.
Chen, K., Zhang, Z., Zeng, W., Zhang, R., Zhu, F., and Zhao, R. Shikra: Unleashing multimodal llm's 'imss referential dialogue magic. arXiv preprint arXiv:2306. 15195, 2023.
Chen, T., Yu, C., J., Zhang, J., Zhu, L., Ji, D., Zhang, Y., and
Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. arXiv preprint arXiv:2304. 08485, 2023.
Zang, Y., Li, Z., and Sun, L. Reasoning3d-groone reasoning in 3d: Fine-grained zero-shot open- voccabbayyyl 3d reasoning part segmentation via large vision- language models. arXiv preprint arXiv:2405. /9326, 2024a.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-. Bleu: a method for automatic evaluation of machine transla- tion. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Chen, Z., Zhao. Z., Luo, H., Yao, H., Li, B., and Zhou, Hale: Object hallucination reduction via adaptive focal-

---

CPCF: A Cross-Prompt Contrastive Framework for Referring Multimodal Large Language Models
Peng, Z., Wang, W., Dong, L.,, Hao, Y., Huang, S., Ma, S., and Wei, F. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 2023.
Xuan, S., Guo, Q.. Yang, M., and Zhang. S. Pink: Unveiling the power of referential comprehension for multi-modal Ilms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13838- 13848, 2024.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. Er- mon, S., and Finn, C. Direct preference optimization: Your language model secredly reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
You, H., Zhang, H., Gan, Z., Du, X., Zhang, B., Wang, Z., Cao, L.,, Chang, S.-F., and Yang, Y. Ferret: Refer and ground anything anywhere at any granularity, 2023.
Yuan, Y., Li, W., Liu, J., Tang, D., Luo, X., Qin, C., Zhang, L., and Zhu, J. Osprey: Pixel understanding with visual instruction tuning. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Puttern Recognition, pp. 28202-28211, 2024.
Rasheed, H., Maaz, M., Shaji, S., Shaker, A., Khan, S., Cholakkal, H., Anwer, R. M., Xing, E., Yang, M.-H., and Khan, F. S. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13009- 13018, 2024.
Yue, T., Cheng, J., Guo, L.,, Dai, X., Zhao, Z., He, X., Xiong, G., Lv, Y., and Liu, J. Sc-tune: Unleashing self-consistent referential comprehension in large vision language mod- els. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 13073-13083, 2024.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A,, Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302. 2023.
Zhang, H., You, H., Dufter, P., Zhang, B., Chen, C., Chen, H.-Y., Fu, T.-J., Wang, W. Y., Chang, S.-F., Gan, Z., al. Ferret-v2: An improved baseline for referring and grounding with large language models. arXiv preprint urXiv:2404.07973 2024.
Vedantam, R., Lawrence Zitnick, C., and Parikh, D. Cider: Consensus- based image description evaluation. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566 4575, 2015.
Veit, A., Matera, T., Neumann, L., Matas, J., and Belongie, S. Coco-text: Dataset and benchmark for text detec- tion and recognition in natural images. arXiv preprint 16X1v0/60.00144 2016.
Zhang, S., Sun, P., Chen, S., Xiao, M., Shao, W., Zhang, W., Liu, Y., Chen, K., and Luo, P. Gpt4roi: Instruction tuning large language model on region-offinteres.. arXiv preprint arXiv:2307.03660 2023.
Wan, D., Cho, J., Stengel-Eskin, E., and Bansal, M. Con- trastive region guidance: Improving grounding vision- language models without training. In European Confer- ence on Computer Vision, pp. 198-215. Springer, 2025.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.1033 2023.
Wan, Z., Wang, X., Liu, C., Alam, S., Zheng, Y., Liu, J., Qu, Z., Yan, S., Zhu, Y., Zhang, Q. et al. Effi- cient large language models: A survey. arXiv preprint arXiv:2312.03866 2023.
Zhu, L., Chen, T., Ji, D., Ye. J., and Liu, J. Llafs: When large language models meet few-shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 3065-3075, 2024a.
Wang, W., Lv, Q. Yu, W., Hong, W., Qi, J., Wang, Y., Ji, J.. Yang, Z., Zhao, L., Song, X., et al. Cogylm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03073 2023.
Zhu, L., Ji, D., Chen, T., Xu, P., Ye. J., and Liu, J. Ibd: Alleviating hallucinations in large vision-language models via image-biased decoding. arXiv preprint arXiv:2402.1 18476, 2024b.
Wang, X., Pan, J., Ding, L., and Biemann, C. Miti- gating hallucinations in large vision-language models with instruction contrastive decoding. arXiv preprint 27Xiv:2403.18715, 2024.
Zhu, L., Chen, T., Ji, D., Ye, J., and Liu, J. Not every patch is needed: Towards a more efficient and effective backbone for hideo-based person re-identification. IEEE Transactions on Image Processing, 2025a.
Wu, M., Cai, X., Ji. J., Li, J., Huang, O., Luo, G., Fei, H., Jiang, G., Sun, X., and Ji, R. Controlmllm: Training-fre free visual prompt learning for multimodal large language models. arXiv preprint arXiv:2407.2155. 2024.
Zhu, L., Chen, T., Xu, Q., Liu, X., Ji, D., Wu, H. Soh, D. W., and Liu, J. Popen: Preference- based optimization and ensemble for baldd reasoning segmentation. arXiv preprint wxXiv:2504.00644 2025b.
10

---

CPCF: A Cross-Prompt Contrastive Framework for Referring Multimodal Large Language Models
A. More Details of Method
A.1. Extension to Multi-visual-pronp Scenarios
In the main paper, to simplify the illustration,, we describe our method for the inputs containing only one visual prompt r. Our method can be easily adapted to multi-r scenarios with the following extensions:
Given multiple input visual prompts {r:) we generate contrastive prompts (f W for each r2 using the method described in Sec.3.2. During the MILMM's contrastivedeeoding--bee baasd prediction process, we sequentially replace each r; with each of its corresponding Êi,n while keeping the other visual prompts unchanged, generating a prediction from the MILMM and contrasting it with MILMM's original prediction from {r:) Finally, the contrastive results obtained from all (f >. NN are averaged to produce the final output.
Contrastive Decoding Prediction.
n=1
RAG in Self-training. In the self-training method introduced in Sec.3.3, we employ a RAG strategy retrieve a similar D' (I'r'r )'} from 2 to guide question generation for each { F} in D. During this process, we compute an average CLIP feature of region r' as a local representation fí and that of F as fi. then calculate the cosine similarity si between each fi and fi for retrieval. In the actual implementation, to enhance the model's capability in multi-r scenarios, for each / in a subset of D. we randomly sample N, (N, > 1) bounding boxes and convert them into input visual prompts ( We compute f for each ry. For each sample D3 in D that contains N, visual prompts {re,) N- we also compute the corresponding ( ) Next, we match }/ with " using bipartite matching and compute the cosine similarity for each matched pair. The average similarity over all pairs serves as the local score s/. which is used for retrieval in the multi-r scenarios.
J==
daaperr Module for Distillation. As illustrated in 3.4, in the single-execution model 6.. we use an adapter generation network (AGN) to generate a set of adapter modules A for distillation. When the input contains multiple visual prompts {ri) we also apply Eq.3 to generate an E,,, for each contrastive prompt fi,n of every rú. The resulting { {EE., n= i== are then summed and passed through linear layers W to obtain A. During this process, in addition to its original inputs [fr: f e). the AGN also receives an additional embedding computed as the position embedding of {ra} 's index This embedding is concatenated with [f;; : :e to serve as the AGN's input, providing explicit information about which {ri) the fi.. corresponds to.
N; Nr )
A.2. MILM Prompt for Description Generation
As illustrated in Sec.3.3 of the main paper, we employ the RAG method to generate question for each image Î in D. To achieve this, we retrieve an image I; and its visual prompt r; from the original dataset D. Subsequently, the MLLM is prompted to generate textual descriptions d, and d,, which correspond to J; and the region of r,, respectively, to guide the question generation process. The detailed prompts for generating d, and d, are as follows:
Prompt for generating d, from T;: Describe this image in detail.
Prompt for generating d, from r; Please provide a description of the region <prompt Ti> in a sentence.
A.3. Answer Generation with Chain-of-thought
accurate answer As detailed in 3.c.3.3 of the main paper, we use ofain-of-tthough (CoT) prompt Pe to generate
dcot for each image-prompt pair ( and question q. i.e., MLLM(T, i, q, Pc). The prompt Pe is written as: Please think step-by-step to answer the question. First, describe the image and indicated region. Next, explain what information is needed to answer the question. Then, identify this information from the image or world knowledge. Finally, based on this reasoning, provide the answer.
We use GPT-4o evaluate whether deot or directly generated without using Pe is more accurate. Among 2000 randomly sampled instances {f, f. q). GPT-4o determines that aoot is better in 91% % of the cases. This result validates the rationale of using {ā, acot) } pairs for DPO training.
B. More Details of Tasks and Benchmarks
B.I. Input Instruction for Evaluation
As illustrated in Sec.4.2 of the main paper, we evaluate the model's 's performance on four tasks and benchmarks. The input instructions for the referring reasoning and referring captioning tasks on Ferret-Bencd are directly included in the dataset.

---

CPCF: A Cross-Prompt Contrastive Framework for Referring Multimodal Large Language Models
Table 9. Evaluation of errors caused by adjacent regions.
Table 10. Evaluation of emorr caused by similar regions.
Models
Accuracy
Models
Accuracy
Kosmos2-11 1.6B (Peng et al., 2023)
51.5
Kosmos2-1 1.68 (Peng el al., 2023)
39.6
Shikr--BB (Chen el al., 2023)
66.8
Shikra-BB (Chen et al., 2023)
52.3
CugVLM-17B (Wang et al., 2023)
67.4
CogVLM- 17B (Wang et al., 2023)
58.8
Osprey-7B (Yuan et al., 2024)
73.1
Osprey-7B (Yuun et al., 2024)
67.8
GlaMM- 7B (Rasheed et al., 2024)
74.8
GI aMM-7B (Rasheed et al., 2024)
62.5
Ferret-7B (You et al., 2023)
73.9
Ferret-71 (You et al., 2023)
67.0
Ferret-13B (You et al., 2023)
76.2
Ferret 13B (You et al., 2023)
70.9
CPCF-7B (Ours)
91.3
CPCE-7B (Ours)
83.9
CPCF-13B (Ours)
92.0
CPCF-13B (Ours)
85.7
The MLLM's input instructions for the other three tasks and benchmarks are as follows:
Referring Object Classification (ROC): Is the object sprompt r> <class A> or a <class B>?
Referring Text Classification (RTC): Is the text <prompt of the image '<Text A>' or <Text B>'? please select only one.
Referring Description (RD): Can you provide description of the region <prompt T> in a sentence?
B.2. Evaluation Metries
The referring object classification (ROC) task and the referring text classification (RTC) task are both constructed as binary classification tasks. Therefore, we directly use accuracy as the metric to evaluate model performance. For the referring description (RD) task, we follow previous methods (Wu et al., 2024) and adopt four metrics: BLEU @4 (B@4) (Papineni et al., 2002), METEOR (M) (Banerjee Lavie, 2005), CIDEr-D (C) (Vedantam al., 2015), and SPICE (S) (Anderson al., 2016). For the referring reasoning and referring captioning tasks on Ferret-Bench, we use the same approach as Ferret (You al., 2023), prompting GPT-4 for scoring. Readers refer to (You et al., 2023) for more details on the evaluation methods used for Ferret-Bench.
C. More Experimental Results
C.1. Evaluation of RAG-Generated Questions
As illustrated in Sec.3.3 of the main paper, for each image-prompt pair {i, in D, we employ the RAG method to retrieve a similar r from D and use the corresponding question guide generation of questions for {I, This approach is motivated by our observation that directly prompting the MLLM to generate questions often results in errors and hallucinations, producing questions unrelated {I, f). In contrast, providing the MILM with similar example from D for reference can significantly improve reliability. To validate this, we use GPT-4o to score the questions generated with and without RAG on scale from 0 to 10 according to their relevance to the given {I. Among 2000 randomly sampled f) from D. questions generated using RAG achieve an average score of 9.3, significantly higher than the 6.4 achieved without RAG. This demonstrates the rationality of our RAG-based approach.
C.2. Errors Caused by Misleading Regions
As indicated in the main paper, referring MILMS often exhibit suboptimal performance due to the confusion caused by misleading areas adjacent to or similar to the target region. To demonstrate the effectiveness our method addressing this issue, we conduct the following validation experiments.
Errors Caused by Adjacent Regions. To evaluate the error rate of the model being misled by areas adjacent the indicated visual prompt, sample 2000 images from the LVIS dataset. For each image, select two adjacent but vategory-differen bounding boxes, b and b2, belonging to classes and c2, respectively. Using b as the visual prompt, we ask the referring MLLM "Is the object <prompt b, > <class Ci > or a <class c2>?", , and then calculate the accuracy based on its responses. As shown in Table 9, our method achieves the best performance, significantly outperforming the baseline method Ferret and all other advanced methods used for comparison. This demonstrates the high effectiveness of our approach in mitigating errors caused by regions adjacent the input visual prompt.
12

---

CPCF: A Cross-Prompt Contrastive Framework for Referring Multimodal Large Language Models
81
81
79
79
Accrrac
78
Accuracy
78
77
76-
Box
76
Box
75
Mask
Mask
Scribble
Scribble
74
Point
Point
73
73
32
Number of Adupter Modules
Number N, of Tokens Each Adapter Module
Figure Number of adapter modules.
Figure Number N. tokens each adapter module.
Table 11. Number of queries input the prompt extraction network.
Table 12. Ablation study results
Table 13. Comparison results with general MLLMs on Berret-Benchs
a used in Eq.
Query Number
ROC-Box
ROC-Mask
a
ROC-Box
ROC-Mask
Method
Referring Captioning Referring Reasoning
16
76,68
77.09
0.1
77.91
79.24
V1.255
80.4
79.2
32
77.75
78.43
0.5
78.37
79.55
Qwen2.5-VL
79.9
80.5
64
78.37
79.55
78.31
79.53
CPCF (Ours)
83.5
82.9
128
78.50
79.98
Errors Caused by Similar Regions. To evaluate the error rate caused by the areas similar to the input visual prompt, we sample 2000 images from the LVIS dataset, each containing two bounding boxes, b and b2, belonging to the same category c. We use GPT-4o to generate descriptions t and f2 respectively highlighting a different attribute between b and b2 (e.g.. color, pose, position, and relationships with surrounding objects). After that, we ask the referring LLM "Which text correctly describe the properties of object <prompt b,>, '<descripp iedeccriptioo
41>' or '<descripe pt i 12>'? Please select only one." The results presented in Table 10 highlight the significant superiority of our CPCF compared to previous methods, demonstrating the effectiveness of our approach in avoiding errors caused by misleading areas similar to the visual prompt.
C.3. Ablation Study of Hyperparameters
Number of Adapter Modules. In our method, to facilitate training during the distillation process, N adapter modules inserted at evenly distributed intervals throughout the MILM of the aigle-executton model Φ.. We conduct an ablation study on the number N of adapter modules, with the results presented in Figure 6. When N < 4, the model's performance improves as the number of adapters increases. However, when N > 4, the performance saturates, and further increasing the number of adapters yields only minimal improvement. Based on these findings, we use 4 adapter modules in our method.
Number N. of Tokens in Each Adapter Module. We further evaluate the effect for the number N₂ of tokens in each adapter module. As shown in Figure 7, similar trend is observed: when N. < 32, increasing the number of tokens significantly improves the model's performance; however, when N. > 32, the performance tends to saturate. Based on these observations, we set the number of tokens in each adapter module to 32.
Number of Learnable Queries Input to the Prompt Extraction Network. In our proposed prompt extraction network, a set of learnable query embeddings are input into the image encoder. As shown in Table 11, when the number of such learnable queries is too small, the network may miss critical information, leading to performance degradation; whereas when the number is too large, the model performance saturates, resulting in limited improvement if further increasing the query number (only +0.17 accuracy on ROC-Box when the query number increases from 64 to 128). Based on these results, choose 64 as the number of input query embeddings in our method.
Ablation Study of in Eq.1. As shown in the Table 12, when a is set to 0.1, 0.5, or 1, the model achieves ROC-box accuracy of 77.91, 78.37, and 78.31, respectively, demonstrating stable performance across a range of values and consistently outperforming the previous SOTA result (72.83). In fact, the setting of a is directly adopted from existing contrastive decoding approaches, and we found that it can already work very well without requiring specific modifications.
13

---

CPCF: A Cross-Prompt Contrastive Framework for Referring Multimodal Large Language Models
C.4. Comparison with General MLLMs
Some recent general MLLMs such Qwen and Intern VL also supports referring tasks. We further compare our CPCF with these MILMs and the results on Ferret-Bench are presented in Table 13. As shown by the table, CPCF-7B outperforms both ViternVL2.5540 and Qwen2.5-VL-7B significantly, demonstrating the advantages and effectiveness of our novel method benefiting the novel designs tailored to the referring tasks proposed in this paper.
Question: Which airline does the aircraft in [region] belong to?
Ferret: The aircraft belongs to Singapore Airlines.
Ours: It belongs to Cathay Pacific.
Question: What is this player [region] doing?
This player is attempting to steal the ball from another player.
Ferret:
14
Ours:
This player dribbling past another player's defense.
What is the role of the man [region] the band?
Question:
Ferret: He is responsible for playing the viola.
Ours:
He is the cellist in the band.
What is the difference between [region and [region2]?
Question:
12
Ferret: [region 1] is dog while [egionn2 cat.
Ours:
Their fur colors are different. [region1] is white, and [region2] yellow.
Figure 8. More visualization results of the contrastive prompts generated by the prompt extraction network and the quantitative comparison hetween our method Ferret. The incorrect contents generated by Ferret and corresponding correct contents generated by our method are highlighted in green respectively.
14


Reviews:
[{"review_id": "SRwjWEIa3t", "reviewer": "Reviewer_hUfo", "summary": "This paper introduces CPCF, a novel framework designed to enhance referring capabilities in MLLMs. The method leverages a cross-prompt contrastive strategy, in which responses generated from visual prompts are contrasted with those from misleading regions. The framework further incorporates a prompt extraction network to identify contrastive prompts, a self-training approach to synthesize training data from unlabeled images, and a distillation technique to reduce computational overhead.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1 Discussion with other contrastive decoding (CD) methods (like CRG)**\n\n* Although both methods use CD, the contrastive targets are different. CRG constructs contrastive targets by removing the target region directly from the image, which may severely disrupt the image’s integrity—especially when the target region is large—potentially leading the model to capture noisy information during contrast. Differently, our method does not alter the image content, mitigating this issue and resulting in better accuracy.\n* CRG and most other CD methods significantly increase computational cost, as each contrastive object must be processed separately by the MLLM. Our CPCF is the first CD method to introduce a distillation technique to address this issue, reducing inference costs by 73%. \n* Even when CRG is equipped with our enhancements (self-training, distillation), its resulting accuracy of 76.74 on ROC-Mask still falls short of CPCF (79.55), further highlighting our advantages.\n\n**Q2 Ablation for components in prompt extraction network** \n\nThese ablation studies are included in main paper Table 5. Thank you! \n\n**Q3 Robustness on synthetic data quality**\n\nPlease see our answer to Reviewer nWKK Q2&Q3. Thanks!\n\n**Q4 Simpler reward-based tuning**\n\nNice insight! It is indeed possible to use other reward-based tuning methods, but they still require preference pairs to compute rewards in DPO or to train a reward model in RLHF. Other methods typically rely on additional models (e.g., GPT) or human annotations to obtain these preference pairs, which can be costly and labor-intensive. This serves as our motivation for using the model itself to generate preference pairs for DPO in this work, which is more cost- and labor-efficient.\n\n**Q5 Improvement**\n\nKindly note that **across all 12 metrics in the 4 benchmarks presented in Table 1 and Table 2, our method outperforms the second-best one by more than 2% in 9 cases—accounting for 75% of the total.** This shows the significant improvements of our method. Also note that our method incurs only very minimal additional computational cost during inference (just +4.3% than the baseline Ferret). The significant performance improvement achieved with such a small increase in inference cost further shows the advantage of our approach.\n\n**Q6 Improvement from data or method?**\n\n**Even without using the better extra dataset, our method can still achieve SOTA**. Using the original dataset $\\mathcal{D}$ without the extra dataset $\\tilde{\\mathcal{D}}$ in our method achieves an accuracy of 74.55 on ROC-Box, which still significantly outperforms previous methods such as Ferret (71.71), Osprey (72.15), and Shikra (64.60). This indicates that **the better extra dataset only serves as a strategy for further enhancement, but is not the sole reason for our method’s strong performance**. The proposed network architecture and training strategy also play a crucial role in achieving high effectiveness.\n\n**Q7 Wall-clock of distillation**\n\nAs indicated in Sec 4.1, distillation requires 25 hours of training for the 7B model on 8 A100 GPUs. Considering that this method reduces inference-time computation by 74% with only a very minimal drop in accuracy (see Table 4), we believe the additional training cost is worthy.\n\n**Q8 Pipeline complexity**\n\nPlease see our answer to Reviewer WGMH's Q1. Thanks!\n\n**Q9 Contrastive decoding (CD) w/o other components**\n\nGood advice! We evaluate a method where all other components are removed, and contrastive prompts are manually sampled from regions adjacent to or similar to the target object for CD. This method achieves accuracies of 74.33 on ROC-Box and 75.08 on ROC-Mask, outperforming the baseline Ferret by 2.62 and 2.69, respectively, but still falling short of our full method by 4.04 and 4.47. These results demonstrate that both CD and other designed components are useful.\n\n**Q10 OOD prompts and images**\n\n* OOD prompts: We remove the data with scribble prompts and train the model using only other prompts. When evaluated with scribble prompts, this model achieves an accuracy of 73.09—lower than our fully trained model (77.97), but still higher than the fully trained Ferret (71.58).\n\n* OOD Images: Images for text recognition are NOT included during training, so the Referring Text Classification (RTC) task in Table 1 is an OOD setting. In this scenario, our method outperforms all previous methods, showing its strong generalization ability.\n\n**Q11 Overlapping or ambiguous prompts**\n\nGood advice! We select 422 crowded scene images with overlapping or ambiguous prompts. Our CPCF achieves an accuracy of 66.98, significantly outperforming Ferret (59.79), showing the high robustness of our method. These results will be included in the revised paper. Thanks! \n\nIn summary, **CPCF significantly improves accuracy (more than 2% in 75% of all metrics), while requiring very minimal additional inference cost (+4.3% than Ferret)**.  We'll illustrate more clearly in paper. Thank you!", "claims_and_evidence": "The central claim is that CPCF improves the robustness and accuracy of referring MLLMs by mitigating the influence of misleading visual regions. While the empirical gains are consistent across benchmarks, some of the claims particularly regarding the “significant differences” from prior contrastive decoding methods like CRG, are overstated. The evidence for CPCF’s advantage over CRG largely hinges on controlled benchmarks and implementation-level choices, without a rigorous apples-to-apples comparison.", "methods_and_evaluation_criteria": "The methodology is generally well-motivated but not without caveats:\n\n- The prompt extraction network is a reasonable architectural choice, but its complexity (e.g., priors like semantic similarity and distance maps, clustering, noise injection) introduces many design decisions with limited ablation or theoretical justification.\n\n- The self-training method relies on synthetic question-answer generation, which raises concerns about data quality. Although the authors attempt to mitigate this via RAG-based retrieval and CoT prompting, the robustness of these synthetic annotations is unclear.\n\n- The use of DPO loss with CoT vs. direct generation is under-motivated; it's unclear if simpler reward-based tuning methods would suffice.\n\n- The evaluation tasks are standard, but most of the experimental results show improvements in the range of ~1–2%, which is modest given the complexity of the proposed pipeline.", "theoretical_claims": "No.", "experimental_designs_or_analyses": "While the experimental section is thorough in scope, some issues limit its interpretability:\n\n- The baseline comparisons (e.g., with Ferret) are fair, but CRG, the most relevant contrastive approach, is not re-implemented with CPCF’s enhancements (e.g., distillation), limiting the claim that CPCF is “significantly different and better”.\n\n- Appendix tables suggest that CPCF’s improvements often arise from better training data rather than the contrastive mechanism per se (see Table 7).\n\n- The computational efficiency claims rely on distillation, but this step is itself complex and introduces extra modules (AGN, adapters). No wall-clock or energy consumption comparisons are provided.", "relation_to_broader_scientific_literature": "The paper is aware of recent work in referring MLLMs and contrastive decoding. However, the differences with works like CRG and ICD are primarily empirical and implementation-level, not conceptual. The line between CPCF and these prior methods is thinner than the authors suggest.", "essential_references_not_discussed": "The paper does not cite work on robust grounding or saliency-based region selection, which could offer alternative approaches to handling misleading regions.", "other_strengths_and_weaknesses": "Strengths:\n\n- Innovative idea of leveraging misleading prompts in a structured contrastive setup.\n\n- Distillation step is well-motivated to reduce inference cost.\n\n- Strong empirical results on standard benchmarks.\n\nWeaknesses:\n\n- Significant complexity in the pipeline with many interdependent components.\n\n- Marginal gains relative to added engineering overhead.\n\n- Limited generalization claims beyond the specific tasks used in training/evaluation.", "other_comments_or_suggestions": "No.", "questions_for_authors": "- Can you isolate the impact of contrastive decoding without prompt extraction, self-training, or distillation?\n\n- How robust are the gains when tested on out-of-distribution prompts or images?\n\n- How would CPCF behave with overlapping or ambiguous prompts (e.g., in crowded scenes)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oJLtYeWRqI", "reviewer": "Reviewer_avef", "summary": "This paper presents CPCF, a cross-prompt contrastive learning framework designed to enhance the performance of referring multimodal large language models (MLLMs). The proposed approach aims to address a key issue in existing referring MLLMs: errors caused by misleading visual regions adjacent to or similar to the target region. The authors introduce three contributions:\n1. A prompt extraction network to automatically identify optimal contrastive prompts.\n2. A self-training method leveraging unlabeled data to improve training quality.\n3. A distillation approach to reduce the computational overhead associated with contrastive decoding.\n\nThe paper is well-organized, the motivation is clearly stated, and experiments demonstrate the effectiveness of CPCF across multiple benchmarks. However, there are some areas that require improvement, as outlined below.\n\n\n**Pros:**\n1. The proposed method is thoroughly evaluated across multiple datasets, including Referring Object Classification (ROC), Referring Text Classification (RTC), and Referring Description (RD).\n2. The results show state-of-the-art performance, significantly outperforming existing models like Ferret, GPT4RoI, and Shikra.\n\n**Cons:**\n\n1. There are many shortcomings in the writing of the paper, including some grammatical issues. For example, in line 022, the phrase \"from...from...from...\" appears. There are also other expression issues throughout the paper. I'm also confused by the naming of the method (CPCF) in this paper. \n2. Figure 1 does not reflect the image editing experiment mentioned on the right side of line 39 in the paper. It is unclear what the authors intend to convey with Figure 1 in the main text and Figure 8 in the supplementary material.\n3. Grounding or referring multimodal large language models is currently a very popular topic, but the authors' introduction to related work is incomplete. They may refer to relevant works mentioned in the recent survey \"Towards Visual Grounding: A Survey\" [1]. For example, besides Ferret and Shikra, which are mentioned in the paper, there are many similar works, such as LLaVA-Grounding [2], Grounding-GPT[3], Next-Chat, Groma, LION, Ferret-V2, and u-LLaVA, etc, most of these related works should be discussed.\n4. The self-training approach using generated data mentioned in this paper is not novel. Similar methods have been widely used in weakly or unsupervised visual grounding works, such as CLIP-VG [4] and VG-annotator [5]. However, the author without include any discussions.\n5. While the paper claims CPCF is efficient, a more detailed breakdown of computational overhead (e.g., FLOPs, memory usage, or inference speed comparisons) would strengthen this claim.\n6. The paper compares CPCF with Ferret + VCD, Ferret + ICD, and Ferret + CRG in Table 3. However, it lacks a qualitative discussion on why CPCF outperforms these methods beyond numerical results.\n7. A deeper comparison with CRG (Wan et al., 2025) is needed, as both use contrastive decoding but differ in implementation. The authors should highlight how CPCF's automatic prompt extraction provides advantages over CRG's perturbation-based method.\n8. Table 4 shows an increase in inference time without distillation. The authors should clarify whether the distillation affects model accuracy trade-offs.\n9. Most importantly, the experiments in this paper are solid, but the work appears to be an combination of existing modules and methods. Techniques such as prompting, pseudo-label self-training, and distillation are commonly used in current research. The paper does not present particularly outstanding innovations that would be sufficient for publication at a top-tier conference like ICML.\n\n--\n\n\n[1] Towards Visual Grounding: A Survey. arXiv preprint arXiv:2412.20206.\n\n[2] Llava-grounding: Grounded visual chat with large multimodal models. In European Conference on Computer Vision (pp. 19-35). Cham: Springer Nature Switzerland.\n\n[3] GroundingGPT: Language Enhanced Multi-modal Grounding Model. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 6657-6678).\n\n[4] VG-Annotator: Vision-Language Models as Query Annotators for Unsupervised Visual Grounding. In 2024 IEEE International Conference on Multimedia and Expo (ICME) (pp. 1-6). IEEE.\n\n[5] Clip-vg: Self-paced curriculum adapting of clip for visual grounding. IEEE Transactions on Multimedia, 26, 4334-4347.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Q1 Writing**\nWe will carefully revise the writing based on your suggestions. “CPCF” is an abbreviation formed from the initials of \"Cross-Prompt Contrastive Framework\". We will clarify this in the revised paper.\n### **Q2 Figures**\nWe will incorporate results of the image editing experiment into Figure 1. Figure 1 aims to highlight a common limitation of prior methods: generating wrong responses tailored to misleading regions that are adjacent or similar to the target object, which is the motivation for this work. Figure 8 in supp provides cases comparing our CPCF with the baseline Ferret, where CPCF produces correct responses while Ferret fails, showing the advantages of our method.\n### **Q3 Related works about grounding and referring MLLMs** \nThank you for pointing out these works! Different from them, our CPCF designs the first automatic contrastive decoding technique for referring MLLMs and with many task-specific designs tailored to this setting (see the answer for Q9 for details). We will cite and include a discussion of these papers in our revised paper.\n### **Q4 Comparison with other self-training methods** \nThank you for recommending these two works, **which are very excellent! We promise that we will cite them (CLIP-VG [4] and VG-Annotator [5]), and include corresponding discussions in our revised paper**.  Our method differs significantly from [4,5] in 2 key aspects: \n* **Different data Generation Methods**: [4,5] rely on templates, scene graphs, or small NLP models for generation, which may lack diversity in the generated data. In contrast, our CPCF leverages the more powerful MLLMs with **a novelly designed RAG framework**, where a similar labeled example is applied to guide the generation process for each unlabeled data. The stronger generative capabilities of MLLMs enhance data diversity, while RAG improves stability and mitigates errors. \n* **Different Training Strategies on Generated Data**: [4, 5] use conventional CE loss, L1 loss, or mIoU loss, to train the generated data. In contrast, our method introduces a carefully designed Direct Preference Optimization (DPO) framework, and the preference pairs in DPO are task-tailored designed—specifically, commonly generated vs. CoT-generated answers. **This is the first application of DPO to referring MLLMs (and with task-specific designs, not only simple usage)**, and it significantly outperforms CE loss (Table 6). \n\nWith these innovations, we believe our self-training method is novel. Thank you!\n### **Q5 Computational overhead**\nPlease see our answer to Reviewer Xoy3 Q1. Thanks!\n### **Q6 & Q7 Discussion with other CD methods (like CRG)**\nPlease see our answer to Reviewer hUfo Q1. Thanks!\n### **Q8 Effect of distillation on accuracy** \nYes, distillation slightly affects accuracy, as shown in Table 4, where box accuracy and mask accuracy decrease by 0.71 and 0.60, respectively, after applying distillation. However, given that the distillation can largely reduce average inference time by 74%, we believe this minor loss in accuracy is a worthwhile trade-off and completely acceptable. More importantly, the distilled model still significantly outperforms the previous SOTA by +6.22 in box accuracy and +5.36 in mask accuracy, demonstrating that it remains highly effective despite the compression.\n### **Q9 Innovations** \nOur framework contains multiple components, including contrastive decoding, self-training, and distillation. However, they are NOT simple combinations of existing modules, but each with the following key novelties: \n* (1) **Contrastive Decoding (CD)**: Prior CD methods typically rely on manually constructed contrastive objects (e.g., VCD uses perturbed images generated from random noise), which may introduce uncertainty and are difficult to be optimal (e.g., it is difficult to identify the most appropriate random noise for VCD). In contrast, **we are the first to propose an automated and learnable approach for identifying contrastive objects within the CD framework**, where a prompt extraction network is trained to find the optimal contrastive target **automatically**. This makes our method fundamentally different from previous CD methods and with better performance (Table 3). \n* (2) **Self-Training**: Kindly see our answer to Q4. \n* (3) **Distillation**: **We are the first to apply distillation to CD**, effectively addressing the issue of high computational cost in CD. More importantly, our framework does NOT simply adopt existing distillation techniques; instead, **we propose a novel distillation loss (Eq. 5, $\\mathcal{L}_{inp}$) specifically tailored to the characteristics of CD and referring MLLMs**, which is highly effective (Table 8).\n\nIn summary, **we have introduced task-specific, novel designs for each key component of our framework. Therefore, we believe our method is novel and can provide new insights**. We will clarify these more clearly in the revised paper and cite your recommended papers. Thank you!", "claims_and_evidence": "see above.", "methods_and_evaluation_criteria": "see above.", "theoretical_claims": "see above.", "experimental_designs_or_analyses": "see above.", "relation_to_broader_scientific_literature": "see above.", "essential_references_not_discussed": "see above.", "other_strengths_and_weaknesses": "see above.", "other_comments_or_suggestions": "see above.", "questions_for_authors": "see above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OhwGswNaS8", "reviewer": "Reviewer_Xoy3", "summary": "This paper addresses the performance limitations of referring multimodal large language models (MLLMs), which often misinterpret ambiguous or misleading visual regions during referring comprehension tasks. To overcome this limitation, the authors propose the Cross-Prompt Contrastive Framework (CPCF), which improves both region understanding and response generation through systematic contrastive prompting.\n\nThe framework operates through three core innovations: (i) it automatically extracts discriminative visual prompts using a Q-Former module to highlight critical image regions; (ii) it employs a self-training paradigm that combines retrieval-augmented question generation with direct preference optimization (DPO) to enhance response quality; and (iii) it incorporates a knowledge distillation mechanism that facilitates efficient contrastive decoding while significantly reducing computational overhead.\n\nFurthermore, this work presents an integration of retrieval-augmented generation for the creation of synthetic training data, along with contrastive learning principles tailored for visual-language alignment. Evaluations across multiple benchmarks demonstrate that CPCF achieves state-of-the-art performance, with quantitative results indicating significant accuracy improvements over existing methods. The authors further validate that the contrastive approach of their framework effectively reduces hallucination errors while maintaining computational efficiency through their proposed distillation strategy.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your time in reviewing our paper and the valuable comments! We are sincerely encouraged by your recognition that our method is novel and well-developed, experiments are extensive, and writing is good. For your questions and concerns, we are happy to provide the following responses:\n\n### **Q1 Inference speed comparison**\n\nThank you for this comment! The comparison of inference time between our method and the baseline Ferret is presented in Section 4.4, Lines 429–432. **Compared to Ferret, our method incurs only a 4.3% increase in average inference time while achieving significantly better performance** (please refer to Table 1). **We further compare our CPCF with Shikra and GLaMM: CPCF requires only 8.9% and 2.5% additional inference time, respectively, while achieving accuracy improvements of 13.77 and 8.44.** This demonstrates that the additional inference computation introduced by our method is minimal while the performance advantage is significant. We will include a comprehensive inference time comparison with all methods listed in Table 1 and incorporate the results into the revised paper.\n\n### **Q2 Ablation on contrastive prompt quantity**\n\nThank you for your comment. We would like to clarify that **the number of contrastive prompts is NOT a predefined or fixed hyperparameter**; instead, it is automatically determined by the network. Specifically, as detailed in Lines 184–189 of the main paper, we utilize the Mean Shift algorithm to cluster the positions of all valid prompts to generate the contrastive prompts. Since Mean Shift is inherently a clustering method without a fixed cluster number, the number of resulting contrastive prompts varies accordingly, with an average of 3.2 in our experiments. We will provide a more detailed statistical analysis in the revised paper. A potentially related hyperparameter is **the number of learnable queries input to the Prompt Extraction Network**. As shown in the table below, when the number of learnable queries is too small, the network may miss critical information, leading to performance degradation; whereas when the number is too large, the model performance saturates, resulting in limited improvement if further increasing the query number (only +0.17 accuracy on ROC-Box when the query number increases from 64 to 128). We will include a more detailed ablation study on the number of learnable queries in the revised paper. Thanks!\n\n| Query Number | ROC-Box | ROC-Mask |\n| ------------ | :-----: | :------: |\n| 16           |  76.68  |  77.09   |\n| 32           |  77.75  |  78.43   |\n| 64           |  78.37  |  79.55   |\n| 128          |  78.50  |  79.98   |\n\n\n\n### **Q3 Ferret-7B finetuned using CRG** \n\nGood advice! We follow your suggestion by fine-tuning Ferret-7B on $\\tilde{\\mathcal{D}}$ using the contrastive decoding method CRG. As shown in the table below, **this method achieves an accuracy of 75.95% on the ROC-Box scenario, outperforming baseline Ferret (71.71%) and naive fine-tuning (74.22%), but still falling significantly short of our CPCF method (78.37%)**. This demonstrates the significant advantage of our proposed automatic contrastive prompt extraction approach over the manually designed strategy in CRG. We will include this result into Table 7 of the revised paper.  \n\n| Method                         |  ROC-Box  | ROC-Mask  |\n| :----------------------------- | :-------: | :-------: |\n| Ferret                         |   71.71   |   72.39   |\n| Ferret + naive fine-tuning     |   74.22   |   74.37   |\n| Ferret + CRG (w/o fine-tuning) |   75.40   |   76.20   |\n| Ferret + CRG (w/ fine-tuning)  |   75.95   |   77.06   |\n| **CPCF (Ours)**                | **78.37** | **79.55** |\n\n\n\nThank you again for your valuable comments and thoughtful questions! We hope that our responses can address your concerns, and we would be happy to engage in any further discussion if you have any questions. Thank you so much!", "claims_and_evidence": "Yes, the claims are supported by evidence in the paper.", "methods_and_evaluation_criteria": "Yes, appropriate benchmarks and metrics are used.", "theoretical_claims": "The paper does not seem to make complex theoretical claims requiring proof verification. However, the core idea of contrastive decoding  is supported by empirical results.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "N/A.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "### Strengths\n- **Novel Methodology**: CPCF is a novel framework that leverages contrastive prompts to enhance referring MLLMs. The proposed framework is well-developed and incorporates several improvements that enhance both accuracy and efficiency.\n- **Extensive Experimental Validation**: The authors conduct extensive experiments across multiple benchmarks and four diverse tasks, demonstrating state-of-the-art performance.\n- **Good Writing**: The paper is clearly written and well-structured.\n\n### Weaknesses\n- **Lack of inference speed comparison**: The paper asserts that the proposed distillation technique enhances efficiency; however, it lacks a direct comparison with other methods regarding inference time.\n- **Missing ablation on contrastive prompt quantity**: Table 5 presents ablation studies on the semantic similarity map and the relative distance map; however, there is no analysis of the impact of the number of contrastive prompts used.\n- **Missing baselines**: Table 7 demonstrates that directly fine-tuning Ferret on the new dataset $\\hat{D}$ achieves a box accuracy of 74.22, while CPCF with contrastive decoding attains an accuracy of 78.37. However, there has been no evaluation of Ferret-7B fine-tuned using contrastive decoding methods (e.g., CRG).", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "Refer to the weaknesses above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6jhIFZMuqN", "reviewer": "Reviewer_nWKK", "summary": "This paper introduces CPCF, a cross-prompt contrastive framework for referring multimodal large language models. It improves region-specific response accuracy by automatically generating contrastive prompts from misleading regions, training these through a self-training strategy on additional unlabeled data, and then distilling the multi-execution model into a single-pass efficient model—all validated via extensive experiments across several referring benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the time and effort you dedicated to reviewing our paper, as well as for your highly valuable comments!  We would like to provide the following responses to address your concerns and questions:\n### **Q1 Concerns on feature contradictions**\nThank you for this comment! We would like to kindly clarify that the semantic similarity map and relative distance map are NOT directly added to the image tokens. Instead, they are first processed by a learnable CNN before being fused with the image tokens. Through training, the CNN can be optimized to transform the map information into the same feature space as the image tokens. Moreover, the features generated by the CNN have the same spatial sizes as the image tokens produced by the vision encoder. Therefore, the problem of feature contradictions will not arise. In fact, this is a common strategy also widely used in many other models—for example, in the prompt encoder of the Segment Anything Model.\n### **Q2&Q3 Quality of $\\tilde{\\mathcal{D}}$ and errors / computations of RAG and DPO**\nThank you for raising this important point! We would like to kindly provide the following clarifications: \n* (1) **The generated data (including questions and answers) in $\\tilde{\\mathcal{D}}$ are of high quality, thus they can be effectively used for training without introducing many errors**.  As indicated in Appendix C.1, we use GPT-4o to score the questions generated with RAG on a scale from 0 to 10 according to their relevance to the given image and input prompt. Among 2000 randomly sampled data from $\\tilde{\\mathcal{D}}$, these generated questions achieve a very high average score of 9.3, demonstrating their excellent quality. Furthermore, as reported in Appendix A.3, GPT-4o judges that in 91% of the sampled cases, the CoT-based answers are more accurate than the standard answers, demonstrating the effectiveness of using these pairs for DPO training. We also conduct a user study on 200 randomly selected generated samples, where volunteer ratings indicate that 94% of the generated data are correct and free of significant errors. These results demonstrate the robustness and effectiveness of our RAG- and CoT-based data generation method. \n* (2)  **Even with a small amount of errors, the impact on model performance is very minimal**. We find that models trained on our generated data perform almost identically to those trained on fully clean data. Specifically, we use GPT-4o to filter out noisy question–answer pairs from the generated dataset $\\tilde{\\mathcal{D}}$—approximately 8% of the total—and replace them with corrected versions. The model retrained on this cleaned dataset achieves ROC-Box and ROC-Mask accuracies of 78.90 and 79.96, respectively—only marginally higher than those of our original model (78.37 and 79.55), with differences of just 0.53 and 0.41. These minor performance gaps indicate that small errors in the generated cannot affect model performance significantly, further demonstrating the robustness of our method. \n* (3) The original dataset $\\mathcal{D}$ is highly diverse, containing rich images from a wide range of domains. Therefore, when constructing $\\tilde{\\mathcal{D}}$, it is unnecessary to deliberately search for images that specifically match the domains in $\\mathcal{D}$. Instead, randomly sampling images from various domains is sufficient and can be easily implemented in practical applications.\n* (4) Compared to directly using $\\mathcal{D}$ or training with conventional CE loss, **using RAG to generate dataset $\\tilde{\\mathcal{D}}$ or training with the DPO loss increases training time by only 4.7 hours and 9.2 hours**, respectively—an overhead we believe is entirely acceptable in practical applications, especially considering the significant performance improvements they bring (as shown in main paper Table 6).\n### **Q4 Comparison of inference time between the proposed method and baselines**\nGood suggestion! The comparison of inference time between our method and the baseline Ferret is presented in Section 4.4, Lines 429–432. **Compared to Ferret, our method incurs only a 4.3% increase in average inference time while achieving significantly better performance** (please refer to Table 1). Additionally, Table 4 of the main paper compares the inference time of our method with and without distillation, showing that the distilled model reduces inference time by more than threefold. We further compare our CPCF with Shikra and GLaMM, and CPCF requires only an additional 8.9% and 2.5% average inference time, respectively. These demonstrate that the additional inference computation introduced by our method is minimal. We will include a comprehensive inference time comparison with all methods listed in Table 1 and incorporate the results into the revised paper. Thanks!\n\nThank you again for your valuable comments. We hope that our responses can address your concerns, and we sincerely wish you all the best in your future life and work!", "claims_and_evidence": "Contrastive decoding using automatically extracted prompts reduces errors from distracting regions.\\\nSelf-training on extra data (with a RAG-based question generation method) further improves prompt extraction. \\\nThe distillation method can lower inference costs while preserving performance.", "methods_and_evaluation_criteria": "Please check the section **Other Strengths And Weaknesses**.", "theoretical_claims": "The paper does not include theoretical claims.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The paper builds upon and extends several lines of recent research. It relates to works on referring MLLMs (e.g., Ferret, Kosmos2), contrastive decoding for reducing hallucinations, and self-training and distillation techniques common in large-model training.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "1. In Section 3.2, the authors construct a semantic similarity map and a relative distance map, process them through a lightweight CNN, and incorporate them into the image tokens. Directly adding these maps to the image tokens may introduce feature contradictions, leading to noisy updates.\n2. The quality of $\\tilde{D}$ must be exceptionally high, requiring both $D$ and $\\tilde{D}$ from the same image domain to ensure consistency.\n3. Although the self-training process in Section 3.3 leverages RAG and DPO, each component remains susceptible to errors, and the computational cost is significantly high.\n4. The distillation step reduces inference costs, but the algorithm in general increases the overall complexity and computational burden of training. I recommend including experimental results comparing inference time between the proposed method and baselines to better illustrate efficiency gains.", "other_comments_or_suggestions": "Please check the section **Other Strengths And Weaknesses**.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ur35GDJUE1", "reviewer": "Reviewer_Vjf8", "summary": "The article aims to solve the defects of existing referring MLLMs: it is difficult to accurately locate the prompt, and designs several solutions: 1. an effective referring MLLM framework that contrasts input prompts with contrastive prompts from misleading regions, 2. an automatic prompt extraction mechanism to identify optimal contrastive prompts, a self-training method to improve network optimization\n\n## update after rebuttal\nI think the author answered my questions very well. After the author supplemented the experiment, the experimental part of the article became more complete. In addition, I agree with the author's explanation of DPO and SFT. I suggest that the author supplement this content in the revision. I decided to maintain 4 points.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your time in reviewing our paper and the valuable comments! We are sincerely encouraged by your recognition that our experimental results are good, the robustness analysis is detailed, the innovation is clever, and the overall quality of the article is high. For your questions and concerns, we are happy to provide the following responses:\n\n### **Q1 Comparison with general MLLMs**\n\nGreat idea! The comparison results with InternVL2.5-8B and Qwen2.5-VL-7B on Ferret-Bench are shown in the Table below. Our CPCF-7B outperforms both methods significantly, demonstrating the advantages and effectiveness of our novel method. We will compare on more benchmarks and incorporate these results into the revised paper. \n\n| Method          | Referring Cpationing | Referring Reasoning |\n| --------------- | :------------------: | :-----------------: |\n| InternVL2.5     |         80.4         |        79.2         |\n| Qwen2.5-VL      |         79.9         |        80.5         |\n| **CPCF (Ours)** |       **83.5**       |      **82.9**       |\n\n\n### **Q2 Usage of DPO and STAR** \n\nOur method employs DPO rather than the standard SFT-based iterative training like STaR. This choice is motivated by the fact that compared to SFT, preference-based fine-tuning with DPO allows for more fine-grained control over model optimization and has been shown in prior work to better mitigate hallucination issues. The results in Table 6 of the main paper demonstrate the effectiveness of DPO: compared to conventional CE loss for SFT, DPO achieves improvements of 2.90, 3.47, 2.75, and 2.13 on box, mask, scribble, and point prompt types, respectively. Nevertheless, we agree that the STaR is a promising approach, and we plan to explore its potential in the context of referring MLLMs in future work. Thank you so much!\n\n### **Q3 Improvement of 13B compared to 7B** \n\nThe limited performance improvement observed when scaling from the 7B to the 13B model may result from the relatively small size of existing referring multimodal datasets, which are insufficient to exploit the full potential of larger models. In fact, prior SOTA methods have also similarly exhibited marginal performance differences between their 7B and 13B variants; for example, Ferret-v2-13B even underperforms its smaller version, Ferret-v2-7B, on the Ferret-Bench benchmark. In contrast, despite a similarly modest performance gap, **our CPCF-13B consistently outperforms CPCF-7B across all benchmarks**, demonstrating our method’s effective scalability to larger models. For example, on the ROC task, CPCF-13B improves over CPCF-7B by 0.72, 1.17, 0.88, and 1.05, respectively across the 4 input prompt types. We believe that the development of larger and higher-quality referring multimodal datasets is crucial for unlocking the full potential of large referring MLLMs, which we identify as a key direction for our future work.\n\n\n\nThank you again for your valuable comments. We hope that our responses can address your concerns, and we sincerely wish you all the best in your future life and work!", "claims_and_evidence": "The article has clear experimental results and robustness analysis experiments to prove its credibility", "methods_and_evaluation_criteria": "The article proposes three different strategies, including iterative DPO, which can enhance the performance of the model, which is verified in the robustness analysis experiment of the article.", "theoretical_claims": "The article is an application-oriented article and does not provide any theoretical proof.", "experimental_designs_or_analyses": "The experiments in this article are quite sufficient, but the comparison objects lack general MLLMs, such as Qwen-VL2.5. However, it has a detailed robustness analysis experimental module.", "relation_to_broader_scientific_literature": "No", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths：\n1. The experimental results of the article are very good, far exceeding the previous referring MLLMs\n2. The article provides a very detailed robustness analysis experiment, which makes me very confident in the effectiveness of the designed module.\n3. The innovation of this article is quite clever. It integrates the idea of ​​contrastive decoding into referring MLLMs and achieves good results.\n\nWeaknesses：\nI think the overall quality of the article is high, and there are no logical errors or experimental setting errors, but I have the following doubts about the experimental part, I think it would be better to explain these clearly\n\n1. Why not compare some general MLLMs? For example, Qwen, InternVL, which both support referring tasks?\n\n2. Why use iterative DPO? Have you tried iterative training similar to STAR?\n\n3. Why is there no improvement in the trained model of 13B compared to 7B? Does it mean that this pipeline is difficult to scale up?", "other_comments_or_suggestions": "As shown in Other Strengths And Weaknesses", "questions_for_authors": "As shown in Other Strengths And Weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zzLbmXzKdp", "reviewer": "Reviewer_Za66", "summary": "The paper introduces CPCF (Cross-Prompt Contrastive Framework), a novel approach to improving Referring Multimodal Large Language Models. These models extend standard Multimodal Large Language Models by allowing them to focus on specific image regions using visual prompts. CPCF leverages a contrastive decoding strategy, where responses generated from an intended visual prompt are contrasted against those generated from misleading prompts.\n\nThrough extensive experiments across multiple benchmarks, CPCF achieves state-of-the-art performance, demonstrating significant improvements in multiple tasks. The model effectively mitigates errors caused by misleading regions, significantly outperforming previous approaches like Ferret.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your time in reviewing our paper and the valuable comments! We are sincerely encouraged by your recognition that our contributions are meaningful, the proposed method is reasonable, and the ablation study results are sound. We would like to provide the following responses to address your concerns and questions:\n\n### **Q1 Clarification about \"eliminating the uncertainty and instability inherent in manual methods\"**\n\nAs indicated in the introduction, we observe that the model tends to generate incorrect answers tailored to misleading regions that are adjacent or similar to the target object. Motivated by this observation, we initially attempted to manually select a random point within these misleading regions as the contrastive prompt for contrastive decoding. However, we empirically found that model performance is highly sensitive to the choice of this point. For instance, selecting two different points within the same misleading region—only 10 pixels apart—may result in significantly different contrastive decoding outcomes. This sensitivity is why we describe manual methods as exhibiting “uncertainty and instability.” In contrast, our prompt extraction network, trained through deep learning optimization, can learn to automatically identify the most appropriate contrastive prompt, thereby alleviating the instability introduced by random manual sampling, leading to better performance. We acknowledge that our original use of the term “eliminate” was too absolute, and we will revise it to “alleviate” in the updated version of the paper. Thank you!\n\n### **Q2 Computational costs of training**\n\nThank you for this valuable comment! Building on the pretrained Ferret model, our method introduces two additional training stages: (1) self-training and (2) distillation,  which require 30 hours and 25 hours, respectively, to train for 50K steps on 8 A100 GPUs for the 7B model. In practice, if training resources are limited, it is entirely feasible to largely reduce the number of training steps while still maintaining strong performance. We observe that the model can achieve over 95% of its final performance after just 25K steps, reducing the required training time to 15 hours and 12 hours for the two stages, respectively. Even under this shorter training schedule, the model still achieves SOTA performance, making the overall training cost and effectiveness entirely acceptable for practical use. Additionally, we would like to kindly emphasize that, once training is completed, **the additional computational cost during inference for our method is very minimal—only +4.3% compared to the baseline Ferret**. This efficiency is achieved because our model supports end-to-end inference without requiring multiple MLLM executions as other contrastive decoding (CD) methods,  which substantially reduces CD's computational complexity, highlighting the advantages of our method. \n\n### **Q3 Hyperparameter**\n\nThank you for the comment! Our method is not sensitive to the choice of the hyperparameter $\\alpha$ in Eq. 1. As shown in the table below, when $\\alpha$ is set to 0.1, 0.5, or 1, the model achieves ROC-box accuracy of 77.91, 78.37, and 78.31, respectively—demonstrating stable performance across a range of values and consistently outperforming the previous SOTA result (72.83). In fact, most of the hyperparameters in our method—such as $\\alpha$ and other training settings—can be directly adopted from existing contrastive decoding and referring MLLM approaches. We found that these settings already work very well without requiring specific modifications. Nevertheless, we are happy to conduct more ablation studies across a wider range of implementation settings and will include these results in the revised paper.\n| $\\alpha$ | ROC-Box | ROC-Mask |\n| -------- | :-----: | :------: |\n| 0.1      |  77.91  |  79.24   |\n| 0.5      |  78.37  |  79.55   |\n| 1        |  78.31  |  79.53   |\n\n### **Q4 Figure issues**\n\nThe out-of-box issue will be fixed. Thank you so much!\n\nThank you again for your valuable comments and thoughtful questions. We hope that our responses can address your concerns, and we sincerely wish you all the best in your future life and work!", "claims_and_evidence": "The claims made in the paper are generally supported by empirical evidence, including extensive experiments and comparisons with prior models.\n\nHowever, I have several severe questions. The paper mentions that the frameworks \"eliminating the uncertainty and instability inherent in manual methods. \" How is it possible that the method \"eliminates\" uncertainty and instability? And what do these two properties, uncertainty and instability, mean in practice?", "methods_and_evaluation_criteria": "The contrastive decoding framework directly addresses the issue of models being misled by adjacent or similar visual regions, and the prompt extraction network provides an automated way to generate meaningful contrastive prompts. I suppose that this method is reasonable.", "theoretical_claims": "There are no theoretical claims provided or any proof. The equations in this paper seem to be correct.", "experimental_designs_or_analyses": "The contrastive decoding effectiveness is demonstrated through both quantitative results (Tables 1-3) and qualitative visualizations (Figure 5). The authors also considered multiple baselines and datasets for comparison and evaluation. The ablation study results are also sound.", "relation_to_broader_scientific_literature": "The novelty lies in task-specific contrastive decoding, automated prompt selection, and efficiency-aware distillation. These are meaningful contributions to the literature.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\nThe self-training mechanism leverages unlabeled images to generate synthetic question-answer pairs, enhancing model robustness without requiring extensive manual annotation.\n\nWeaknesses:\n\nCPCF introduces multiple additional components (contrastive decoding, self-training, and distillation), making training more computationally intensive than standard referring MLLMs. The authors did not provide a sufficient analysis about the computational costs, which could be a problem.\n\nThe weighting of contrastive prompts and distillation settings (e.g., hyperparameters like α in contrastive decoding) may significantly impact performance, requiring careful tuning.", "other_comments_or_suggestions": "The figures and illustrations are slightly messy, especially the framework figure. Eq. (5) is out of box.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w1RMLyENXd", "reviewer": "Reviewer_WGMH", "summary": "This paper focus on addressing the incorrect responses of referring MLLMs tailored to misleading areas adjacent to or similar to the target region. Specifically, it introduces contrastive visual prompts generated by prompt extraction network, which is trained on extra dataset. Besides, to alleviate the computational cost, the distillation is adopted to run single-execution decoding. The writing is good in general. The experimental results shows its effectiveness of proposed method. \n\nHowever, training on extra dataset compared with other competitors weakens its effectiveness. Besides, the training process is much complex compared with competitors.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Q1 Training process**\n\nYes, we agree that our training process involves additional stages—specifically, self-training and distillation—which require 30 hours and 25 hours, respectively, to train for 50K steps on 8 A100 GPUs for the 7B model. However, we would like to kindly clarify the following points: \n\n* (1) As shown in Table 4 of the main paper, **both stages are crucial**: self-training significantly improves model performance, while distillation greatly enhances computational efficiency. These results validate the rationality of our multi-stage training pipeline. \n* (2) Moreover, **if training resources are limited, it is entirely feasible to largely reduce the number of training steps while still maintaining strong performance**. We observe that the model can achieve over 95% of its final performance after just 25K steps, reducing the required training time to 15 hours and 12 hours for the two stages, respectively. Even under this shorter training schedule, the model still achieves SOTA performance, making the overall training cost and effectiveness entirely acceptable for practical use. \n* (3) Additionally, once training is completed, **the additional computational cost during inference for our method is very minimal—only +4.3% compared to the baseline Ferret**. This efficiency is achieved because our model supports end-to-end inference without requiring multiple MLLM executions as other contrastive decoding (CD) methods,  which substantially reduces CD's computational complexity, highlighting the advantages of our method. \n\n### **Q2 Extra dataset**\n\nYes, our method utilizes an additional dataset for training. This is intended to mitigate potential overfitting issues that may arise when performing contrastive decoding training directly on the original dataset (please refer to Lines 192–218 of the main paper for details). However, we would like to respectfully clarify the following points: \n\n*  (1) The additional dataset we use is NOT a fully annotated, supervised dataset, but rather an unannotated corpus without any question-answer labels,  which is **easy to collect in real-world scenarios**. Moreover, training a model with such unlabeled data is non-trivial, and we have carefully designed a self-training approach to address this challenge.  As such, **a new method for training referring MLLMs using unlabeled data is itself a key contribution of this work.** \n\n*  (2) **Even without using the extra dataset, our method can still achieve state-of-the-art performance**. To be specific, using the original dataset $\\mathcal{D}$ without the extra dataset $\\tilde{\\mathcal{D}}$ in our method achieves an accuracy of 74.55 on ROC-Box, which still significantly outperforms previous approaches such as Ferret (71.71), Osprey (72.15), and Shikra (64.60). This indicates that the extra dataset only serves as a strategy for further enhancement, but is not the sole reason for our method’s strong performance. The proposed network architecture and training strategy also play a crucial role in achieving high effectiveness. In the revised version of the paper, we will incorporate the results without the extra dataset into Table 1 and Table 2. Thank you very much!\n\n### **Q3  \"not used’ and \"not included’ in Line 315-316**\n\nHere, the definitions ‘not used’ and ‘included’ are measured from both image-level and region-level, thus the region-level information will NOT be leaked to the test set.\n\n### **Q4 The ‘[region]’ on the top part of figure 3 should be green.**\n\nThank you for indicating this typo! We will fix it in the revised paper. \n\n### **Q5 The contrast visual prompts are point or mask? Is it fixed across all datasets?**\n\nThe contrastive prompt is a point, and this choice is fixed across all datasets. We also explored generating different types of prompts based on the input type—for example, generating box contrastive prompts for box inputs and mask contrastive prompts for mask inputs. However, this approach increases the optimization difficulty for the prompt extraction network, as it would require generating multiple prompt types from a single network. While using a separate prompt extraction network for each type is a potential solution, it would significantly increase the model’s parameter count. Therefore, we adopt a unified and simplest form of contrastive prompt across all datasets: the point.\n\n\n**In summary, our method can achieve SOTA performance with an acceptable training time and requires very minimal additional computation during inference (only +4.3% than Ferret). While an extra dataset is used, it does not require any manual question–answer annotations, making it easy to obtain in practice.** Thank you again for your valuable commnets and thoughtful questions! We hope that our responses can address your concerns, and we would be happy to engage in any further discussion if you have any questions. Thank you so much!", "claims_and_evidence": "Yes, the claims are supported by quantitative and qualitative results.", "methods_and_evaluation_criteria": "Yes, the method seems make sense for the problem.", "theoretical_claims": "There are no proofs for theoretical claims.", "experimental_designs_or_analyses": "Yes, the comparison and ablation studies show its effectiveness of proposed method.", "relation_to_broader_scientific_literature": "It uses Ferret as baseline to conduct referring.", "essential_references_not_discussed": "The references seem adequate.", "other_strengths_and_weaknesses": "(1)The training process is much complex than competitors, which consumes computation resources.\n(2)Besides, it utilizes extra dataset to train prompt extraction network and also the MLLM. In Line 315-316, the definition ‘not used’ and ‘included’ is measured from image-level? The region-level information may be leaked to test set.", "other_comments_or_suggestions": "The ‘[region]’ on the top part of figure 3 should be green.", "questions_for_authors": "The contrast visual prompts are point or mask? Is it fixed across all datasets?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received mixed reviews, including two accepts, three weak accepts, and two weak rejects. Reviewers generally found the problem of improving referring MLLMs important and appreciated the paper’s well-structured empirical results, especially the performance gains achieved with modest inference overhead. The proposed CPCF framework includes automatic contrastive prompt extraction, self-training, and distillation techniques.

However, several reviewers raised critical concerns: (1) the method is a composition of existing components with limited novelty; (2) comparisons with key baselines such as CRG are incomplete or implementation-dependent; (3) claims of substantial improvements were challenged and, though rebutted with quantitative clarification, some reviewers remained unconvinced; (4) the overall system is complex, and the practical gains over previous contrastive decoding methods are incremental.

The rebuttal was detailed and addressed some concerns, but given the concerns about novelty and complexity, the AC recommends weak accept. The authors are encouraged to further highlight the novelty and consider simplifications for broader applicability.
Decision: Accept (poster)