ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models
Authors: Mingrui Wu
Xinyue Cai
Jiayi Ji
Jiale Li
Oucheng Huang
Gen Luo
Hao Fei
GUANNAN JIANG
Xiaoshuai Sun
Rongrong Ji
Venue: NeurIPS 2024 poster (2024)
Keywords: Training-Free；Visual Prompt；Multimodal Large Language Models
Primary area: generative_models

Abstract: In this work, we propose a training-free method to inject visual prompts into Multimodal Large Language Models (MLLMs) through learnable latent variable optimization. We observe that attention, as the core module of MLLMs, connects text prompt tokens and visual tokens, ultimately determining the final results. Our approach involves adjusting visual tokens from the MLP output during inference, controlling the attention response to ensure text prompt tokens attend to visual tokens in referring regions. We optimize a learnable latent variable based on an energy function, enhancing the strength of referring regions in the attention map. This enables detailed region description and reasoning without the need for substantial training costs or model retraining. Our method offers a promising direction for integrating referring abilities into MLLMs, and supports referring with box, mask, scribble and point. The results demonstrate that our method exhibits out-of-domain generalization and interpretability.

ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models
Li', Oucheng Huang', Gen Luo', Hao Fei", Guannan Jiang', Xiaoshuai Sun', Rongrong Ji Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China
Mingrui Wu', Xinyue Cai', Jiayi
2 National University of Singapore 3 CATL mingrui0001@@mail.ccm
Abstract
In this work, propose training-free method to inject visual prompts into Multimodal Large Language Models (MLLMs) through learnable latent variable optimization. We observe that attention, the module of MLLMs, connects text prompt tokens and visual tokens, ultimately determining the final results. Our approach involves adjusting visual tokens from the MI.P output during inference, controlling the attention response to ensure text prompt tokens attend to visual tokens in referring regions. We optimize learnable latent variable based on an energy function, enhancing the strength of referring regions in the attention map. This enables detailed region description and reasoning without the need for substantial training costs or model retraining. Our method offers a promising direction for integrating referring abilities into MLLMs, and supports referring with box. mask, scribble and point. The results demonstrate that our method exhibits out-of-domain generalization and interpretability. Code: https: htops://githul thub http///wuummac//Conooo
Introduction
In recent times, there has been a surge in the development and adoption of large language mod- els (LLMs), such as GPT-4 [ and Llama (53). showcasing remarkable capabilities in addressing wide range of researchers in exploring the integration of LLMs with visual inputs. Consequently, new class of models known as Multimodal Large Language Models (MLLMs) has emerged [36 33
questions. The success of these models has sparked interest among
However, despite their widespread adoption, traditional MLLMs often face limitations due to their reliance on coarse image-level alignments. This restricts users guiding MLLMs solely through text prompts for detailed region description and reasoning. However, text often fails to capture the intricate visual nuances present in an image.
Addressing this challenge, recent efforts [68 5 abilities within MLLMs, which enables users provide input by pointing to specific coordinates of the objects or regions, as shown in Figure [ (left). However, these endeavors typically entail substantial training costs to equip MLLMs with referring capabilities. Additionally, the model must undergo retraining to adapt to new data domains or new base MLLMs.
75
12 37 have pioneered the integration of referring
In this work, propose training- free method to inject the visual prompts into the Multimodal Large Language Models via learnable latent variable optimization. The method originates from our observation of the attention maps from the MI.LM decoder, which model the relationship between
"Correspondigg Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Training Method
Ours Training-free Method
Training stage:
Itaining with original domain region-text training pairs
Inference on free domain prompt
MLLM
MLLM
Whar color the har the persion weering!
color the person vicarin??
Inference stage:
Inference on free domain prompt
Inference on target domain prompt
MLLM
SAL
MLLM
SALE
Deseribe the TExt The region
SALE -PEE
Describe the text
Figure 1: Comparison between the training method and our training-free method. The training method typically requires a large amount of in-domain data for training and cannot generalize to out-of-domaal prompts. In contrast, our method can easily adapt to prompts from new domain in training-free manner.
the pixels and text prompt tokens and encompass rich semantic relations that significantly influence the generated text. However, MILMs typically involve fine-tuning MLP layer to bridge the gap between visual and linguistic representations, which means that the output of the MLP layer can indirectly impact the relationship between text prompt tokens and pixels in the attention layers of the MLLM decoder, thereby altering the model's output.
Thus, key idea is that MLP output during the inference process, controlling which text prompt tokens attend to which visual tokens in the attention layers. Specifically, latent variable. Subsequently, we optimize the learnable latent variable based on an energy function designed to enhance the strength of the referring regions in the attention map between the text tokens and the visual tokens.
alter the outputs of MLLMs by adjusting the visual tokens from the
augment visual tokens with an additional learnable
Our method enables referring MLLMs with various visual prompts, including box, mask, scribble and point, and does not require model training, fine-tuning, or extra data. We also demonstrate that method exhibits out-of-domain generalization and interpretability.
2
Related Work
53), there is burgeoning trend among researchers to develop a diverse range of Multimodal Large Language Models (MLLMs)
MLLMs
Motivated by the accomplishments of Large Language Models (LLMs)
58 35| 20 21 19]. These MLLMs typically comprise visual encoder, language decoder, and an image- alignment module. The visual encoder and the language decoder are often sourced from pre-trained models, such as CLIP 44,, DINOv2 [1I. Llama (53], and Vicuna [16]. Meanwhile, the image-text alignment module trained on image-text pairs and fine-tuned through visual instruction tuning to enhance its visual conversation capabilities. These Multimodal Large Language Models (MLLMs) often confront limitations stemming from their reliance on coarse image-level alignments.
33 36 17
67
32
38 39
22
15
Referring MLLMs In recent research, there has been a noticeable trend towards integrating foundation models with tasks involving referring dialogue. These models (69 74
75 12
43
64
introduce spatial visual prompts as extra input and are trained using region-text pairs. By leveraging this approach, they effectively bridge the gap between textual prompts and visual context, enabling comprehensive understanding of
26
70 79
51
46
80
63
8 24
45
52 72

---

image content at the regional level. However, these methods inevitably require substantial training burden.
Training-free Control in Text-to-Image There are numerous works on controllable text-to-image generation, among which training-free methods [27 14 62 30] are most relevant to research. Among them.
[27] explore the role of attention in text-visual interactions in Stable Diffusion 447 model, while Layout-Guidance [14] indirectly bias attention in Stable Diffusion model by optimizing an energy function. These contributions significantly inform our investigation into enhancing controllability and interpretability in MLLMs.
Visual Prompt The visual prompt can be categorized into two main techniques: hard prompt and soft prompt. The hard visual prompt works 48 57 66 65 that direct the model's attention to the region or enable visual grounding abilities in the Multimodal Models in training- free and convenient manner by directly manipulating images, such color guidance [60] 23). However, these methods inevitably compromise the structural information of the images, or a strong understanding of the corresponding patterns by the model is required. In contrast, the soft visual prompt works [28] 4 777
integrate learnable visual prompts into models to adapt them for different downstream tasks. However, these methods do not support region guidance and require fine-tuning the model with downstream data. In contrast, we optimize a learnable latent variable to support referring, MILM in the test time, without any downstream training data required, and TPT 49 is most related to our work.
3 Background
Multimodal Large Language Models (MLLMs): The MLLMs typically consist of visual encoder, an LLM decoder, and an image-text alignment module. Given an image I, the frozen vision encoder and a subsequent learnable MLP are used to encode into a set of visual tokens e,. These visual tokens e, then concatenated with text tokens e, encoded from text prompt pt, forming the input for the frozen LLM. The LLM decodes the output tokens sequentially, which can be formulated as:
= fI, Pt, yo. V1, ... -.i-1).
Considering LLaVA-liked (36] MILMs, the LLM in MLLMs typically employs a transformer model (54) with the attention layer its core. In such model, the attention maps represent the relationships between the visual tokens and the text prompt tokens. The attention map in attention layer r, computed on the transformed visual-text concatenated embeddings [e,, ex)(s) is obtained as follows:
[e,,er](r) (len, (()).ee)(1)))
1 softmax
(2)
vdi
where de is a scaling factor. ( consists of 157 with i,j E {1,... n). representing the relation- ship between the i-th token and the j-th token, and their impact the output.
Training Referring MLLMs:
The objective of training referring MLLMs is inject the visual prompt into the MILMs to achieve referring ability via model parameter learning. The visual prompt r can take various forms, such as box, mask scribble, or point, to indicate specific locations or regions within the image.
Current referring MLLMs typically need to be fine-tuned on a training set with positional annotations before they can be effectively used. The fine-tuning process involves maximizing the log likelihood of generating the text conditioned on Pt. and r over the entire training dataset. This can be formulated as:
0+ arg max log P(y: P. No. "1.... ". 1),
where 9 represents the parameters of the model f. and U is the number of samples in the training set. This method significantly enhances the model's fine-gra ine-grained understanding and interactivity. However, it incurs high training costs. Additionally, such fine-tuning strategies result in domain- specific behaviors, which have been shown to compromise the out-of-distribution generalization and robustness of MILMS [49]. Therefore, when domain shifts occur, the model needs be retrained, leading to lack of flexibility.
3

---

Input Image
Attention L_0
Attention L_7
A tttention 15
Attention L_23
Attention L 31
w/
context token
Prompt: "What coloris the hat the person wearing?" Output: "The hat the person is wearing is green."
Figure The attention maps in various layers of the MLLMs, with the numbers indicating the respective layer indices. The top line visualizes the attention between the prompt token "hat" and the visual tokens, while the bottom line visualizes the attention between the context token (mentioned in and the visual tokens.
Sec. 4.2
4 Method
We aim to propose a training-free method to overcome the inconveniences of traditional training. Training-free free referring MLLM maintains the model parameters & frozen, eliminating the need for any training or fine-tuning with samples from the training set. During inference, the only information available is the single test sample without label information, as shown in Figure (right).
In this section, we explore and design a solution to address the challenges MLLMs. The key task is to flexibly embed visual prompts during the inference phase while maintain- ing the model's reasoning capabilities. To begin with, we delve into the mechanism MLLMs (see Sec. 4. our key observation is the attention mechanism in LLM capturing the relationship between the model's 's output and the input pixels. Further, the visual tokens inputted into the LLM influence the values of the attention maps indirectly control the model output. Building this analysis, we propose the Latent Variable learning (a test-time prompt tuning strategy [49], see See. (.2) to edit the visual tokens, shown in Figure This method effectively integrates visual prompts into pre-trained MILMM,, enabling fine-grained visual reasoning.
Training-free Referring
4.1 Analysis of the Attention in LVLMs
We begin by analyzing which factors in the model truly capture the relationship between input and output? In other words, we seek to understand how to interpret the association between the model's output and the input pixels.
As demonstrated by Equation Multimodal Large Language Models (MLLMs) fundamentally model the maximum likelihood output based on visual input and text prompts. By conditioning on the text prompt, the model can determine which parts of the image have the greatest impact on the output. Building on the discussions in the Sec. and illustrations the Figure (top line), we can observe that the attention map models the influence of visual tokens on the output conditioned by the text prompt. Therefore, the attention map in MLLMs only provides interpretability regarding the relationship between model output and input pixels but also facilitates guiding the model's output.
natural idea is that by IBD [82, we achieve this by adding an adjustment coefficient to the attention related to the visual tokens corresponding the referring region, which can be formulated as,
directly alter the model's output by editing the attention maps. Inspired
A (T) softmax (en, (T) (ler, et) (T)T
+ M),
Vdl
(4)
M =
if i
else 0,

---

Input Image
Vanilla Inference
Prompt: "What color the hat the person is wearing?"
Output: "The hat the person is wearing is green."
Step by step
Ours optimize
optimize Latent Variable
Edit Att. n
Edit
Edit Att. T 20
Edit Att. n 10
Context Tokens
b)
Output: "The person wearing the is wearing green hat."
Outpat: "red."
Outpur: s.""
Output: "red and red."
Output:
Output: "The person wearing the hat wearing red hat."
Figure 3: Manipulating attention with various methods: (a), (b), and (c) demonstrate the manipulation of the attention map by adding an adjustment coefficient r) on the attention map in the first step during the model inference. (d) illustrates the step-by step-by-step editing approach. (e) showcases that optimizing learnable context tokens (mentioned in Sec. (42)) instead of visual tokens, while (f) presents the results of our method optimizing the learnable latent variable.
where M is mask with the same shape the attention map, denotes the referring region. However, we have to carefully select suitable coefficient for each example. When the 1) is too small, it leads to ineffective control (as shown in Figure 3 a). and when it is too large, it can impact the language capabilities of the LLM (as shown in Figure c). Additionally, we found that
sufficient to manipulate the attention map at the 0-th step during model inference (as shown in Figure b,b.), as it is most directly associated with the text prompt, and manipulating attentions step by step also affects the expression of the LLM (as shown in Figure d). Overall, directly manipulating attention maps is not viable approach because overlooks the relationships between attention layers and not all layers' visual tokens decide the output (13.
note that the most MLLMs, typically the MLP layer is trained for image-text alignment. This implies that MILMs indirectly affect the values of the attention map by learning the parameters of the MLP layer to alter the visual tokens. In other words, the visual tokens inputted into the LLM directly influence the values of the attention maps.
is also worth noting that the input text prompt also directly influences the model's output, particularly regarding non- non-visual-red visual-related [76] output content. However, we aim to explain the correlation between the output and the input image. Therefore, we do consider the direct impact of the text prompt on the output in our analysis.
4.2 Manipulating Attention via Latent Variable Learning
Based on the analysis above, our core idea is to indirectly influence the attention maps by editing visual tokens, thereby focusing on the referred regions. We achieve this by optimizing a learnable latent variable based on an energy function [14] (59,, which calculates the relationship between input referring and the attention maps. To this, we first need to determine which attention maps use. One approach is to use attention maps between each text prompt token and all visual tokens. However, because visual tokens typically have a significant impact on the result based on only few most relevant text prompts (referred to highlight text tokens), using all attention maps would be computationally redundant. Yet, for users, identifying the highlight text tokens can be challenging. Therefore, we simply average pool the attention maps generated for each text prompt token to represent the global context of the text prompt (referred to as the context token) and its association with visual tokens. We found that this simple method of using context tokens produces attention maps similar to those generated by highlight text tokens, as shown in Figure (bottom line). We leave the optimization based highlight text tokens for future work.
Specifically, our method supports four types of referring shapes, including box, mask, scribble, and point. We employ two types of energy functions to respectively support these referring shapes: hard

---

red
Output:
"The hat the person wearing is green."
Pool Att.
Self Cross)
LLM
N>
E
A tbention
tu
Optimization
T
P. + cv
e1
Ev
Pr
MLP
Pool Each
Tokenizer & Embedding
Visual Encoder
learnable token
Prompt: "W'hat color is the hat the person is wearing?"
frozen tolen
context token
Po E
Figure 4: The overview of our method. With the provided visual prompt, convert it into mask, and compute the mask-based energy function between the mask and the pooled attention map. During the inference process, conduct backpropagation to optimize learnable latent variable. This process is executed at the (-th step of model inference and iterated T times.
mask-based energy function for box and mask referring, and a soft mask-based energy function for scribble and point referring.
Hard Mask-based Energy Function We first zero initialize learnable latent variable p. with the same shape as Eg, and add it to the e,. Then we can get N attention maps from N attention layers which model the relation between the context token and the novel visual tokens. Given the referring box or mask, we first convert it into a binary mask. Then, we compute the mask-based energy function based on the mask and the attention map A(et), which is obtained by averaging pooling from N attention maps. The energy function can be formulated as:
-11 Eier A(o))
E' (A(())))
(5)
A,
where denotes referring region. Then the gradient of the loss D is computed via backpropagation to update the learnable latent variable:
P. P aV (A(ct),))
where a > 0 is a hyperparameter controlling the strength the guidance. By optimizing Pe through the Equation 6 we indirectly guide the attention maps to produce higher responses in the referring region r, thereby increasing the influence of the visual content of region r on the output.
Soft Mask-based Energy Function Since scribble and point lack the concept of the region, it is optional to use an extra SAM [31] model to obtain a mask for applying the Hard Mask-based Energy Function. However, this incurs additional inference cost, so also provide an optional soft mask-based energy function based on a distance matrix D, which is computed via applying the OpenCV [7] diistanceTransfors function the given scribble point. Then the soft mask-based energy function can be formulated as:
( Eier V2TT
(ct)
E (A(ot),))
A.4.
where is the standard deviation of the Gaussian function, which is set to 0.1. By optimizing p through the Equation 7 the closer the region of attention map is the given scribble or point, the higher the response.

---

Input Image &
LLaVA Atention Map
LLaVA Ours Atentton Map
Visual Prompt
Input Prompt & Output Text
Prompt: What is the object?
LLaVA: The object is a boat.
a)
LLaVA + Ours: The object is a life ring, which is type of personal flot.
Prompt: What is the object?
LLaVA: The object is a chair.
b)
LLaVA Ours: The object is a desk with chair,
plant, various decorative items
Prompt: What is unusual of the
LLaVA: The unusual aspect of the cat that it
jumping in the air while holding another cat in its mouth.
LLaVA Ours: unusual aspect the that it
has paw on its face, which is not a common feature.
Prompt: W hat is unusual of the object?
LLaVA: The unusual aspect of the object in the image
d
that it is a turtle with shell on its back, and It wearing hat. This not a typical sight, as turtles do not usually wear hats or have shells on their backs.
LLaVA Ours: The unusual aspect of the object is that is a snail shell, which is not typical object to be used as hat or decoration. T he snail shell is placed on the head of a frog, which is an unconventional combination of animals. This creates an interesting and unexpected scene, as it is not common to see a snail shell being used in such manner.
Figure 5: The examples of referring MILMM with four types of visual prompt, including box (a), mask (b), scribble (c) and point (d). The correct referring expressions are marked in green, incorrect referring expressions are marked in red, and hallucinated expressions are marked in orange. Compared to the baseline model, our method enhances interpretability and controllability with visual prompts, while also helping the model mitigate hallucination issues.
Finally, iteratively optimize the learnable latent variable T times at the 0-th step of model inference. In addition, to prevent overfitting. we employ Early Stop (ES) and Exponential Moving Average (EMA) strategies to enhance model stability. More details are shown in Appendix B.1
Experiments
5.1
Experiment Details
Unless explicitly stated otherwise, the MLLM we use is (LLVA--V1.-70 (35), T=5, 4=400 and 0.5. All experiments are conducted on two RTX 3090 GPUs with 24 GB memory each.
5.2
Applications
Referring with Different Visual Prompts. We first demonstrate referring QA with different visual prompts, including box, mask, scribble and point in the Figure Our method consistently demonstrates significant controllability with four types of visual prompts. And our method improves the interpretability compared to basic model (column vs column 2), demonstrates stronger correlation between the attention response areas and the generated descriptions.
Out-of-Domall Task. We present examples of the performance on out-of-domain tasks OCR and Screenshots. As shown in Figure 6 compared to Ferret, our method correctly identified the text in the referring region. Additionally, as shown in Figure our method correctly recognized the app in the mobile screenshot, unlike Ferret.
7

---

Prompt: what the text?
Models
LLaVA
Ferret
LLaVA Ours
SPECIAL OFFER SALE UPTO 80% OFF
SPECIAL OFFER
SPECIAL OFFER
Super Deal!
Super Deal!
Input Image
Deal
UPTO 80% OFF
UPTO 80% OFF
The text is "SPECIAL OFFER!
The text states \ "Suuprmme sale!" in black letters on pink background.
Output
The text "Sale".
SUPER SALE! 80% OFF!".
Prompt: what is the text?
Models
LLaVA
Ferret
LLaVA + Ours
SPECIAL OFFER SALE UPTO 80% OFF
SPECIAL OFFER SALE
SPECIAL OFFER SALE
Super
Super Deal!
Super Deal!
Input Image
Output
The isxt 'SPECIAL OFFER! The text appears to be sale
The text UPTO 80% OFF".
SUPER SALE! 80% OFF!".
advertisement, possibly for a car.
Figure 6: Examples of comparing with training method Ferret OCR.
Impact on Hallucinations. Our method guides the model to focus on specific regions, potentially helps the model mitigate hallucination issues, as shown in Figure (c.d output in orange color).
5.3 Comparisons
Comparison on Referring Object Classification Task. Following Ferret 68 74
we use the
Referring Object Classification (ROC) task to evaluate whether our method can accurately pinpoint and understand the semantic of the referring region. The task requires the model to correctly identify the target within the referring region. We follow the setting of Ferret to form 1,748 questions (in which 1,548 for test and 200 for validation) based on LVIS (25] validation dataset, with corresponding box, mask, scribble and point. We consider the edit attention with = 10 (as Equation and ( as the baseline model. And compare several training methods [43 75 12
68 We
also evaluate the lower and upper limits of LLaVA' recognition capability by assessing LLaVA without referring region, well background blur outside the referring region, which presented in gray. Additionally, we evaluate a method that highlights regions with color as a comparable training-free method. More details and the input examples are shown in Appendix B.2
The results are shown Table [ Our method shows a better performance than the training method GPT4-ROI with box referring (60.59 vs 58.59) and the Shikra-7B with point referring (58.85 vs 56.27). However, due to the limitations of LLL capabilities (as shown the results of the LLaVA+Bluu), we present a performance gap compared to the latest training method Ferret (68]. Our method also demonstrates superiority compared to training- free color prompt-base based method and baseline method.
Comparison on Referring Text Classification Task. We consider the Referring Text Classifica- tion (RTC) task as the out-of-domain task, to verify the model's 's out-of-domain transfer capability. Similar to the ROC task, we formulate the problem as a binary classification task and construct 1.372 questions based on the COCO-Text (56) dataset. Since point and scribble referring methods are not
8

---

Table I: The results on Referring Object Classification Task (test set). The prompt of the task is featured as "Iy the object (location) (class A) (class B)?". "_" denotes the method does not support this type of referring. Results in gray font are provided for reference only.
Table 2: The results Referring Text Classification Task. The prompt of task is featured as "Is the text (location) of the image (text A) or (text By ?please select only one.".
Models
Box
Mask
Scribble
Point
Models
Box
Mask
Training Methods:
Training Methods:
Kosmos-2 2
43
55.17
Kosmos-2 43
16.55
GPT4-RO ROI
75
58.59
GPT4-ROI
75
54.23
Shikra-7B -B
12
64.60
56.27
Shikra-7B
50.07
Ferret- 7B
68
71.71
72.39
71.58
68.54
Ferret-7B
55.47
56.34
Training- Free Methods:
Training- Fine Methods:
LLaVA 36
54.72
54.72
54.72
54.72
LLaVA [36]
53.57
55.47
LLaVA Blur
73.39
71.32
LLaVA Blur
83,60
74.49
LLaVA Color
55.10
56.72
LLaVA Color
56.34
54.23
LLaVA Edit Att
36.24
37.08
LLaVA Edit Att
26.09
29.16
LLaVA Ours
60.59
60.79
58.33
58.85
L.LVV Ours
61.22
60.28
suitable for text due to the non-connectivity of the text, we only evaluate the RTC task with box and mask referring.
The results are shown in Table 2 All the training methods we evaluated exhibited poor out-of-domain generalization performance. Specifically, Ferret achieves only 55.47% accuracy on the RTC task, despite its excellent in in-domain performance shown in Table [ In contrast, our training-free method still demonstrates the best out-of-domain generalization performance. We also present comparative examples of oft-of-domain tasks, as shown in Figure 6 and Figure 9
More Tasks and MILMss We also validate our method through the Referring Description Task on LLLVA-115-7B and InstructBL.IP-7B [17]. The results are shown in Table Our method consistently improves the model's 'ss referring description performance. And validate method the more MLLMs through the ROC and RTC Tasks, MLLMs including InstructBLIP-
and LLaVA- VA-HR- 7B [39]. more details are shown in Appendix B.3 The results are shown in Table 4 method consistently improves performance across different MLLMs. Due to InstructBLIP's relatively poor text recognition capabilities, However, thanks to the beneficial effect of image resolution on the RTC task, our method achieves a relative improvement of approximately 11.59% LLaVA-HR.
method results in only modest improvement in the RTC task.
5.4 Ablation Study
The ablation studies primarily focus on the box referred object classification. Furthermore, inspired by DIFNet [61), we calculate relevancy between the model's 's output and pixels within the referring region to assess the extent to which the model's 's output is influenced by visual content within the region. More details and additional experiments are shown in Appendix B.2 and B.3
Impact of T and a. As shown in Table 5 as T increases, the relevancy between the model's output and the referring regions also increases. However, the larger T results decrease in the model's accuracy on the ROC task, also with excessively large relevancy scores, showing that excessively large relevancy scores also indicate overfitting of the learnable latent variable. Therefore, the value of the relevancy score provides us with guidance to alleviate model overfitting, particularly when the relevancy score is around 0.18. typically resulting better performance. And the value of a affects the convergence speed of optimization, with larger also leading to overfitting of the model.
Impact of EMA and ES. As shown in Table when equipped with a smaller 3 value, effectively mitigates the overfitting issue associated with the learnable latent variable. For instance, with a 400 and 3 = 0.3, the model's performance improves from 53.5 to 62.5. However, a smaller value of also results in slower convergence of the learnable latent variable. Therefore, we combine the Early Stop strategy, allowing us use a slightly larger to accelerate the convergence of the learnable latent variable. After incorporating the early stop strategy, opt for slightly larger T to ensure

---

Table 3: The results box Referring Description Task on RefCOCOg [29]. The prompt of task is featured "Can you provide description of the region (location) in a sentence?".
Table 4: The results of combining with different MLLMs ROC and RTC tasks (box, test set).
Models
B@4
M
C
Models
ROC
RTC
LLaVA [36
5.02
13.15
55.61
17.61
VLaVA 36
54.72
53.57
LLaVA + Color
5.37
11.57
55.27
17.01
LLaVA Ours
60.59
61.22
LLaVA + Ours
5.53
14.00
59.75
19.08
InstructBLIP
17
49.81
26.46
InstructBLIP 17
1.24
8.70
9.89
7.95
InstructBLIP Ours
54.91
28.94
InstructBLIP + Color
1.27
8.26
14.16
6.92
LLaVA-HR 33
53.81
47.01
InstructBLIP + Ours
1.39
8.77
10.28
8.24
LLaVA-IIR Ours
58.92
58.60
that the model is adequately optimized on challenging samples. The early stop strategy allows us to attain superior model performance while reducing the impact of overfitring. The additional experiment about ES is shown Table
Impact of Different Text Prompts and the Size of Visual Prompts. We explore the impact of different text prompts and the size of visual prompts in Figure [0 and Figure [ respectively The results demonstrate that combining a clear and specific text prompt with an appropriate visual prompt size typically leads to improved controllability.
Limitations
While we have demonstrated visual prompt control by optimizing only visual tokens, our technique subject to a few limitations. First, there is some additional inference overhead, while various engineering approaches (such Ollama F) can significantly speed up the process. Therefore, this limitation can be reasonably overlooked. Second, our method is applicable only to white-box models and relies the basic capabilities of the models themselves. However, to these ongoing advancements in foundational models. Third, currently, our method supports only single region visual prompt, extending this to multi-region control is a direction for future work. Fourth, our current optimization strategy is relatively simple, and the selection of text prompts can also affect the optimization results. We plan to focus improving this aspect in future research.
approach is orthogonal
Conclusion
In this work, we present training-free method to integrate visual prompts into Multimodal Large Lan- guage Models (MLLMs) through learnable latent variable optimization. By adjusting visual tokens during inference, our approach enhances the attention to referring regions, enabling detailed descrip- tions and reasoning without additional training costs. Our method supports various referring formats such box, mask, scribble, and point. The results show that approach demonstrates strong out-of-domain generalization and interpretability, making it promising direction for embedding referring abilities into MLLMs.
Acknowledgments and Disclosure of F Funding
This work was supported by National Key R&D Program of China (No.2023YTB450088M) the National Science Fund for Distinguished Young Scholars (No.62025603), the National Natural Science Foundation of China (No. U22B2051, No. U21B2037, No. 62072389, No. 62302411), 1), the Natural Science Foundation of Fujian Province of China (No.2021)06003), and China Postdoctoral Science Foundation (No. 2023M732948).
/https//ollama.com/
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint vrXiv:2303.08772 2023.
[2] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pages
382-398, Springer, 2016.
[3] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus- Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7):e0130140) 2015.
[4] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203. 17274, 2022.
[5] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308..0 12966, 2023.
[6] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72. 2005.
[7] Gary Bradski. The opencv library. Dr. Dobb's 's Journal: Software Tools for the Professional Programmer, 25(1 11:1200-12 2000.
[8] Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee. Making large multimodal models understand arbitrary visual prompts. arXiv preprint arXiv:2312.00706
2023.
[9] Hila Chefer, Shir Gur, and Lior Wolf. Generic attention-mon explainability for interpreting bi-modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 397-406, 2021.
[10] Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 782-791, 2021.
[11] Chi Chen, Ruoyu Qin, Fuwen Luo, Xiaoyue Mi, Peng Li, Maosong Sun, and Yang Liu, Position- enhanced visual instruction tuning for multimodal large language models. arXiv preprint arXiv:2308. 13437, 2023.
[12] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao Shikra: Unleashing multimodal I1m's referential dialogue magic. arXiv preprint 2rXX::2306.1515 2023.
[13] Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-pla inference acceleration for large vision- languaee models. arXiv preprint arXiv:2403.0m 200.0664..6 2024.
[14] Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free free layout control with cross-attention guidance. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5343-5353, 2024.
[15) Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, al. Internvl: Scaling up vision foundation models and aligning for generic visual- linguistc tasks. arXiv preprint arXiv:2312. 14238, 2023.
[16] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vieuna: An open-source chatbot impressing 4pt-4 with 90%* chatgpt quality, March 2023.
[17] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances Neural Information Processing Systems, 36, 2024.
11

---

[18] Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang. Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Zhe Chen, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Kai Chen, Conghui He, Xingcheng Zhang. Jifeng Dai, Yu Qiao, Dahua Lin, and Jiaqi Wang. antenlm-xcomposer language model handling resolutions from 336 pixels to 4k hd. arXiv preprint ar arXiv:2404.065 12, 2024.
A pioneering large vision-
[19] Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, Meishan Zhang, Mong-Li Lee, and Wynne Hsu. Video-of-thought: Step-by-step video reasoning from perception to cognition. In Forty-first International Conference on Machine Learning, 2024.
[20] Hao Fei, Shengqiong Wu, Hanwang Zhang, Tat-Seng Chua, and Shuicheng Yan. Vitron: A unified pixel-level vision Ilm for understanding, generating, segmenting, editing, 2024.
[21] Hao Fei, Shengqiong Wu, Meishan Zhang, Min Zhang, Tat-Seng Chua, and Shuicheng Yan. Enhancing tideo-language representations with structural spatio-temporal alignment. IEEE Trunsuctions on Puttern Analysis and Machine Intelligence, 2024.
[22] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304. 15010, 2023.
[23] Yunpeng Gong, Liqing Huang, and Lifei Chen. Person re-identification method based color attack and joint defence. Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages
2022.
[24] Qiushan Guo, Shalini De Mello, Hongxu Yin, Wonmin Byeon, Ka Chun Cheung, Yizhou Yu, Ping Luo, and Sifei Liu. Regiongpt: Towards region understanding vision language model. arXiv preprint arXiv:2403.02330, 2024.
[25] Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A dataset for large vocabulary instance seg- mentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[26] Junwen He, Yifan Wang, Lijun Wang, Huchuan Lu, Jun- He, Jin-Peng Lan, Bin Luo, and Xuansong Xie. Multi-modal instruction tuned Ilms with fine-grained visual perception. arXiv preprint (xXiv:2403.02999 2024.
[27] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompp image editing with cross attention control. arXiv preprint urXiv:2208.011 01666, 2022.
[28] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pages 709-727. Springer, 2022.
[29] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787-798, 2014.
[30] Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung-Woo Ha, and Jun Yan Zhu. Dense text-to-image generation with attention modulation. Proceedings the IEEE/CVF International Conference on Computer Vision, pages 7701-7711. 11, 2023.
[31] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4015-4026. 2023.
[32] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu, Otter: A multi-modal model with in-context instruction tuning, arXiv preprint arXiv:2305.03722 2023.
[33] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[34] Weifeng Lin, Xinyu Wei, Ruichuan An, Peng Gao, Bocheng Zou, Yulin Luo, Siyuan Huang, Shanghang Zhang, and Hongsheng Li. Draw-and-understand:: Leveraging visual prompts to enable milms to comprehend what you want. arXiv preprint 2rX 2024.
12

---

[35] Haotian Liu, Chumyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, arXiv preprint arXiv:2310.03744, 2023.
[36]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[37] Junyu Lu, Ruyi Gan, Dixiang Zhang, Xiaojun Wu, Ziwei Wu, Renliang Sun, Jiaxing Zhang, Pingjian Zhang, and Yan Song. Lyries: Boosting fine-grained language-vision alignment and comprehension via semantic-aware visual objects. arXiv preprint arXiv:2312.05278, 2023.
[38] Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji. Cheap and quick: Efficient vision-language instruction tuning for large language models. Advances in Neural Information Processing Systems, 36, 2024.
[39] Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. Feast your eyes: Mfxture-of adaptation for multimodal large language models. arXiv preprint arXiv:2403.0 03003, 2024.
[40] Chuofan Ma, Yi Jiang. Jiannan Wu, Zehuan Yuan, and Xiaojuan Qi. Groma: Localized visual tokenization for grounding multimodal large language models. arXiv preprint arXiv:2404. 13013, 2024.
[41] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin EI- El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.071955 2023.
[42] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
[43] Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. /4824, 2023.
[44] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 9748-8763. PMLR, 2021.
[45] Kanchana Ranasinghe, Satya Narayan Shukla, Omid Poursaced. Michael S Ryoo, and Tsung- Yu Lin. Learning to localize objects improves spatial reasoning in visual-Ilms. arXiv preprint arXiv:2404.07449, 2024.
[46] Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Erix Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. arXiv preprint SIti:2211..33556 2023.
[47] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2021.
[48] Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about a red circle? visual prompt engineering for vlms. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1 11987-11997, 2023.
[49] Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. Advances in Neural Information Processing Systems, 35:14274-14289 2022.
[50] Gabriela Ben Melech Stan, Raanan Yehezkel Rohekar, Yaniv Gurwicz, Matthew Lyle Olson, Anahita Bhiwandiwalla, Estelle Aftalo, Chenfei Wu, Nan Duan, Shao- Yen Tseng, and Vasudev Lal. Lvlm-intrepret: An interpretability tool for large vision- vingon-language models. arXiv preprint 1rX1v:2404.03118. 2024.
[51] Zeyi Sun, Ye Fang. Tong Wu, Pan Zhang, Yuhang Zang, Shu Kong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. Alpha-clip: A clip model focusing on wherever you want. arXiv preprint arXiv:2312.038788 2023.
[52] Yunjie Tian, Tianren Ma, Lingxi Xie, Jihao Qiu, Xi Tang, Yuan Zhang, Jianbin Jiao, Qi Tian, and Qixiang Ye. Chatterbox: Multi-round multimodal referring and grounding. arXiv preprint
2rXiv:2401.13307, 2024.
13

---

[53] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2300. 13971, 2023.
[54] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[55] Ramakrishna Vedantam, Lawrence Zitmick, and Devi Parikh. Cider: Consensus-base image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575, 2015.
[56] Andreas Veit, Tomas Matera, Lukas Neumann, Jiri Matas, and Serge Belongie. Coco-text: Dataset and benchmark for text detection and recognition in natural images. arXiv preprint arXiv:1601.07144 2016.
[57] Teng Wang. Jinrui Zhang, Junjie Fei, Yixiao Ge, Hao Zheng, Yunlong Tang, Zhe Li, Mingqi Gao, Shanshan Zhao, Ying Shan, et al. Caption anything: Interactive image description with diverse multimodal controls. arXiv preprint arXiv:2305.088 22n.:2275500677,. 2027.
[58] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogylm: Visual expert for pretrained language models. arXiv preprint arXiv:23 I. 1.03079, 2023.
[59] Mingrui Wu, Oucheng Huang, Jiayi Ji, Jiale Li, Xinyue Cai, Huafeng Kuang, Jianzhuang Liu, Xiaoshuai Sun, and Rongrong Ji. Tradiffusion: Trajectory-based training-free image generation. arXiv preprint arXiv:2408. 09739, 2024.
[60] Mingrui Wu, Jiayi Ji, Oucheng Huang, Jiale Li, Yuhang Wu, Xiaoshuai Sun, and Rongrong
Ji. Evaluating and analyzing relationship hallucinations in large vision-language models. In Proceedings of the 41st International Conference on Machine Learning. volume 235 of Proceedings of Machine Learning Research, pages 53553-53570. PMLR, 21-27 Jul 2024.
[61] Mingrui Wu, Xuying Zhang, Xiaoshuai Sun, Yiyi Zhou, Chao Chen, Jiaxin Gu, Xing Sun, and Rongrong Ji. Difnet: Boosting visual information flow for image captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18020-18029, 2022.
[62] Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training- free box- box-constrained diffu- sion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7452-7461, 2023.
[63] Jiarui Xu, Xingyi Zhou, Shen Yan, Xiuye Gu, Anurag Arnab, Chen Sun, Xiaolong Wang, and Cordelia Schmid. Pixel aligned language models. arXiv preprint arXiv:2312.092237, 2023.
[64] Shiyu Xuan, Qingpei Guo, Ming Yang, and Shiliang Zhang. Pink: Unveiling the power of referential comprehension for multi-modal 11ms. arXiv preprint 1xXi:2310000588 2023.
[65] Jianwei Yang. Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310. 1441, 2023.
[66] Lingfeng Yang, Yueze Wang. Xiang Li, Xinlong Wang, and Jian Yang. Fine- fine-grained visual prompting. Advances in Neural Information Processing Systems, 36, 2024.
[67] Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow12: Revolutionizing multi- modal large language model with modality collaboration. arXiv preprint arXiv:2311. 244.22277, 2023.
[68] Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du Bowen Zhang, Zirui Wang, Liangliang Cao, Fuih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07700 2023.
[69] Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. Ferret-ui: Grounded mobile ui understanding with multimodal 1lms. arXiv preprint arXiv:2404.0577 055719, 2024.
[70] Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. arXiv preprint
10000 2023.
14

---

[71] Tongtian Yue, Jie Cheng, Longteng Guo, Xingyuan Dai, Zijia Zhao, Xingjian He, Gang Xiong. Yisheng Lv, and Jing Liu. Sc-tune: Unleashing senssconsistent referential comprehension in large vision language models. arXiv preprint arXiv:2403. 13263, 2024.
[72] Yufei Zhan, Yousong Zhu, Hongyin Zhao, Fan Yang. Ming Tang, and Jinqiao Wang. Griffon v2: Advancing multimodal perception with high-resolution scaling and visual-language co-referring. arXiv preprint 2tth::2400009333. 2024.
[73] Hao Zhang, Hongyang Li, Feng Li, Tianhe Ren, Xueyan Zou, Shilong Liu, Shijia Huang, Jianfeng Gao, Lei Zhang, Chunyuan Li, et al. Llava :roon:g: Grounded visual chat with large multimodal models. arXiv preprint arXiv:23/2.0294) 2023.
[74] Haotian Zhang. Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong- You Chen, Tsu- Jui Fu, William Yang Wang, Shih-Fu Chang, Zhe Gan, et al. Ferret-v2: An improved baseline for referring and grounding with large language models. arXiv preprint arXiv:2404.07973 2024.
[75] Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region of-interest. arXiv preprint arXiv:2307.0360) 2023.
[76] Xuying Zhang, Xiaoshuai Sun, Yunpeng Luo, Jiayi Ji, Yiyi Zhou, Yongjian Wu, Feiyue Huang, and Rongrong Ji. Rstnet: Captioning with adaptive attention on visual and non-visual words. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15465-15474, 2021.
[77] Yichi Zhang, Yinpeng Dong, Siyuan Zhang, Tianzan Min, Hang Su, and Jun Zhu. Exploring the transferability of visual prompting for multimodal large language models. arXiv preprint
24KY::2044.12000 2024.
[78] Yuechen Zhang, Shengju Qian, Bohao Peng. Shu Liu, and Jiaya Jia. Prompt highlighter: Interactive control for multi-modal llms. arXiv preprint arXiv:2312.00 22.0222,,,4 2023.
[79] Liang Zhao, En Yu, Zheng Ge, Jinrong Yang, Haoran Wei, Hongyu Zhou, Jianjian Sun, Yuang Peng, Runpei Dong, Chumrui Han, et al. Chatspot: Bootstrapping multimodal Ilms via precise referring instruction tuning. arXiv preprint arXiv:2300..00 09474, 2023.
[80] Qiang Zhou, Chaohui Yu, Shaofeng Zhang, Sitong Wu, Zhibing Wang, and Fan Wang. Region- blip: A unified multi-modal pre-training framework for holistic and regional comprehension. arXiv preprint arXiv:2338..0222 2023.
[81] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing language understanding with advanced large language models. arXiv preprint arXiv:2304.10530 2023.
[82] Lanyun Zhu, Deyi Ji. Tianrun Chen, Peng Xu, Jieping Ye, and Jun Liu. Ibd: Alleviating hallucinations in large vision-language models via image-biased decoding. arXiv preprint arXiv:2402. 18476, 2024.
15

---

A Broader Impact
The integration of Multimodal Large Language Models (MLLMs) has far-reaching implications across various sectors. These models enhance accessibility, improve education, advance healthcare, and revolutionize media and entertainment. They offer intuitive interfaces for diverse communication needs, assist in medical diagnosis and treatment, and enable immersive multimedia experiences. However, ethical considerations must be addressed to ensure equitable and responsible deployment. Collaboration among researchers, policymakers, and industry is essential to maximize the societal impact of MLLMs.
B Appendix supplemental material
B.1 Details of EMA and ES
EMA. Model weight Exponential Moving Average (EMA) is technique used to stabilize the training of deep neural networks by maintaining a smoothed version the model parameters. calculates the moving average of the model weights by giving more weight to recent updates while gradually decreasing the influence of past updates. Mathematically, EMA is defined as:
EMA = B() + (1 - 3) -)
where OMM is the EMA the model weights at time t, g00) is the model weights at time t, of(-1) is the EMA of the model weights the previous time step, 8 is the smoothing factor, ranging from to 1. EMA helps to stabilize the training process by reducing the variance the parameter updates, which can prevent the model from diverging oscillating during training. We employ the EMA on the learnable latent variable in our experiments during visual token optimization.
ES. Early Stop (ES) is regularization technique commonly used during the training of machine learning models to prevent overfitting. The main idea behind ES is to monitor the performance of the model on validation set during training and stop the training process if the performance begins deteriorate. Specifically, in our experiments, ES tracks the loss on the test sample and halts the visual token optimization process if the metric does not improve for certain number of consecutive epochs. Mathematicallyy ES can be described as follows:
Let I (6) denote the loss at optimization process and let A represent a predefined threshold for acceptable loss degradation, then we
Stop optimizing if abs (I. r /I 2 for consecutive process.
We 0.25 in our experiments. ES helps prevent overfitting by stopping the optimization process before the model starts to lose generalization ability. It provides a simple yet effective way to regularize the optimization process and improve the generalization performance of the model.
B.2 Additional Experiment Details
Details of Relevancy Score. To elucidate the influence of input visual pixels outputs, employ a technique known as Relevancy Map. This map highlights regions or features of the input data that contribute most significantly to the model's decision-making process. Specifically, the Relevancy Map assigns importance scores to different parts of the input, indicating their relative impact the model's output. This interpretability tool not only enhances our understanding of the model's sss behavior but also facilitates error analysis and model debugging. More details can be found
in the paper [6]] 3
10 50].
In our experiments, we provide the relevancy scores alongside model predictions to provide insights into the model's decision- decision-making process. In MLLMs, the typical use of Key- Value cache technique LLMs, for convenience, we propagate the relevance from the model's first output token to the input of L.M e, and e, ). Then the relevancy scores corresponding to visual tokens are reshaped into grid that matches the layout of the original image. Since relevancy maps are akin to attention maps and often exhibit significantly higher values in localized regions, taking the average may dilute their
16

---

Table 5: The ablation of T, a, EMA. We report Accuracy and Relevancy ROC task (validation set). The best performance is highlighted in bold, while the paired Relevancy scores are indicated with underlines.
Accuracy
Relevancy
T
a 200
57.00
57.50
59.00
60.50
58.00
0,1667
0.1679
0.1698
0.1743
0.2058
a 300
57.00
57.50
60.00
59.00
58.50
0.1667
0.1684
0.1722
0.1986
0.2792
a 500
$7.00
58.50
60.50
61.00
53,50
0.1667
0.1689
0.1749
0.2238
2.5542
$7.00
59.50
60.50
56.00
43.00
0.1667
0.1694
0.1799
0.2650
0.4542
w/ EMA (a= 400)
3 0.3
$7.00
$7.00
58.00
60.50
62.00
0.1667
0.1674
0.1689
0.1712
0.1753
8 0.5
$7.00
57.50
60.00
61.00
60.00
0.1667
0.1679
0.1704
0.1767
0.2071
3 U.7
57.00
57.50
61.00
62.00
54.00
0.1667
0.1683
0.1728
0.1957
0.2992
Table 7: Impact of LLM Size in Dif- ferent MLLMs on ROC task (box,
Table 6: The ablation of ES (a =
400, 3 0.5, validation set).
test set).
T +
0
4
5
Models
Vanilla
Ours
Acc. Rel. 0.1667
57.00
60.50
62.50
LLaVA-I.5-78
54.72
60.59
0.1841
0.1937
LLaVA-I.5 1,5-1BBB
55.69
58,40
InstructBLIP-7B
49.81
54.91
InstructBLIP-13B
54.33
59.24
significance. So we extract the max value in the referring region of relevancy map final relevancy score.
It is also worth noting that we found relevancy map plays a similar role to attention map, directly modeling the relationship between input and output of the model. This suggests that we may be able to directly control the model's 's output based on relevancy map. However, calculating the relevancy map requires computing the gradient for each relevant tensor in the model. For convenience, the approach presented in this paper only utilizes attention for implementation, while the relevancy map only be used to assess the extent of visual impact on the output.
Details of Implement. We apply low-bit quantization to the basic model we implemented to further optimize memory usage. Nevertheless, we still achieve performance competitive with training-based methods (without quantization).
Details and Input Examples of ROC Task. The box and mask are from the LVIS GT boxes and mask, the scribble and the point are randomly sampled inside the boxes. It is worth noting that we follow Ferret to choose negative object whose central point is close to the GT object. Although this is somewhat disadvantageous for us, still achieve competitive performance compared to other methods. And we show the input examples of different methods on ROC task in Figure 7
B.3 Additional Experiments
Application scene text recognition (OCR). Results are shown in Figure Our method can also perform referring region text recognition.
Examples on Out-of-Domain Task. We present examples of the performance on out-of-domain tasks OCR and Screenshots. As shown in Figure 6,, compared to Ferret, our method correctly identified the text in the referring region. Additionally, as shown in Figure our method correctly recognized the app in the mobile screenshot, unlike Ferret.
Effeet on More MILMs. We validate our method on various MILMs, including LLa -LLVA--1.5-71 BistructBLII-7BB. and I.LVA--HR-7B, InstructBLIIP eemploys a cross-attention image-text alignment module called Q-Former. Specifically, in InstructBIIIP, the interaction between visual tokens and text tokens occurs within Q-Former utilize the cross-attention map from Q-Former to optimize
17

---

Table 8: The inference cost with different actual output token numbers single GTX3090 GPU.
LLaVA + Ours with T without using Early Stop here.
Models
speed(s)
Max GPU Mem.
LLaVA tokens)
1.22
7G
V. + Ours (7 tokens)
3.56
I4G
LLa (436 tokens)
5.78
7G
LLa VA + Ours (439 tokens)
7.45
14G
visual tokens. LLaVA-HR supports input images with larger resolutions. The results are shown in Table
Comparison on Referring Description Task. We also validate our method through the Referring Description Task. Specifically, we construct the test set based on region-text pairs from the Ref- COCOg test split and evaluate the method using traditional captioning metrics BLEU@ (EUU@4 4 (B @4) 42),
METEOR (M) [6), CIDEr-D (C) [55], and SPICE (S) 22.. However, it is important to note that these metrics significantly influenced by the style and distribution of the model's output text. Typically, outputs that are more similar in distribution to RefCOCOg yield better results, while those with some unique styles lead poorer results. Therefore, this experiment is only used for internal validation of the model and not for comparison between different models. The results are shown in Table Our method consistently improves the model's referring description performance.
Impact of LLM Size on Different MLLMs. We validate the impact of L.M size on LLaVA-1.5 and InstructBIIIP IIP, focusing on the 7B and 13B models. The results are shown in Table 7 Our method exhibited poorer performance in LLaVA-1.5-13B which may be due to the increased number of attention maps, making the optimization of visual tokens more challenging. Therefore, it may be essential to adopt different hyprraarameters for different models. In contrast, in InstructBLIP-7B and InstructBLIP-I33 our method consistently yielded performance improvements. This is likely because the interaction between visual tokens and text tokens occurs in Q-Former for InstructBLIP. thereby mitigating the optimization challenges associated with larger LLMs.
We compare the inference cost of our method and the basic LLaVA model. LLaVA + Ours model with T 5 and does not use Early Stop here. Results are shown Table X when outputting only 7 tokens, However, when generating about 400 tokens, the proportion the extra inference time produced by our method significantly decreases. When combined with an early stop strategy, the proportion of additional inference time will be further reduced.
Inference Cost.
method noticeably adds approximately 2 seconds of inference time.
Impact of Different Text Prompt. Results are shown in Figure 10 Different text prompts can significantly affect the performance of our method. For instance, our method fails when using an ambiguous text prompt like "describe the region in the image." However, it succeeds with more specific referring text prompt such as "what is unusual about the region of the building?" Therefore, clear and specific text prompts to achieve better control and performance.
it is recommended
Impact of the Size of Visual Prompt. Results are shown in Figure T1 when they do not fully cover the referring object, failure control may occur. This is because the highest attention response for the desired outcome may fall on any unexpected area of the object. Therefore, is recommended cover the object with a larger-sized visual prompt.
For box and mask prompts,
Comparing Highlight Token and Context Token based Optimization. We compare Highlight Token and Context Token based Optimization as shown in Figure [2] Directly optimizing based on the highlight token is faster but also prone to overfitting. This may be due to the direct connection between the highlight token and visual token, while other text tokens contain redundant visual associations. However, this also ignores the potential influence of other text tokens.
More Examples of Referring MILMS with Scribble and Point. Figure we also show more examples of referring MLLMs with scribble (right) and point (left).
18

---

LLaVA
LLaVA + Color
LLaVA Blur
Prompt: Is the object of the image necktie or a person?
Prompt: Is the object in red bounding box of the image necktie person?
Prompt: the object of the image necktie person?
Training Method
LLavA Ours
Prompt: Is the object [0.49, 0.38, 0.60, 0.84] of the image necktie person?
Prompt: Is the object of the image necktie or a person?
Figure 7: The input examples of ROC Task.
Prompt: what the in the region? Models
LLaVA
LLaVA + Ours
Input Image
Biman
Sky Chels
Output
The text in the region reads "BI-MAN."
The text the region is 'Sky Chefs." "
Figure 8: Application on natural scene text recognition (OCR).
19

---

Prompt: What is this app in the picture used for?
Models
Input Image
Output
The app in the picture is mobile phone app, which is designed to provide various functionalities to the user. It features a collection of icons representing different apps and services, such as G Google, Calendar, Camera, and others. These icons are organized in grid-like pattem, making it easy for users to access and navigate thrughh the different functionalities offered by the app. The app is likely used for communication, organization, and various other tasks that can be performed on smartphone.
Seauch phocee
LLaVA
CG
phocee
The app seems to be a communication app, as suggested by the icon displaying a person on it. It's probably used for messaging or staying connected with others.
Cimol
Tilendde
Comera
Ferret
The app in the picture is used for tracking and managing time, specifically for keeping track of time spent on various tasks. is designed to help users stay organized and productive by providing visual representation on of their time usage. This app can be particularly useful for individuals who want to manage their time effectively, such as students, professionals, or anyone who wants to optimize their daily activities.
Colention
LLaVA + Ours
Coocll
Prompt: What is this app in the picture used for?
Models
Input Image
Output
The app in the picture is mobile phone app, which is designed to provide various functionalities to the user. It features a collection of icons representing different apps and services, such as G Google, Calendar, Camera, and others. hese icons are organized in grid-like pattern, making it easy for users to access and navigate throuh the different functionalities offered by the app. The app is likely used for communication, organization, and various other tasks that can be performed on smartphone.
phione
LLaVA
Dicck
CQ
Seatch
STeel
Calencii
Cunera
The app is a map application, typically used for navigation purposes. It's an icon on the home screen, offering easy access to directions and location services
Ferrel
Search ptrscte
The app in the picture is game launcher, which is used to quickly access and launch various games on smartphone. It provides a convenient way to manage and organize the games installed on the device, making it easier for users to find and play their favorite games.
LLaVA + Ours
Cancu
Figure 9: Examples of comparing with training method Ferret on Screenshot.
20

---

Prompt: describe the region in the image.
The image features group of people gathered outside a building, possibly a church. There are four people in the scene, with two of them wearing ties, indicating formal or semi- formal occasion. The group consists of bride, groom, and two other people. They are all standing together, possibly posing for picture. The building has black and white color scheme, adding touch elegance the scene.
LLaVA
The image features group of people gathered outside a building, possibly church or wedding venue. There are four people the scene, with two them wearing ties, suggesting a formal occasion. The group is standing on brick walkway, and there potted plants nearby, adding to the ambiance of the location. The people are engaged conversation, creating a lively atmosphere.
LLaVA
+Ors
Prompt: what unusual of the region of the building?
The unusual aspect of the building's region is the presence of potted plant on the side of the building. is not common to see potted plant on the side of a building, especially in a setting where people are dressed formal attire, such wedding. The potted plant adds a unique touch to the scene and stands out among the other elements.
LLaVA
The unusual aspect of the building's region is the presence of a large window in the middle of the structure. This is not a common feature most buildings, as windows are typically placed on the sides or comers of a building. The large window in the middle of the building adds an interesting architectural element to the structure.
LLaVA + Ours
Figure 10: Impact of different text prompt. The clear and specific text prompt attains a better performance.
21

---

Prompt: what is unusual of the region of the building?
The unusual aspect of the building's region is the presence of potted plant on the side of the building. it not common see a potted plant on the side of a building, especially in a setting where people are dressed in formal attire, such wedding. The potted plant adds unique touch to the scene and stands out among the other elements.
LLavA
The unusual aspect of the region of the building is the presence a potted plant on the ground. Typically, potted plants are placed on elevated surfaces, such as tables or shelves, to provide better air circulation and prevent overwatering. However, in this image, the potted plant located on the ground, which is not common practice. This might be due to the specific needs of the plant or the design choice of the building's interior.
The unusual aspect of the building's region is the presence of large window in the middle the structure. This not common feature in most buildings, as windows are typically placed on the sides comers of building, The large window the middle of the building adds an interesting architectural element to the structure.
LLavA Ouurs
The unusual feature the region of the building is the presence a stained glass window. Stained glass windows are not commonly found in modern buildings, especially in urban settings. This adds unique and artistic touch to the building, making it stand out from other structures in the area.
Figure 11: Impact of the size of visual prompt. The larger prompt size attains a better performance.
T=0
T=
T-2
T=3
T=
T
w/
highlight
taken
"The object the image turtle."
a turtle."
a smail shell.
is snail sbell ora sbell-sh.
is small decorated, and possibly fake
is amall. decorated, and possibly take."
w/
The object
context
in the image turtle."
a turtle." turtia.
is turtle."
is snail shell."
is smail shell."
tokmn
w/
"The okjjct the image turtle."
highlight
turtle."
is green frog
sitting on top of a."
is frog sitting turtie."
is frog."
frog."
tokan
w/
"The object
context
the image
is turtle."
is turtle
is turtle."
isa green frog." " is green frog."
token
turtle."
Prompt: "What's in the image?"
Figure 12: Comparing highlight text token and context text token based optimization.
22

---

T-0
T=
T-3
T-
T=
"The wearing the hat weariigg green hat."
The color of the bal is red.
w/o EMA
The the two pieces of the image the and the hues of the two."
The image of the image is The image of the image is The image of the image is a.
red hat" a red het."
"The person wearing the hat is wearing red hat." red hat." groon hat."
w EMA
The color hat person weering red."
a red hot."
Prompt: "What's color of hat the person wearing?"
Figure 13: Impact of EMA. The EMA stabilizes the optimization process.
Input Image & Visual Prompt
Prompt: What is the object?
Prompt: what is oolorof thisbanch?
Input Prompt
The LLaVA: The object leather bag satchel.
LLaVA: The colorof this bench is yellow.
& Output Text
LLaVA + Ours: The object is ring with a large stone in it.
LLaVA + Ours: The of this berch is green.
Figure 14: More examples of referring MLLMs with scribble (right) and point (left).
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction include the claims made in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discuss the limitations of the work Sec 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper includes experiments, and with related information needed to reproduce the main experimental results in See
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [No]
Justification:
The code will be released in the Github.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.com wttp:ccpppbccccceee guwdes/dde/eeee Coooeoooddesssseeeeeee http:///ll./.iiiiiiooo//////eae/
ssinislicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
The paper specify all the experimental setting details in Sec Sec
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: expensive.
The error bars are not reported because it would be too computationally
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
26

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provide sufficient information on the computer resources needed to reproduce the experiments in Sec 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The authors have reviewed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The paper discuss both potential societal impacts of the work in Sec.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original paper that produced the code package dataset.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The documentation is provided alongside the new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "ICGIedW36T", "reviewer": "Reviewer_VBnv", "summary": "This paper proposes a training-free ControlMLLM, which uses optimizable latent variables to inject visual prompts into multimodal large MLLMs. The core idea is to adjust the visual token outputs of the MLP during inference, to control the attention response and ensure that the text prompt tokens focus on the indicated visual regions. It enhances the intensity of the indicated regions in the attention maps by optimizing learnable latent variables based on an energy function, enabling reference to various visual prompts (including boxes, masks, scribbles, and points) without model training, fine-tuning, or additional data. The method demonstrates out-of-domain generalization and interpretability, providing a promising direction for integrating referential capabilities into MLLMs. Experiments show that the proposed model is effective.", "questions": "See above", "limitations": "None", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1), Prompt tuning for MLLMs is an interesting direction, and the proposed ideas are both effective and simple. Motivated by text-to-image works, ControlMLLM aims to control the attention map between the textual tokens and visual patches. This idea makes sense and provides a new direction to improve MLLMs.\n\n2), The comparisons and ablations show the effectiveness of the proposed model.\n\n3), The writing is clear, and the figures and tables help to understand the motivations.", "weaknesses": "1), The visual prompt shows great improvements over the base model LLaVA, however, it requires additional guidance information and more inference time, which may limit the applications of ControlMLLM. Especially in some complex scenarios where the guidance signals are unavailable.\n\n2), In addition, the region signal $r$ plays a core role during optimization, and it controls the output of MLLMs. What if $r$ itself is wrong?  It may mislead the MLLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# **Rebuttal**\n\n>#### ****Q1: The visual prompt shows great improvements over the base model LLaVA, however, it requires additional guidance information and more inference time, which may limit the applications of ControlMLLM. Especially in some complex scenarios where the guidance signals are unavailable.****\n\nThank you for your comment. Indeed, all settings that apply MLLMs to visual grounding, such as Referring MLLMs, require visual prompts [1, 2, 3, 4, 5, 6], this is not unique to our approach. However, our setup allows visual prompts to be injected into MLLMs without additional training, offering greater flexibility. Additionally, our method does not require fine-tuning the large model, which provides stronger generalizability. These features make our model better suited for transfer to other complex scenarios. As shown in Table 2, our method can easily transfer referential capabilities to out-of-domain data.\n\nRegarding the time issue, the comparison is 5.78s vs 7.45s when model output about 400 tokens, as shown in Table 7. Our model is indeed slower, but the difference is acceptable. We will further optimize this aspect in future work.\n\n>#### ****Q2:  In addition, the region signal plays a core role during optimization, and it controls the output of MLLMs. What if itself is wrong? It may mislead the MLLMs.****\n\nThank you for your insightful comments and valuable suggestions. Your feedback has greatly guided our research direction. Visual prompts and text prompts, which are user-provided instructions during inference, play a crucial role in Human-Computer Interaction. While users should accurately specify the desired region, we acknowledge the risk of incorrect input potentially misleading the MLLMs. To address this, we propose incorporating validation mechanisms or fallback options in future versions to ensure robustness even when faced with inaccurate prompts. This approach aims to improve the reliability of the method in real-world applications.\n\n[1] Ferret: Refer and ground anything anywhere at any granularity\n\n[2]  Draw-and-understand: Leveraging visual prompts to enable mllms to comprehend what you want\n\n[3] Kosmos-2: Grounding multimodal large language models to the world \n\n[4] Shikra: Unleashing multimodal llm’s referential dialogue magic\n\n[5] Ferret-ui: Grounded mobile ui understanding with multimodal llms \n\n[6] Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kUGUdRmsyI", "reviewer": "Reviewer_hD7U", "summary": "This paper introduces a training-free approach to integrate visual prompts into MLLMs using learnable latent variables, aiming to enhance the model's interpretability and generalization. It adjusts visual tokens from MLP outputs and optimizes latent variables with an energy function to improve attention on relevant visual regions.", "questions": "1. It is recommended to validate the effectiveness of the method on additional MLLMs.\n2. Additionally, please note that the title listed on the paper submission does not match the title in the PDF.", "limitations": "The limitations have been discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThis paper demonstrates and visualizes how the attention between prompt tokens and visual tokens differs across various layers.\n2.\tFigures 2 and 4 are helpful to understand the method.\n3.\tThe quantitative and visualization experiments validate the effectiveness of the proposed approach.", "weaknesses": "1. Figures 3(a) and 3(b) show visualization results for different values of η. The paper mentions that in 3(a), η is too small to effectively control the attention. However, the focus of attention map in 3(b) does not significantly differ from that in 3(a).\n2. Table 3 shows that the highest accuracy is achieved when α = 400 and T = 3. Why, then, is the value of T ultimately chosen as 4?\n3. Since there are no recent developments, contributions, or updates to the MLLM, its detailed presentation in Eq.s (1)-(3) might be unnecessary and could be omitted. Besides, some symbols are not defined, such as, $I_i$, $A_i^{(ct)}$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Rebuttal\n\n>#### **Q1: Figures 3(a) and 3(b) show visualization results for different values of $\\eta$. The paper mentions that in 3(a), $\\eta$ is too small to effectively control the attention. However, the focus of attention map in 3(b) does not significantly differ from that in 3(a).**\n \n\nThank you for your question. To clarify, there is a noticeable difference in the response of the green hat region between Figures 3(a) and 3(b). This indicates that in certain layers, the attention in the green hat region has a higher response, and these layers may play a crucial role in determining the model's output. It is important to note that not all layers' visual tokens determine the model's output[1]. Some layers might purely serve the purpose of organizing language, which explains why we cannot achieve our goal by editing the attention in all layers directly.\n\nHowever, our approach can implicitly guide the model to cause changes in the attention responses in certain layers without excessively affecting the model's ability to organize language. We will clarify this point in the final version.\n\n>#### **Q2: Table 3 shows that the highest accuracy is achieved when $\\alpha$ = 400 and T = 3. Why, then, is the value of T ultimately chosen as 4?**\n\nAs outlined in the 'Impact of EMA and ES' section, we combined EMA (Exponential Moving Average) and ES (Early Stopping) strategies to enhance the stability and convergence speed of model optimization. We observed in Table 6 that a larger $T$ value, when combined with ES, led to better results. Therefore, we chose a slightly larger $T = 4$ to ensure adequate optimization for more challenging samples, even if it involved a slight trade-off in peak accuracy. This decision was based on our observation that a $T$ value of 4 and a relevancy score around 0.18 helped maintain consistency and robustness across various validation scenarios.\n\n>#### **Q3: Since there are no recent developments, contributions, or updates to the MLLM, its detailed presentation in Eq.s (1)-(3) might be unnecessary and could be omitted. Besides, some symbols are not defined, such as, $I_i$ and $A_i^{(ct)}$.**\n\nWe understand the importance of clarity and will refine the manuscript to ensure that all terms are well-defined. Additionally, we will reassess the inclusion of these equations to ensure that the presentation remains concise and focused on novel contributions.\n\n>#### **Q4: It is recommended to validate the effectiveness of the method on additional MLLMs.**\n\nAs demonstrated in Table 5, we have already validated our method on several additional models, such as InstructBLIP. These experiments provide preliminary evidence of the method's generalizability. We also provide more results in the following table.\n\n| Model         | Task | Vanilla | Ours  |\n|---------------|------|---------|-------|\n| **LLaVA1.5**  | ROC  | 54.72   | 60.59 |\n|               | RTC  | 53.57   | 61.22 |\n| **InstructBLIP** | ROC  | 49.81   | 54.91 |\n|               | RTC  | 26.46   | 28.94 |\n| **LLaVA-HR**  | ROC  | 53.81   | 58.92 |\n|               | RTC  | 47.01   | 58.60 |\n| **Monkey**    | ROC  | 55.26   | 60.68 |\n|               | RTC  | 55.59   | 63.39 |\n\n>#### **Q5: Additionally, please note that the title listed on the paper submission does not match the title in the PDF.**\n\nThank you for pointing this out. The discrepancy arose due to a last-minute change in the paper title after the abstract submission deadline. We will ensure that the final version of the paper has a consistent title across all submission materials to avoid confusion.\n\n[1] An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "elufAECwzT", "reviewer": "Reviewer_1iuY", "summary": "The paper introduces a training-free approach to improve the referring capabilities of multimodal large language models (MLLM). In particular, the authors iteratively adjust the attention maps using a learnable latent variable, which is based on energy functions. They empirically validate the efficacy of their method on referring classification tasks, leveraging the foundation of LLaVA.", "questions": "Please see the Weaknesses.", "limitations": "The authors have discussed it in Sec. 6.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The paper is generally easy to read and follow.\n+ The figures are clear, especially the visualization of attention maps under different conditions.\n+ The proposed method is training-free and theoretically plug-to-play with different foundation models.", "weaknesses": "1. The generalization ability of the proposed method has not been fully verified. (a) It's uncertain whether the method can be applied to MLLMs beyond LLaVA. As the foundational model strengthens, the method's effectiveness could potentially diminish. (b) The energy function, based on a soft mask, heavily relies on the quality of segmentation models. (c) As seen in Table 3, the results appear to be sensitive to hyperparameters. This raises the question: do we need to meticulously adjust the parameters for each model and sample? If that's the case, the practicality of this method could be questionable.\n\n2. There is a naive baseline that requires discussion: directly cropping the referred region and feeding it into the LLM.\n\n3. In Tab. 2&3, the tasks and compared models are indeed limited. It is recommended to discuss more state-of-the-art MLLMs on wider tasks and datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Rebuttal\n>#### **Q1: The generalization ability of the proposed method has not been fully verified. (a) It's uncertain whether the method can be applied to MLLMs beyond LLaVA. As the foundational model strengthens, the method's effectiveness could potentially diminish. (b) The energy function, based on a soft mask, heavily relies on the quality of segmentation models. (c) As seen in Table 3, the results appear to be sensitive to hyperparameters. This raises the question: do we need to meticulously adjust the parameters for each model and sample? If that's the case, the practicality of this method could be questionable.**\n\n(a) We have included more results, such as InstructBLIP model and Referring Description task in Tables 4 and 5 to demonstrate the generalization ability of our method. Our approach fundamentally enhances MLLMs, which primarily possess classification capabilities, by endowing them with referring abilities. Specifically, our method provides MLLMs with localization capabilities rather than relying solely on classification. Since classification relies on the foundational model, our method is orthogonal to the foundational model. As seen in Table 5, the less significant improvement on LLaVA-13b compared to LLaVA-7b is due to the increased complexity of the LLaVA-13b decoder, which makes optimization more challenging. Thus, the effectiveness of our method is related to the difficulty of optimizing the model, not necessarily diminished by the enhancement of the foundational model. \n\nFollowing your suggestions, we have included several methods beyond LLava, as detailed in Q3. We will create a dedicated section to discuss this area, which will make our paper's contributions even more robust.\n\n(b) Our method only requires an additional visual prompt during inference and does not depend on segmentation models, although they could be an optional technical route. In our approach, the soft mask is calculated using OpenCV functions, ensuring it is not heavily reliant on segmentation models' quality.\n\n(c) As demonstrated in Table 5, we conducted experiments on the InstructBLIP model using the same hyperparameters as LLaVA, achieving superior performance. Although fine-tuning the hyperparameters could lead to even better results, we plan to further improve our optimization strategy to make it more adaptable to different models.\n\n>#### **Q2: There is a naive baseline that requires discussion: directly cropping the referred region and feeding it into the LLM.**\n\nThanks for your constructive suggestion. Following your suggestion, we have provided relevant results for reference in the following table. Given that LLaVA's image preprocessing resizes images to 224x224, the classification performance on small objects might be better, potentially resulting in superior performance compared to LLaVA + Blur. We will add the above discussion into the new version.\n\n| Model           | ROC   | RTC   |\n|-----------------|-------|-------|\n| **LLaVA**       | 54.72 | 53.57 |\n| **LLaVA + Blur**| 73.39 | 83.60 |\n| **LLaVA + Crop**| 82.04 | 88.78 |\n| **LLaVA + Ours**| 60.59 | 61.22 |\n\n>#### **Q3: In Tab. 2&3, the tasks and compared models are indeed limited. It is recommended to discuss more state-of-the-art MLLMs on wider tasks and datasets.**\n \nThanks for your great commnet. We have included results for the RD task and the InstructBLIP model in Tables 4 and 5. It is also noteworthy that in the RD task, the performance differences among various foundational models are significantly influenced by the task and evaluation strategies. In contrast, in binary classification tasks like ROC and RTC, the results from different foundational models are relatively robust. This is why we followed Ferret in primarily experimenting on ROC and RTC tasks. We also provide more results in the following table.\n\n | Model         | Task | Vanilla | Ours  |\n|---------------|------|---------|-------|\n| **LLaVA1.5**  | ROC  | 54.72   | 60.59 |\n|               | RTC  | 53.57   | 61.22 |\n| **InstructBLIP** | ROC  | 49.81   | 54.91 |\n|               | RTC  | 26.46   | 28.94 |\n| **LLaVA-HR**  | ROC  | 53.81   | 58.92 |\n|               | RTC  | 47.01   | 58.60 |\n| **Monkey**    | ROC  | 55.26   | 60.68 |\n|               | RTC  | 55.59   | 63.39 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eoq5Ivant2", "reviewer": "Reviewer_LggR", "summary": "This paper proposes a method to integrate visual prompts into MLLMs without requiring additional training. The key idea is to optimize a learnable latent variable to enhance the attention response to visual tokens during inference, thereby improving the model's ability to focus on specific regions in the visual input.", "questions": "See weakness.", "limitations": "See weakness.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "The proposed approach does not requires additional training for unseen datasets.", "weaknesses": "1. Very poor writing. The use of abbreviations and full terms is very inconsistent. For example, \"Multimodal Large Language Models\" is spelled out in full on Line 29-30, while the abbreviation \"MLLMs\" is used frequently earlier in the text. Additionally, \"Linguistic\" in Line 34 should be \"textual.\" There are many grammatical errors in Figure 's prompt: \"What's color of hat the person wearing?\" and output: \"The person wearing the hat is wearing a green hat.\". Even the title in section 4.2 contains grammatical errors.\n2. This paper lacks novelty and is an incremental form of previously proposed methods with no innovative points.\n3. In Line 3, the authors claim that attention connects visual tokens and textual tokens, but in Line 33, it changes to MLP.\n4. The experimental results are insufficient and lack numerous baselines, such as LLaVA1.5[1], LLaVA-NeXT[2], Monkey[3], and Qwen-VL[4].\n5. The motivation for this study is insufficient. I think there is a baseline that by making the prompt descriptions clearer and more comprehensive based on the original MLLM. This baseline can also leverage MLLM's inherent ability to focus on specific regions.\n\n[1] Improved Baselines with Visual Instruction Tuning.\n\n[2] LLaVA-NeXT: Improved reasoning, OCR, and world knowledge.\n\n[3] Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models.\n\n[4] Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Rebuttal\n\n>#### **Q1: The use of abbreviations and full terms is very inconsistent. For example, \"Multimodal Large Language Models\" is spelled out in full on Line 29-30, while the abbreviation \"MLLMs\" is used frequently earlier in the text. Additionally, \"Linguistic\" in Line 34 should be \"textual.\" There are many grammatical errors in Figure 's prompt: \"What's color of hat the person wearing?\" and output: \"The person wearing the hat is wearing a green hat.\". Even the title in section 4.2 contains grammatical errors.**\n\nWe have provided the full term \"Multimodal Large Language Models\" and its abbreviation \"MLLMs\" in earlier text，such as lines 2 and 19. Thank you for pointing out the need for consistent terminology and other writing issues. We will correct these in the final version, including using \"textual\" instead of \"linguistic\" and addressing the grammatical errors.\n\n>#### **Q2: This paper lacks novelty and is an incremental form of previously proposed methods with no innovative points.**\n\nThank you for your comments. We appreciate the opportunity to reiterate the innovations of our paper. Firstly, our motivation is novel. This paper introduces a setting for embedding visual prompts without additional training, enabling the integration of visual prompts into existing models conveniently. On the technical side, we leverage the idea of visual prompts and introduce a test-time prompt tuning strategy to adjust the attention distribution of MLLMs and facilitate the injection of visual prompts. This technique also offers valuable insights for improving the interpretability of MLLMs. Our contributions have been recognized by Reviewer VBnv as well.\n\nWe will rephrase the contributions in the revised version, and your suggestions will help make our paper clearer. Thank you for your valuable feedback.\n\n>#### **Q3: In Line 3, the authors claim that attention connects visual tokens and textual tokens, but in Line 33, it changes to MLP.**\n\nIn lines 5 and 34, we explain that specific MLP layers can influence the attention responses between visual and textual tokens. We provide a detailed analysis of this in Section 4.1, demonstrating how MLP outputs can control the interaction between these tokens.\n\n>#### **Q4: The experimental results are insufficient and lack numerous baselines, such as LLaVA1.5, LLaVA-NeXT, Monkey, and Qwen-VL.**\n\nThank you for your constructive feedback. Our primary baseline is LLaVA1.5, as indicated in the \"Experiment Details\" section. We also present experiments on InstructBLIP in Table 5. Following your suggestion, we provide additional results in the following table. Qwen-VL is a trained referring MLLM, and LLaVA-Next is a recently released project supporting high-resolution input, both of which are classic and influential works. However, due to time and cost constraints, we conducted experiments on LLaVA-HR [1] as an alternative to LLaVA-Next. In the final version, we will include all these methods, including LLaVA-Next, for discussion and comparison\n\n | Model   | Task | Vanilla | Ours  |\n|---------------|------|---------|-------|\n|Training method||||\n|**Qwen-VL**|ROC|72.6|-|\n| |RTC|64.7|-|\n|Training-free method||||\n| **LLaVA1.5**  | ROC  | 54.72   | 60.59 |\n|               | RTC  | 53.57   | 61.22 |\n| **InstructBLIP** | ROC  | 49.81   | 54.91 |\n|               | RTC  | 26.46   | 28.94 |\n| **LLaVA-HR**  | ROC  | 53.81   | 58.92 |\n|               | RTC  | 47.01   | 58.60 |\n| **Monkey**    | ROC  | 55.26   | 60.68 |\n|               | RTC  | 55.59   | 63.39 |\n\n[1] Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models.\n\n>#### **Q5: The motivation for this study is insufficient. I think there is a baseline that by making the prompt descriptions clearer and more comprehensive based on the original MLLM. This baseline can also leverage MLLM's inherent ability to focus on specific regions.**\n\nOur primary motivation is to introduce referring capabilities to MLLMs without significant training costs, rather than just proposing the task of injecting referring capabilities. Regarding the baseline suggested, it primarily serves to demonstrate the necessity of visual prompts. As discussed in Section 2.2, extensive prior work on training referring MLLMs has validated the importance of visual prompts. They simplify the referring process for users, reducing the need for precise text prompts and improving interaction efficiency. Moreover, clearly describing specific regions through text prompts can be challenging and unfriendly for users. Constructing such a baseline requires substantial effort, which is impractical.\n\nThank you for your feedback. Indeed, the value of \"referring MLLMs\" has been explored and studied by many researchers [2-7], and this is not the primary contribution of our paper. The significance of this task lies in addressing scenarios that are challenging for language instructions alone, serving as a valuable complement to them. Constructing the baseline you mentioned is currently quite challenging and a key area for future research, which is one of the significant aspects of the referring MLLMs task.\n\nOur core contribution is in embedding visual prompt information into MLLMs without additional training, which is highly innovative for the field. This approach enables any MLLM to gain referential capabilities plug-and-play while maintaining generalizability. We hope this clarifies our contribution and addresses your concerns.\n\n[2] Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. \n\n[3] Ferret: Refer and ground anything anywhere at any granularity\n\n[4]  Draw-and-understand: Leveraging visual prompts to enable mllms to comprehend what you want\n\n[5] Kosmos-2: Grounding multimodal large language models to the world \n\n[6] Shikra: Unleashing multimodal llm’s referential dialogue magic\n\n[7] Ferret-ui: Grounded mobile ui understanding with multimodal llms", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a training-free Control MLLM. The core idea is to adjust the visual token outputs of the MLP during inference to control the attention, ensuring that the text prompt tokens focus on the specified visual regions. The model enhances the intensity of the indicated regions in the attention maps by optimizing learnable latent variables based on an energy function. This enables the model to refer to various visual prompts (including boxes, masks, scribbles, and points) without requiring fine-tuning or additional data. Experiments demonstrate that the proposed model is effective. A reviewer questioned the generalization capability of the proposed approach, to which the authors responded in their rebuttal.
Decision: Accept (poster)