ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models

Authors: mingrui  wu
xinyue  cai
jiayi  ji
jiale  li
oucheng  huang
gen  luo
hao  fei
guannan  jiang
xiaoshuai  sun
rongrong  ji
training  method
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-24 16:58:54

Abstract: In this work, we propose a training-free method to inject visual prompts into Multimodal Large Language Models (MLLMs) through learnable latent variable optimization. We observe that attention, as the core module of MLLMs, connects text prompt tokens and visual tokens, ultimately determining the final results. Our approach involves adjusting visual tokens from the MLP output during inference, controlling the attention response to ensure text prompt tokens attend to visual tokens in referring regions. We optimize a learnable latent variable based on an energy function, enhancing the strength of referring regions in the attention map. This enables detailed region description and reasoning without the need for substantial training costs or model retraining. Our method offers a promising direction for integrating referring abilities into MLLMs, and supports referring with box, mask, scribble and point. The results demonstrate that our method exhibits out-of-domain generalization and interpretability.

Main paper: 
 introduction: 
 In recent times, there has been a surge in the development and adoption of large language models (LLMs), such as GPT-4 [1] and Llama [53] , showcasing remarkable capabilities in addressing a wide range of human-generated questions. The success of these models has sparked interest among researchers in exploring the integration of LLMs with visual inputs. Consequently, a new class of models known as Multimodal Large Language Models (MLLMs) has emerged [36, 33, 17, 67, 81, 38] . However, despite their widespread adoption, traditional MLLMs often face limitations due to their reliance on coarse image-level alignments. This restricts users to guiding MLLMs solely through text prompts for detailed region description and reasoning. However, text often fails to capture the intricate visual nuances present in an image.Addressing this challenge, recent efforts [68, 5, 75, 12, 37] have pioneered the integration of referring abilities within MLLMs, which enables users to provide input by pointing to specific coordinates of the objects or regions, as shown in Figure 1 (left). However, these endeavors typically entail substantial training costs to equip MLLMs with referring capabilities. Additionally, the model must undergo retraining to adapt to new data domains or new base MLLMs.In this work, we propose a training-free method to inject the visual prompts into the Multimodal Large Language Models via learnable latent variable optimization. The method originates from ourWhat color is the hat the person is wearing?Inference on free domain prompt Inference on free domain promptDescribe the text in the region. observation of the attention maps from the MLLM decoder, which model the relationship between the pixels and text prompt tokens and encompass rich semantic relations that significantly influence the generated text. However, MLLMs typically involve fine-tuning an MLP layer to bridge the gap between visual and linguistic representations, which means that the output of the MLP layer can indirectly impact the relationship between text prompt tokens and pixels in the attention layers of the MLLM decoder, thereby altering the model's output.Thus, our key idea is that we can alter the outputs of MLLMs by adjusting the visual tokens from the MLP output during the inference process, controlling which text prompt tokens attend to which visual tokens in the attention layers. Specifically, we augment visual tokens with an additional learnable latent variable. Subsequently, we optimize the learnable latent variable based on an energy function designed to enhance the strength of the referring regions in the attention map between the text tokens and the visual tokens.Our method enables referring MLLMs with various visual prompts, including box, mask, scribble and point, and does not require model training, fine-tuning, or extra data. We also demonstrate that our method exhibits out-of-domain generalization and interpretability. 
related work: 
 MLLMs Motivated by the accomplishments of Large Language Models (LLMs) [1, 53] , there is a burgeoning trend among researchers to develop a diverse range of Multimodal Large Language Models (MLLMs) [33, 36, 17, 67, 32, 38, 39, 18, 22, 78, 15, 58, 35, 20, 21, 19] . These MLLMs typically comprise a visual encoder, a language decoder, and an image-text alignment module. The visual encoder and the language decoder are often sourced from pre-trained models, such as CLIP [44] , DINOv2 [41] , Llama [53] , and Vicuna [16] . Meanwhile, the image-text alignment module is trained on image-text pairs and fine-tuned through visual instruction tuning to enhance its visual conversation capabilities. These Multimodal Large Language Models (MLLMs) often confront limitations stemming from their reliance on coarse image-level alignments.Referring MLLMs In recent research, there has been a noticeable trend towards integrating foundation models with tasks involving referring dialogue. These models [69, 74, 75, 12, 37, 43, 68, 64, 71, 5, 73, 40, 26, 34, 70, 79, 11, 51, 46, 80, 63, 8, 24, 45, 52, 72] introduce spatial visual prompts as extra input and are trained using region-text pairs. By leveraging this approach, they effectively bridge the gap between textual prompts and visual context, enabling comprehensive understanding of image content at the regional level. However, these methods inevitably require a substantial training burden.Training-free Control in Text-to-Image There are numerous works on controllable text-to-image generation, among which training-free methods [27, 14, 62, 30] are most relevant to our research. Among them, Prompt-To-Prompt [27] explore the role of attention in text-visual interactions in Stable Diffusion [47] model, while Layout-Guidance [14] indirectly bias attention in Stable Diffusion model by optimizing an energy function. These contributions significantly inform our investigation into enhancing controllability and interpretability in MLLMs.Visual Prompt The visual prompt can be categorized into two main techniques: hard prompt and soft prompt. The hard visual prompt works [48, 57, 66, 65] that direct the model's attention to the region or enable visual grounding abilities in the Multimodal Models in a training-free and convenient manner by directly manipulating images, such as color guidance [60, 23] . However, these methods inevitably compromise the structural information of the images, or a strong understanding of the corresponding patterns by the model is required. In contrast, the soft visual prompt works [28, 4, 77] integrate learnable visual prompts into models to adapt them for different downstream tasks. However, these methods do not support region guidance and require fine-tuning the model with downstream data. In contrast, we optimize a learnable latent variable to support referring MLLM in the test time, without any downstream training data required, and TPT [49] is most related to our work.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: The paper discuss the limitations of the work in Sec 6.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
background: 
 Multimodal Large Language Models (MLLMs): The MLLMs typically consist of a visual encoder, an LLM decoder, and an image-text alignment module. Given an image I, the frozen vision encoder and a subsequent learnable MLP are used to encode I into a set of visual tokens e v . These visual tokens e v are then concatenated with text tokens e t encoded from text prompt p t , forming the input for the frozen LLM. The LLM decodes the output tokens y sequentially, which can be formulated as:EQUATIONConsidering LLaVA-liked [36] MLLMs, the LLM in MLLMs typically employs a transformer model [54] with the attention layer as its core. In such model, the attention maps represent the relationships between the visual tokens and the text prompt tokens. The attention map in attention layer τ , computed on the transformed visual-text concatenated embeddings [e v , e t ] (τ ) , is obtained as follows:EQUATIONwhere d k is a scaling factor. A (τ ) consists of A (τ ) ij with i, j ∈ {1, • • • , n}, representing the relationship between the i-th token and the j-th token, and their impact on the output.Training Referring MLLMs: The objective of training referring MLLMs is to inject the visual prompt r into the MLLMs to achieve referring ability via model parameter learning. The visual prompt r can take various forms, such as a box, mask, scribble, or point, to indicate specific locations or regions within the image.Current referring MLLMs typically need to be fine-tuned on a training set with positional annotations before they can be effectively used. The fine-tuning process involves maximizing the log likelihood of generating the text conditioned on I, p t , and r over the entire training dataset. This can be formulated as:EQUATIONwhere θ represents the parameters of the model f , and U is the number of samples in the training set. This method significantly enhances the model's fine-grained understanding and interactivity. However, it incurs high training costs. Additionally, such fine-tuning strategies result in domainspecific behaviors, which have been shown to compromise the out-of-distribution generalization and robustness of MLLMs [49] . Therefore, when domain shifts occur, the model needs to be retrained, leading to a lack of flexibility. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Answer: [No] Justification: The code will be released in the Github.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
method: 
 We aim to propose a training-free method to overcome the inconveniences of traditional training.Training-free referring MLLM maintains the model parameters θ frozen, eliminating the need for any training or fine-tuning with samples from the training set. During inference, the only information available is the single test sample without label information, as shown in Figure 1 (right).In this section, we explore and design a solution to address the challenges of Training-free Referring MLLMs. The key task is to flexibly embed visual prompts during the inference phase while maintaining the model's reasoning capabilities. To begin with, we delve into the mechanism of MLLMs (see Sec. 4.1), our key observation is the attention mechanism in LLM capturing the relationship between the model's output and the input pixels. Further, the visual tokens inputted into the LLM influence the values of the attention maps to indirectly control the model output. Building on this analysis, we propose the Latent Variable learning (a test-time prompt tuning strategy [49] , see Sec. 4.2) to edit the visual tokens, as shown in Figure 4 . This method effectively integrates visual prompts into pre-trained MLLMs, enabling fine-grained visual reasoning.We begin by analyzing which factors in the model truly capture the relationship between input and output? In other words, we seek to understand how to interpret the association between the model's output and the input pixels.As demonstrated by Equation 1, Multimodal Large Language Models (MLLMs) fundamentally model the maximum likelihood output based on visual input and text prompts. By conditioning on the text prompt, the model can determine which parts of the image have the greatest impact on the output. Building on the discussions in the Sec. 3 and illustrations in the Figure 2 (top line), we can observe that the attention map models the influence of visual tokens on the output conditioned by the text prompt. Therefore, the attention map in MLLMs not only provides interpretability regarding the relationship between model output and input pixels but also facilitates guiding the model's output.A natural idea is that we can directly alter the model's output by editing the attention maps. Inspired by IBD [82] , we achieve this by adding an adjustment coefficient η to the attention related to the visual tokens corresponding to the referring region, which can be formulated as, where M is a mask with the same shape as the attention map, r denotes the referring region. However, we have to carefully select a suitable coefficient η for each example. When the η is too small, it leads to ineffective control (as shown in Figure 3 a ), and when it is too large, it can impact the language capabilities of the LLM (as shown in Figure 3 c ). Additionally, we found that it is sufficient to manipulate the attention map at the 0-th step during model inference (as shown in Figure 3 a,b,c), as it is most directly associated with the text prompt, and manipulating attentions step by step also affects the expression of the LLM (as shown in Figure 3 d ). Overall, directly manipulating attention maps is not a viable approach because it overlooks the relationships between attention layers and not all layers' visual tokens decide the output [13] .EQUATIONWe note that in the most MLLMs, typically the MLP layer is trained for image-text alignment. This implies that MLLMs indirectly affect the values of the attention map by learning the parameters of the MLP layer to alter the visual tokens. In other words, the visual tokens inputted into the LLM directly influence the values of the attention maps.It is also worth noting that the input text prompt also directly influences the model's output, particularly regarding non-visual-related [76] output content. However, we aim to explain the correlation between the output and the input image. Therefore, we do not consider the direct impact of the text prompt on the output in our analysis.Based on the analysis above, our core idea is to indirectly influence the attention maps by editing visual tokens, thereby focusing on the referred regions. We achieve this by optimizing a learnable latent variable based on an energy function [14, 59] , which calculates the relationship between the input referring and the attention maps. To do this, we first need to determine which attention maps to use. One approach is to use attention maps between each text prompt token and all visual tokens. However, because visual tokens typically have a significant impact on the result based on only a few most relevant text prompts (referred to as highlight text tokens), using all attention maps would be computationally redundant. Yet, for users, identifying the highlight text tokens can be challenging. Therefore, we simply average pool the attention maps generated for each text prompt token to represent the global context of the text prompt (referred to as the context token) and its association with visual tokens. We found that this simple method of using context tokens produces attention maps similar to those generated by highlight text tokens, as shown in Figure 2 (bottom line). We leave the optimization based on highlight text tokens for future work.Specifically, our method supports four types of referring shapes, including box, mask, scribble, and point. We employ two types of energy functions to respectively support these referring shapes: a hard Hard Mask-based Energy Function We first zero initialize a learnable latent variable p v with the same shape as e v , and add it to the e v . Then we can get N attention maps from N attention layers which model the relation between the context token and the novel visual tokens. Given the referring box or mask, we first convert it into a binary mask. Then, we compute the mask-based energy function based on the mask and the attention map A (ct) , which is obtained by averaging pooling from N attention maps. The energy function can be formulated as:EQUATIONwhere r denotes the referring region. Then the gradient of the loss 5 is computed via backpropagation to update the learnable latent variable:EQUATIONwhere α > 0 is a hyperparameter controlling the strength of the guidance. By optimizing p v through the Equation 6, we indirectly guide the attention maps to produce higher responses in the referring region r, thereby increasing the influence of the visual content of region r on the output.Soft Mask-based Energy Function Since scribble and point lack the concept of the region, it is optional to use an extra SAM [31] model to obtain a mask for applying the Hard Mask-based Energy Function. However, this incurs additional inference cost, so we also provide an optional soft mask-based energy function based on a distance matrix D, which is computed via applying the OpenCV [7] distanceTransform function on the given scribble or point. Then the soft mask-based energy function can be formulated as:EQUATION)where σ is the standard deviation of the Gaussian function, which is set to 0.1. By optimizing p v through the Equation 7, the closer the region of attention map is to the given scribble or point, the higher the response.LLaVA: The object is a boat.The object is a life ring, which is a type of personal flot.Prompt: What is the object?LLaVA + Ours: The object is a desk with a chair, a plant, and various decorative items on it.LLaVA: The object is a chair.Prompt: What is the object?Prompt: What is unusual of the cat? LLaVA: The unusual aspect of the cat is that it is jumping in the air while holding another cat in its mouth.LLaVA + Ours: The unusual aspect of the cat is that it has a paw on its face, which is not a common feature.The unusual aspect of the object in the image is that it is a turtle with a shell on its back, and it is wearing a hat. This is not a typical sight, as turtles do not usually wear hats or have shells on their backs.Prompt: What is unusual of the object?LLaVA + Ours: The unusual aspect of the object is that it is a snail shell, which is not a typical object to be used as a hat or a decoration. The snail shell is placed on the head of a frog, which is an unconventional combination of animals. This creates an interesting and unexpected scene, as it is not common to see a snail shell being used in such a manner.  
experiment details: 
 Unless explicitly stated otherwise, the MLLM we use is LLaVA-v1.5-7B [35] , T =5, α=400 and β = 0.5. All experiments are conducted on two RTX 3090 GPUs with 24 GB of memory each.Referring with Different Visual Prompts. We first demonstrate referring QA with different visual prompts, including box, mask, scribble and point in the Figure 5 . Our method consistently demonstrates significant controllability with four types of visual prompts. And our method improves the interpretability compared to basic model (column 3 vs column 2), demonstrates a stronger correlation between the attention response areas and the generated descriptions.Out-of-Domain Task. We present examples of the performance on out-of-domain tasks OCR and Screenshots. As shown in Figure 6 , compared to Ferret, our method correctly identified the text in the referring region. Additionally, as shown in Figure 9 , our method correctly recognized the app in the mobile screenshot, unlike Ferret. The text appears to be a sale advertisement, possibly for a car. Comparison on Referring Object Classification Task. Following Ferret [68, 74] , we use the Referring Object Classification (ROC) task to evaluate whether our method can accurately pinpoint and understand the semantic of the referring region. The task requires the model to correctly identify the target within the referring region. We follow the setting of Ferret to form 1,748 questions (in which 1,548 for test and 200 for validation) based on LVIS [25] validation dataset, with corresponding box, mask, scribble and point. We consider the edit attention with η = 10 (as Equation 4and Figure 3 (b) ) as the baseline model. And we compare several training methods [43, 75, 12, 68] . We also evaluate the lower and upper limits of LLaVA's recognition capability by assessing LLaVA without referring region, as well as background blur outside the referring region, which are presented in gray. Additionally, we evaluate a method that highlights regions with color as a comparable training-free method. More details and the input examples are shown in Appendix B.2.The results are shown in Table 1 . Our method shows a better performance than the training method GPT4-ROI with box referring (60.59 vs 58.59) and the Shikra-7B with point referring (58.85 vs 56.27). However, due to the limitations of LLaVA's capabilities (as shown in the results of the LLaVA+Blur), we present a performance gap compared to the latest training method Ferret [68] . Our method also demonstrates superiority compared to training-free color prompt-based method and baseline method.Comparison on Referring Text Classification Task. We consider the Referring Text Classification (RTC) task as the out-of-domain task, to verify the model's out-of-domain transfer capability. Similar to the ROC task, we formulate the problem as a binary classification task and construct 1,372 questions based on the COCO-Text [56] dataset. Since point and scribble referring methods are not Training Methods:Kosmos-2 [43] 16.55 -GPT4-ROI [75] 54.23 -Shikra-7B [12] 50.07 -Ferret-7B [68] 55. 47 suitable for text due to the non-connectivity of the text, we only evaluate the RTC task with box and mask referring.The results are shown in Table 2 . All the training methods we evaluated exhibited poor out-of-domain generalization performance. Specifically, Ferret achieves only 55.47% accuracy on the RTC task, despite its excellent in-domain performance as shown in Table 1 . In contrast, our training-free method still demonstrates the best out-of-domain generalization performance. We also present comparative examples of out-of-domain tasks, as shown in Figure 6 and Figure 9 .More Tasks and MLLMs. We also validate our method through the Referring Description Task on LLaVA-1.5-7B and InstructBLIP-7B [17] . The results are shown in Table 3 . Our method consistently improves the model's referring description performance. And we validate our method on the more MLLMs through the ROC and RTC Tasks, MLLMs including InstructBLIP-7B and LLaVA-HR-7B [39] , more details are shown in Appendix B.3. The results are shown in Table 4 , our method consistently improves performance across different MLLMs. Due to InstructBLIP's relatively poor text recognition capabilities, our method results in only a modest improvement in the RTC task. However, thanks to the beneficial effect of image resolution on the RTC task, our method achieves a relative improvement of approximately 11.59% on LLaVA-HR.The ablation studies primarily focus on the box referred object classification. Furthermore, inspired by DIFNet [61] , we calculate a relevancy between the model's output and pixels within the referring region to assess the extent to which the model's output is influenced by visual content within the region. More details and additional experiments are shown in Appendix B.2 and B.3.Impact of T and α. As shown in Table 5 , as T increases, the relevancy between the model's output and the referring regions also increases. However, the larger T results in a decrease in the model's accuracy on the ROC task, also with excessively large relevancy scores, showing that excessively large relevancy scores also indicate overfitting of the learnable latent variable. Therefore, the value of the relevancy score provides us with guidance to alleviate model overfitting, particularly when the relevancy score is around 0.18, typically resulting in better performance. And the value of α affects the convergence speed of optimization, with larger α also leading to overfitting of the model.As shown in Table 5 , when equipped with a smaller β value, it effectively mitigates the overfitting issue associated with the learnable latent variable. For instance, with α = 400 and β = 0.3, the model's performance improves from 53.5 to 62.5. However, a smaller value of β also results in slower convergence of the learnable latent variable. Therefore, we combine the Early Stop strategy, allowing us to use a slightly larger β to accelerate the convergence of the learnable latent variable. After incorporating the early stop strategy, we can opt for slightly larger T to ensure that the model is adequately optimized on challenging samples. The early stop strategy allows us to attain superior model performance while reducing the impact of overfitting. The additional experiment about ES is shown in Table 6 .Impact of Different Text Prompts and the Size of Visual Prompts. We explore the impact of different text prompts and the size of visual prompts in Figure 10 and Figure 11 respectively. The results demonstrate that combining a clear and specific text prompt with an appropriate visual prompt size typically leads to improved controllability. 
limitations: 
 While we have demonstrated visual prompt control by optimizing only visual tokens, our technique is subject to a few limitations. First, there is some additional inference overhead, while various engineering approaches (such as Ollama2 ) can significantly speed up the process. Therefore, this limitation can be reasonably overlooked. Second, our method is applicable only to white-box models and relies on the basic capabilities of the models themselves. However, our approach is orthogonal to these ongoing advancements in foundational models. Third, currently, our method supports only a single region visual prompt, extending this to multi-region control is a direction for future work. Fourth, our current optimization strategy is relatively simple, and the selection of text prompts can also affect the optimization results. We plan to focus on improving this aspect in future research.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: The paper specify all the experimental setting details in Sec 5.1.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 In this work, we present a training-free method to integrate visual prompts into Multimodal Large Language Models (MLLMs) through learnable latent variable optimization. By adjusting visual tokens during inference, our approach enhances the attention to referring regions, enabling detailed descriptions and reasoning without additional training costs. Our method supports various referring formats such as box, mask, scribble, and point. The results show that our approach demonstrates strong out-of-domain generalization and interpretability, making it a promising direction for embedding referring abilities into MLLMs.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The error bars are not reported because it would be too computationally expensive.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: The paper provide sufficient information on the computer resources needed to reproduce the experiments in Sec 5.1.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: The paper discuss both potential societal impacts of the work in Sec. A.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the original paper that produced the code package or dataset.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: The integration of Multimodal Large Language Models (MLLMs) has far-reaching implications across various sectors. These models enhance accessibility, improve education, advance healthcare, and revolutionize media and entertainment. They offer intuitive interfaces for diverse communication needs, assist in medical diagnosis and treatment, and enable immersive multimedia experiences. However, ethical considerations must be addressed to ensure equitable and responsible deployment. Collaboration among researchers, policymakers, and industry is essential to maximize the societal impact of MLLMs.B Appendix / supplemental material θ (t) EMA = β • θ (t) + (1 -β) • θ (t-1) EMA ,where θ(t)EMA is the EMA of the model weights at time t, θ (t) is the model weights at time t, θ(t-1)EMA is the EMA of the model weights at the previous time step, β is the smoothing factor, ranging from 0 to 1. EMA helps to stabilize the training process by reducing the variance of the parameter updates, which can prevent the model from diverging or oscillating during training. We employ the EMA on the learnable latent variable in our experiments during visual token optimization.learning models to prevent overfitting. The main idea behind ES is to monitor the performance of the model on a validation set during training and stop the training process if the performance begins to deteriorate. Specifically, in our experiments, ES tracks the loss on the test sample and halts the visual token optimization process if the metric does not improve for a certain number of consecutive epochs. Mathematically, ES can be described as follows:Let L (i) denote the loss at optimization process t, and let ∆ represent a predefined threshold for acceptable loss degradation, then we Stop optimizing if abs(L (t) -L (0) )/L (0) ≥ ∆ for T consecutive process.We set ∆ = 0.25 in our experiments. ES helps prevent overfitting by stopping the optimization process before the model starts to lose generalization ability. It provides a simple yet effective way to regularize the optimization process and improve the generalization performance of the model.Details of Relevancy Score. To elucidate the influence of input visual pixels on outputs, we employ a technique known as the Relevancy Map. This map highlights the regions or features of the input data that contribute most significantly to the model's decision-making process. Specifically, the Relevancy Map assigns importance scores to different parts of the input, indicating their relative impact on the model's output. This interpretability tool not only enhances our understanding of the model's behavior but also facilitates error analysis and model debugging. More details can be found in the paper [61, 3, 9, 10, 50] .In our experiments, we provide the relevancy scores alongside model predictions to provide insights into the model's decision-making process. In MLLMs, the typical use of Key-Value cache technique in LLMs, for convenience, we propagate the relevance from the model's first output token to the input of LLM ( e v and e t ). Then the relevancy scores corresponding to visual tokens are reshaped into a grid that matches the layout of the original image. Since relevancy maps are akin to attention maps and often exhibit significantly higher values in localized regions, taking the average may dilute their significance. So we extract the max value in the referring region of relevancy map as final relevancy score.It is also worth noting that we found relevancy map plays a similar role to attention map, directly modeling the relationship between input and output of the model. This suggests that we may be able to directly control the model's output based on relevancy map. However, calculating the relevancy map requires computing the gradient for each relevant tensor in the model. For convenience, the approach presented in this paper only utilizes attention for implementation, while the relevancy map only be used to assess the extent of visual impact on the output. Details of Implement. We apply low-bit quantization to the basic model we implemented to further optimize memory usage. Nevertheless, we still achieve performance competitive with training-based methods (without quantization).Examples of ROC Task. The box and mask are from the LVIS GT boxes and mask, the scribble and the point are randomly sampled inside the boxes. It is worth noting that we follow Ferret to choose negative object whose central point is close to the GT object. Although this is somewhat disadvantageous for us, we still achieve competitive performance compared to other methods. And we show the input examples of different methods on ROC task in Figure 7 .Application on scene text recognition (OCR). Results are shown in Figure 8 . Our method can also perform referring region text recognition.Examples on Out-of-Domain Task. We present examples of the performance on out-of-domain tasks OCR and Screenshots. As shown in Figure 6 , compared to Ferret, our method correctly identified the text in the referring region. Additionally, as shown in Figure 9 , our method correctly recognized the app in the mobile screenshot, unlike Ferret.Effect on More MLLMs. We validate our method on various MLLMs, including LLaVA-1.5-7B, InstructBLIP-7B, and LLaVA-HR-7B. InstructBLIP employs a cross-attention image-text alignment module called Q-Former. Specifically, in InstructBLIP, the interaction between visual tokens and text tokens occurs within Q-Former, so we utilize the cross-attention map from Q-Former to optimize 4 .Comparison on Referring Description Task. We also validate our method through the Referring Description Task. Specifically, we construct the test set based on region-text pairs from the Ref-COCOg test split and evaluate the method using traditional captioning metrics BLEU@4 (B@4) [42] , METEOR (M) [6] , CIDEr-D (C) [55] , and SPICE (S) [2] . However, it is important to note that these metrics are significantly influenced by the style and distribution of the model's output text. Typically, outputs that are more similar in distribution to RefCOCOg yield better results, while those with some unique styles lead to poorer results. Therefore, this experiment is only used for internal validation of the model and not for comparison between different models. The results are shown in Table 3 . Our method consistently improves the model's referring description performance.Impact of LLM Size on Different MLLMs. We validate the impact of LLM size on LLaVA-1.5 and InstructBLIP, focusing on the 7B and 13B models. The results are shown in Table 7 . Our method exhibited poorer performance in LLaVA-1.5-13B, which may be due to the increased number of attention maps, making the optimization of visual tokens more challenging. Therefore, it may be essential to adopt different hyperparameters for different models. In contrast, in InstructBLIP-7B and InstructBLIP-13B, our method consistently yielded performance improvements. This is likely because the interaction between visual tokens and text tokens occurs in Q-Former for InstructBLIP, thereby mitigating the optimization challenges associated with larger LLMs.Inference Cost. We compare the inference cost of our method and the basic LLaVA model. LLaVA + Ours model with T = 5 and does not use Early Stop here. Results are shown in Table 8 , when outputting only 7 tokens, our method noticeably adds approximately 2 seconds of inference time. However, when generating about 400 tokens, the proportion of the extra inference time produced by our method significantly decreases. When combined with an early stop strategy, the proportion of additional inference time will be further reduced.Impact of Different Text Prompt. Results are shown in Figure 10 . Different text prompts can significantly affect the performance of our method. For instance, our method fails when using an ambiguous text prompt like "describe the region in the image." However, it succeeds with a more specific referring text prompt such as "what is unusual about the region of the building?" Therefore, it is recommended to use clear and specific text prompts to achieve better control and performance.Impact of the Size of Visual Prompt. Results are shown in Figure 11 . For box and mask prompts, when they do not fully cover the referring object, failure control may occur. This is because the highest attention response for the desired outcome may fall on any unexpected area of the object. Therefore, it is recommended to cover the object with a larger-sized visual prompt.Comparing Highlight Token and Context Token based Optimization. We compare Highlight Token and Context Token based Optimization as shown in Figure 12 . Directly optimizing based on the highlight token is faster but also prone to overfitting. This may be due to the direct connection between the highlight token and visual token, while other text tokens contain redundant visual associations. However, this also ignores the potential influence of other text tokens.More Examples of Referring MLLMs with Scribble and Point. In Figure 14 , we also show more examples of referring MLLMs with scribble (right) and point (left). The app in the picture is used for tracking and managing time, specifically for keeping track of time spent on various tasks. It is designed to help users stay organized and productive by providing a visual representation of their time usage. This app can be particularly useful for individuals who want to manage their time effectively, such as students, professionals, or anyone who wants to optimize their daily activities.The app in the picture is a mobile phone app, which is designed to provide various functionalities to the user. It features a collection of icons representing different apps and services, such as Google, Calendar, Camera, and others. These icons are organized in a grid-like pattern, making it easy for users to access and navigate through the different functionalities offered by the app. The app is likely used for communication, organization, and various other tasks that can be performed on a smartphone.Prompt: What is this app in the picture used for?The app seems to be a communication app, as suggested by the icon displaying a person on it. It's probably used for messaging or staying connected with others.The app in the picture is a game launcher, which is used to quickly access and launch various games on a smartphone. It provides a convenient way to manage and organize the games installed on the device, making it easier for users to find and play their favorite games.The app in the picture is a mobile phone app, which is designed to provide various functionalities to the user. It features a collection of icons representing different apps and services, such as Google, Calendar, Camera, and others. These icons are organized in a grid-like pattern, making it easy for users to access and navigate through the different functionalities offered by the app. The app is likely used for communication, organization, and various other tasks that can be performed on a smartphone.Prompt: What is this app in the picture used for?The app is a map application, typically used for navigation purposes. It's an icon on the home screen, offering easy access to directions and location services The unusual aspect of the building's region is the presence of a large window in the middle of the structure. This is not a common feature in most buildings, as windows are typically placed on the sides or corners of a building. The large window in the middle of the building adds an interesting architectural element to the structure.The unusual aspect of the building's region is the presence of a potted plant on the side of the building. It is not common to see a potted plant on the side of a building, especially in a setting where people are dressed in formal attire, such as a wedding. The potted plant adds a unique touch to the scene and stands out among the other elements.Prompt: what is unusual of the region of the building?LLaVA LLaVA + Ours Prompt: describe the region in the image.The image features a group of people gathered outside a building, possibly a church. There are four people in the scene, with two of them wearing ties, indicating a formal or semiformal occasion. The group consists of a bride, a groom, and two other people. They are all standing together, possibly posing for a picture. The building has a black and white color scheme, adding a touch of elegance to the scene.The image features a group of people gathered outside a building, possibly a church or a wedding venue. There are four people in the scene, with two of them wearing ties, suggesting a formal occasion. The group is standing on a brick walkway, and there are potted plants nearby, adding to the ambiance of the location. The people are engaged in conversation, creating a lively atmosphere. The unusual aspect of the building's region is the presence of a large window in the middle of the structure. This is not a common feature in most buildings, as windows are typically placed on the sides or corners of a building. The large window in the middle of the building adds an interesting architectural element to the structure.The unusual aspect of the building's region is the presence of a potted plant on the side of the building. It is not common to see a potted plant on the side of a building, especially in a setting where people are dressed in formal attire, such as a wedding. The potted plant adds a unique touch to the scene and stands out among the other elements.Prompt: what is unusual of the region of the building?The unusual feature of the region of the building is the presence of a stained glass window. Stained glass windows are not commonly found in modern buildings, especially in urban settings. This adds a unique and artistic touch to the building, making it stand out from other structures in the area.The unusual aspect of the region of the building is the presence of a potted plant on the ground. Typically, potted plants are placed on elevated surfaces, such as tables or shelves, to provide better air circulation and prevent overwatering. However, in this image, the potted plant is located on the ground, which is not a common practice. This might be due to the specific needs of the plant or the design choice of the building's interior. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.https://ollama.com/ 


Decision: This paper proposes a training-free Control MLLM. The core idea is to adjust the visual token outputs of the MLP during inference to control the attention, ensuring that the text prompt tokens focus on the specified visual regions. The model enhances the intensity of the indicated regions in the attention maps by optimizing learnable latent variables based on an energy function. This enables the model to refer to various visual prompts (including boxes, masks, scribbles, and points) without requiring fine-tuning or additional data. Experiments demonstrate that the proposed model is effective. A reviewer questioned the generalization capability of the proposed approach, to which the authors responded in their rebuttal.

Reviews: review: paper_summary: paper_summary: This paper proposes a training-free ControlMLLM, which uses optimizable latent variables to inject visual prompts into multimodal large MLLMs. The core idea is to adjust the visual token outputs of the MLP during inference, to control the attention response and ensure that the text prompt tokens focus on the indicated visual regions. It enhances the intensity of the indicated regions in the attention maps by optimizing learnable latent variables based on an energy function, enabling reference to various visual prompts (including boxes, masks, scribbles, and points) without model training, fine-tuning, or additional data. The method demonstrates out-of-domain generalization and interpretability, providing a promising direction for integrating referential capabilities into MLLMs. Experiments show that the proposed model is effective.
strength_weakness: strength_weakness: 1), The visual prompt shows great improvements over the base model LLaVA, however, it requires additional guidance information and more inference time, which may limit the applications of ControlMLLM. Especially in some complex scenarios where the guidance signals are unavailable.

2), In addition, the region signal $r$ plays a core role during optimization, and it controls the output of MLLMs. What if $r$ itself is wrong?  It may mislead the MLLMs.
questions: questions: See above
limitations: limitations: None
paper_summary: paper_summary: This paper introduces a training-free approach to integrate visual prompts into MLLMs using learnable latent variables, aiming to enhance the model's interpretability and generalization. It adjusts visual tokens from MLP outputs and optimizes latent variables with an energy function to improve attention on relevant visual regions.
strength_weakness: strength_weakness: 1. Figures 3(a) and 3(b) show visualization results for different values of η. The paper mentions that in 3(a), η is too small to effectively control the attention. However, the focus of attention map in 3(b) does not significantly differ from that in 3(a).
2. Table 3 shows that the highest accuracy is achieved when α = 400 and T = 3. Why, then, is the value of T ultimately chosen as 4?
3. Since there are no recent developments, contributions, or updates to the MLLM, its detailed presentation in Eq.s (1)-(3) might be unnecessary and could be omitted. Besides, some symbols are not defined, such as, $I_i$, $A_i^{(ct)}$.
questions: questions: 1. It is recommended to validate the effectiveness of the method on additional MLLMs.
2. Additionally, please note that the title listed on the paper submission does not match the title in the PDF.
limitations: limitations: The limitations have been discussed.
paper_summary: paper_summary: The paper introduces a training-free approach to improve the referring capabilities of multimodal large language models (MLLM). In particular, the authors iteratively adjust the attention maps using a learnable latent variable, which is based on energy functions. They empirically validate the efficacy of their method on referring classification tasks, leveraging the foundation of LLaVA.
strength_weakness: strength_weakness: 1. The generalization ability of the proposed method has not been fully verified. (a) It's uncertain whether the method can be applied to MLLMs beyond LLaVA. As the foundational model strengthens, the method's effectiveness could potentially diminish. (b) The energy function, based on a soft mask, heavily relies on the quality of segmentation models. (c) As seen in Table 3, the results appear to be sensitive to hyperparameters. This raises the question: do we need to meticulously adjust the parameters for each model and sample? If that's the case, the practicality of this method could be questionable.

2. There is a naive baseline that requires discussion: directly cropping the referred region and feeding it into the LLM.

3. In Tab. 2&3, the tasks and compared models are indeed limited. It is recommended to discuss more state-of-the-art MLLMs on wider tasks and datasets.
questions: questions: Please see the Weaknesses.
limitations: limitations: The authors have discussed it in Sec. 6.
paper_summary: paper_summary: This paper proposes a method to integrate visual prompts into MLLMs without requiring additional training. The key idea is to optimize a learnable latent variable to enhance the attention response to visual tokens during inference, thereby improving the model's ability to focus on specific regions in the visual input.
strength_weakness: strength_weakness: 1. Very poor writing. The use of abbreviations and full terms is very inconsistent. For example, "Multimodal Large Language Models" is spelled out in full on Line 29-30, while the abbreviation "MLLMs" is used frequently earlier in the text. Additionally, "Linguistic" in Line 34 should be "textual." There are many grammatical errors in Figure 's prompt: "What's color of hat the person wearing?" and output: "The person wearing the hat is wearing a green hat.". Even the title in section 4.2 contains grammatical errors.
2. This paper lacks novelty and is an incremental form of previously proposed methods with no innovative points.
3. In Line 3, the authors claim that attention connects visual tokens and textual tokens, but in Line 33, it changes to MLP.
4. The experimental results are insufficient and lack numerous baselines, such as LLaVA1.5[1], LLaVA-NeXT[2], Monkey[3], and Qwen-VL[4].
5. The motivation for this study is insufficient. I think there is a baseline that by making the prompt descriptions clearer and more comprehensive based on the original MLLM. This baseline can also leverage MLLM's inherent ability to focus on specific regions.

[1] Improved Baselines with Visual Instruction Tuning.

[2] LLaVA-NeXT: Improved reasoning, OCR, and world knowledge.

[3] Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models.

[4] Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
questions: questions: See weakness.
limitations: limitations: See weakness.

Comments: comment: Dear Reviewer,

Thank you for taking the time to review our paper and offer your valuable suggestions. We have thoroughly addressed your concerns in the rebuttal. We kindly ask if you would consider raising the score for our paper.
Dear Reviewer,

Thank you for taking the time out of your busy schedule to review our paper and provide valuable feedback. We have addressed your concerns in the rebuttal. We kindly ask if you would consider raising the score for our paper.
Dear Reviewer,

Thank you for carefully reading our paper and rebuttal, and for recognizing our work. We wish you the best of luck.
Thanks to the authors' response. Although some of my concerns have been addressed, I believe there is still significant room for improvement and enhancement in this work. Therefore, I will maintain my score.
Hello Reviewer,

The author has submitted a response to your comments. Whether or not it addresses your concerns, it would be greatly appreciated if you could acknowledge that you have reviewed the reply.
Hello Reviewer,

The author has submitted a response to your comments. Whether or not it addresses your concerns, it would be greatly appreciated if you could acknowledge that you have reviewed the reply.
Hello Reviewer,

The author has submitted a response to your comments. Whether or not it addresses your concerns, it would be greatly appreciated if you could acknowledge that you have reviewed the reply.
Thanks to the authors for their response, and I read the other comments as well. This paper makes an interesting idea of training-free prompt tuning for MLLM. As a result, I have decided to raise my rating.