Instruction-Guided Visual Masking

Authors: jinliang  zheng
jianxiong  li
sijie  cheng
yinan  zheng
jiaming  li
jihao  liu
yu  liu
jingjing  liu
xianyuan  zhan
tsinghua  air
  university
sensetime  research
  mmlab
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 12:12:35

Abstract: Instruction following is crucial in contemporary LLM. However, when extended to multimodal setting, it often suffers from misalignment between specific textual instruction and targeted local region of an image. To achieve more accurate and nuanced multimodal instruction following, we introduce Instruction-guided Visual Masking (IVM), a new versatile visual grounding model that is compatible with diverse multimodal models, such as LMM and robot model. By constructing visual masks for instruction-irrelevant regions, IVM-enhanced multimodal models can effectively focus on task-relevant image regions to better align with complex instructions. Specifically, we design a visual masking data generation pipeline and create an IVM-Mix-1M dataset with 1 million image-instruction pairs. We further introduce a new learning technique, Discriminator Weighted Supervised Learning (DWSL) for preferential IVM training that prioritizes high-quality data samples. Experimental results on generic multimodal tasks such as VQA and embodied robotic control demonstrate the versatility of IVM, which as a plug-and-play tool, significantly boosts the performance of diverse multimodal models, yielding new state-of-the-art results across challenging multimodal benchmarks. Code, model and data are available at https://github.com/2toinf/IVM.

Main paper: 
 introduction: 
 Multimodal instruction following is a fundamental multimodal task, powering a wide-range of applications such as visual question answering (VQA) [18] , visual captioning [1, 41] , and embodied robotic control [14] . To effectively solve this task, one critical capability required is nuanced imagelanguage grounding, which current multimodal models grow implicitly and slowly through dataintensive end-to-end training without explicit grounding supervisions. Two challenges emerge in this indirect learning of image-instruction alignment: 1) How to accurately localize targeted image regions that corresponds to a specific textual instruction, as illustrated in Figure 1 . 2) How to generalize to diverse visual representations (e.g., same object with different colors, compositions, or backgrounds) that reflect similar textual instruction (e.g., Q3 in Figure 1 ). Lacking an effective and direct solution to these challenges, the most advanced Large Multimodal Models (LMMs) [1, 6, 41, 14] still suffer from hallucinations even when trained with high-quality data in the magnitude of billions [34] .We introduce Instruction-guided Visual Masking (IVM), a versatile plug-and-play model designed to enhance multimodal instruction following via nuanced surgical visual grounding. To eliminate the distraction of instruction-irrelevant visual regions, IVM automatically masks out these regions to sharpen the focus of instruction following, and meticulously crops visual input to tailor for a specific [31] . Traditional methods such as semantic segmentation [68] and referring expression comprehension [64] are limited to fixed categories or fixed instruction formats, thus inapplicable to complex instruction following tasks. RS has reasoning ability, but only allows single object localization. IVM, instead, is universally applicable to any instruction. instruction and enforce multimodal models to zoom in on task-related visual content. Existing visual grounding methods are limited either to predefined object categories, which cannot cover diverse instruction-related visual content; or they subscribe to a fixed instruction format, which restricts the expressiveness of instructions. As shown in Figure 2 , such simplistic grounding techniques often fail to comprehend complex instruction-following tasks.Learning an IVM model requires pixel-level, fine-grained, instruction-guided mask annotations that provide explicit grounding supervisions. To create such a dataset, we build a LLM-empowered Mixture of Expert pipeline with SOTA visual grounding models [52, 50, 31, 20] to efficiently create abundant reliable labels. To compensate the noises in auto-generated labels, we further manually label a smaller dataset with clean annotations, and integrate the two into an IVM-Mix-1M dataset that contains 1 million image-instruction pairs.To reduce demand on costly human labels and ensure optimized utility of machine-generated labels, we employ a Discriminator-Weighted Supervised Learning (DWSL) framework for IVM training, inspired by recent advances in offline imitation learning [60] . Specifically, we introduce a discriminator to assign weights to masks, where high values are assigned to high-quality annotations and vice versa. Thus, these weights generated by the discriminator can naturally act as a weighting function for the IVM training objective, allowing for a preferential training process that prioritizes learning from reliable samples and discards misleading ones.Extensive experiments demonstrate great versatility of the IVM model when integrated into existing multimodal chatbots (commercial and open-sourced) without fine-tuning. Our IVM-enhanced LMMs gain significant performance improvement across new challenging benchmarks such as V*Bench [58] , EgoThink [10] and POPE [34] , achieving new state of the art. IVM model also proves valuable in vision-language robotic manipulation tasks, where data collection is notoriously challenging and generalization is a major concern [35] . With the integration of IVM, our enhanced robot model exhibits boosted performance and better generalization capabilities.Our contributions are summarized as follows: 1) We propose Instruction-guided Visual Masking (IVM), a novel approach that serves as a versatile plug-and-play module to enhance multimodal models through visual grounding. 2) We introduce the IVM-Mix-1M dataset and propose an LLMempowered Mixture of Expert pipeline to create visual grounding labels. 3) We present the DWSL algorithm for IVM training that automatically prioritizes high-quality training samples. 
related work: 
 Large Multimodal Models. LLaVA [41] first demonstrates promising capabilities in following complex instructions. Subsequent works such as LLaVA-1.5 [38] , MiniGPT4 [69] Qwen-VL [6] and CogVLM [57] , further enhance LMMs via refined model design and enriching the quality of training data, achieving state-of-the-art performance on diverse downstream tasks including visual grounding [36] , visual reasoning [55] , visual question and answering [18] . Moreover, by integrating the robotics action modality, LMMs perform versatile planning and manipulation in instructiondriven robotics tasks. Notable studies in this line of inquiry include PaLM-E [14] , the series of RT models [8, 9, 54] , and text-guided video planning diffusion models [15, 62, 7] . Despite the success, LMMs still struggle with complex visual grounding challenges, often misreading instructionirrelevant visual contents (Figure 1 ). To address this, researchers have tried to adapt existing visual modules to higher-resolution images to obtain better perception [40] , but with limited improvement.Visual Grounding Tasks. Visual grounding requires precisely localizing image regions corresponding to a referring expression, among which the RefCOCO series [64] is the most well-known benchmark, and numerous public visual grounding data are available [36, 63, 19] . Recently, LMMs incorporate these visual grounding data via visual-instruction tuning [41, 38, 69] , establishing new SOTA in this area [50] . To further broaden the reasoning ability of visual grounding, LISA [31] introduces a new task, reasoning segmentation, which demands higher capabilities in instruction comprehension. However, visual grounding is still limited to align simple instruction with specific objects, which cannot adapt to more complex instruction following tasks (e.g. Figure 2 ).Visual Grounding Augmented LMMs. Recently, a series of visual grounding methods emerged to enhance the performance of LMMs in complex visual scenes. V* [58] employs a heuristic search strategy to search, locate, and crop image areas relevant to instructions through a multi-step iterative process. VisualCot [50] is trained end-to-end with a customized dataset to achieve target localization capabilities. These two methods allow LMMs to dynamically focus on visual inputs until the correct answer is derived. However, these complex inference pipelines lead to substantial computational overhead, and their heuristic designs further hinder the extension beyond VQA to other multimodal instruction following tasks such as robotic control.Besides these explicit strategies incorporating additional visual grounding modules, other studies pursue refining data or introducing extra training targets to enhance the grounding capabilities of LMMs implicitly. ViGor [61] proposes a fine-grained reward modeling to enhance visual grounding of LMMs, and SynGround [23] introduces a pragmatic framework for image-text-box synthesis tailored for visual grounding. These methods, however, are primarily focused on the visual grounding task itself, overlooking its influence on downstream multimodal instruction following tasks. Distinct from previous efforts, this paper introduces a generic visual grounding model that is adaptable to any multimodal instruction following tasks, and provides a systematic investigation into the advantages of integrating an additional visual grounding model into downstream applications.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Please see Conclusion and Appendix A for details.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
instruction-guided visual masking: 
 To help multimodal models focus on instruction-sensitive image regions without distractions from irrelevant visual elements, we introduce Instruction-guided Visual Masking (IVM), a versatile plugand-play model that enhances multimodal instruction following via surgical targeted visual grounding. IVM aims to produce a heatmap H, given an image x img and a textual instruction x txt . The heatmap H identifies the critical image region to follow the instructions, as illustrated in Figure 3 , allowing multimodal models to easily zoom in on targeted image regions while ignoring neighboring areas.This formulation evokes the problem definition of Reasoning Segmentation (RS) [31] . There are two main differences: 1) IVM addresses a more challenging problem. RS tries to target single objects from simple instructions, e.g., "what is.., where is.., who is...", while IVM aims to include all instruction-related visual regions within the image given any instruction, which demands advanced and nuanced image-language grounding ability (as illustrated in Figure 2 ). 2) RS has clear ground truths but IVM does not. The instructions in RS primarily correspond to simple and semantic-meaningful objects that are straightforward for human annotations. IVM, however, deals with broader and more ambiguous instruction-related regions (e.g., the left bank regions in Figure 3 ), making the training and annotating much more challenging.To train an IVM model, the first main challenge is the scarcity of training data. Most existing Visual Grounding (VG) datasets [64, 31] typically feature simple instructions focused primarily on prominent objects within images, lacking both diversity and complexity required for IVM. To tackle this, we compiled one million data from various sources, including labeled visual grounding, unlabeled multimodal instruction following, and robotics data. As outlined in Section 3.1, scaling human annotations is challenging due to the high complexity of such data. Therefore, we introduce an LLM-empowered Mixture of Expert pipeline that integrates SOTA visual grounding models to efficiently generate reliable annotations. We further manually annotate a smaller dataset to compensate inaccuracies in auto-generated labels. The resulted combined dataset, IVM-Mix-1M, comprises one million data samples ready for IVM training, which can be found in https://github.com/2toinf/IVM. LLM-empowered Mixture of Expert Annotation Pipeline. Leveraging the power of LLM, this pipeline can efficiently generate high-quality annotation, which consists of two components (Figure 4 ): 1) Labeled visual grounding data. We collect 250K labeled VG data from multiple sources including VG caption [36] , Flickr30K [63] , VSR [3] , OpenImage [30] , and RefCoCo [64, 37] , which provide bounding boxes with simple instructions for each image. To increase the diversity and complexity of instructions, we utilize GPT-4 [1] , known for its robust language understanding and generation capabilities, to create diverse instruction-answer pairs based on existing language instructions. 2) Unlabeled Visual-Instruction-Following (VIF) and robotics data. We sample a 700K subset from LLaVA-Instruction-tuning [41] for VQA-type data, and a 50K subset from OpenX [54] for robotics data. Given that these data lack grounded labels but contain complex instructions, we use GPT-4 to simplify the language instructions by prompting it to infer the names of targeted objects necessary for following the instructions. These simplified instructions then guide existing VG models to generate candidate labels. To ensure the quality of these labels and compensate for the ambiguous nature of the IVM task, we integrate proposals from several VG experts, such as Grounding-Sam [49] , LISA [31] , AlphaClip [52] , and OwlViT [20] , via an ensemble approach.Manual Annotation. Despite integrating the most advanced models, the auto-generation design still faces challenges that can lead to data inaccuracies. First, employing LLM to simplify or complicate language annotations without considering image content can introduce uncontrollable biases. Second, as the task exceeds the capabilities of existing models, it becomes impossible to totally exclude low-quality annotations that contain irrelevant visuals or mistakenly filter out critical contents. Thus, to enhance the overall quality of the dataset, we further manually annotate a 10K subset of the constructed dataset to inject human expert knowledge. Data Analysis. Here, we provide quantitative analysis on the IVM-Mix-1M dataset. Figure 5 depicts the data quantities w.r.t the percentage of annotated instruction-related image area. Here, each ratio range is further categorized by different data sources, where manually annotations are treated as a separate category (Human), while all others are machine-generated. Our analysis reveals that the instruction-related image regions only occupy a small fraction of the total image area (e.g. most data have less than 40% instruction-relevant image regions), indicating that most visual contents may cause distraction and corroborating the necessity of visual masking for instruction following tasks.The challenge now is to train the IVM model with a small high-quality human-annotated dataset (D e ) as well as a large but mixed-quality auto-generated dataset (D o ). Training naively on the combined dataset may yield suboptimal results due to inaccuracies in auto-generated labels, while solely using limited human-annotated data is insufficient. Inspired by recent advances in imitation learning using mixed-quality data [60, 65] , we employ a Discriminator-Weighted Supervised Learning (DWSL) framework to effectively leverage the strengths of both auto-and human-annotated data.Discriminator Training. Specifically, we introduce a discriminator d optimized by Eq. ( 1) to assign high weights to high-quality annotations and vice versa: where (x img , x txt , H) are image-instruction-heatmap pairs sampled from D o and D e datasets. Eq. ( 1) is similar to the one in GAN [17] , but the "fake" data in [17] is replaced by machine-generated data from D o . After training with Eq. ( 1), the discriminator d assigns high weights to high-quality human-annotated data from D e and relatively high values to similarly high-quality data from D o that aligns with human preferences, acting as a judge for annotation quality.EQUATIONDiscriminator-weighted IVM Training. Then, we apply the trained discriminator as a weighting function for the IVM training objective:EQUATIONEQUATIONwhere λ bce and λ dice are set to 1.0 and 1.0 to balance the binary cross-entropy loss (BCE) and the DICE loss for segmentation (DICE) [28] , respectively. f (x) ≥ 0 can be any non-negative, nondecreasing function. For simplicity, we set f (x) := min(max(0.1, x), 1). This allows the weighting function f (d(•)) in Eq. ( 2) to dynamically prioritize training with high-quality data determined by the discriminator d. This approach maximizes the usage of reliable annotations in D o to compensate for the small D e , while minimizing the impact of low-quality data in D o , thus optimizing performance.The overall model framework is illustrated in Figure 6 . Due to its complexity, IVM requires both reasoning and precise localization of the target object, closely paralleling reasoning segmentation [31] . Consequently, for the heatmap generator, we adopt a model design similar to that of LISA [31] . Specifically, we first extract dense image features using an isolated vision backbone and multimodal representation from an LMM, which processes image-instruction pairs. These two types of features are then fed into a lightweight generator that integrates them to produce a dense prediction.For the discriminator, we deploy a lightweight discriminator that encodes the segmentation map using a two-layer convolution network. This discriminator interacts with the outputs of the LMMs through multiple cross-attention operators and finally outputs a quality score for each sample.Trainable Parameters. To enhance training efficiency, we freeze the pre-trained large foundation models and perform LoRA finetuning [25] . The vision backbone, inherited from Segment Anything Model [29] , is completely frozen, while the lightweight generator and discriminator are fully finetuned. Notably, we utilize a shared LMM for both the generator and discriminator branches but employing separate LoRA parameters to avoid interference between the two tasks.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiments: 
 In this work, we employ LLaVA-7B [42] as the LMM and SAM-H [29] as the vision backbone for our IVM model (Figure 6 ), which is trained on the IVM-Mix-1M dataset using the proposed DWSL algorithm. More details on the architecture and training can be found in Appendix C. We conduct extensive experiments to assess the effectiveness of the IVM model. Specifically, we utilize the heatmap generated by the IVM for image post-processing. These processed images can then be seamlessly fed into downstream multimodal models for diverse tasks, as shown in Figure 7 . Unless otherwise specified, we use the image post-processing method of overlaying and cropping to discard instruction-irrelevant image content. A detailed discussion on post-processing methods is presented in Section 4.2. We also provide more evaluation results and analysis in Appendix E. Integration with Commercial Chatbot. We use GPT4-V [1] as the base model. Considering the superior perception and reasoning capability of GPT4-V, we evaluate IVM-enhanced GPT4-V on V*bench [58] , a recently proposed challenging VQA-type benchmark characterized by images with abundant redundancies. Results are presented in Table 1 . The accuracy of the vanilla GPT4-V is mediocre (55.0%). Our IVM model, however, can significantly improve the performance (+26.2%) and establish a new state of the art on this benchmark, even surpassing the task-specialized SEAL [58] that requires a complex heuristic visual search pipeline.Integration with Open-sourced LMMs. To demonstrate the versatility of our IVM model, we further integrate it into an open-sourced LMM, LLaVA-7B [38] . We conduct extensive experiments across various benchmarks, including EgoThink [10] , POPE [34] , MME [16] , GQA [27] , SQA [44] , and VQAv2 [18] . As shown in Table 2 , our IVM-enhanced LLava-7B gains consistent performance improvements, achieving comparable performance to (even surpassing) LLaVA-13B on EgoThink, POPE and MME. Although IVM-enhanced LLaVA-7B and LLaVA-13B [38] have roughly the same number of parameters, the latter integrates more powerful pretrained foundation models. In contrast, our IVM model allows the 7B model to outperform the 13B model by merely simplifying visual input, further validating the power of visual masking.Meanwhile, IVM-enhanced LLaVA-7B does not show significant gains on GQA, SQA and VQAv2, which is expected, as these benchmarks do not heavily rely on grounding capabilities: VQAv2 and GQA contain relatively simple visual input where most regions of the images are instruction-relevant, while SQA primarily focuses on assessing model reasoning capability.Comparison with Reasoning Segmentation Model. We also compare against LISA [31] , which is most analogous to IVM. We provide carefully tailored prompts like "what should we focus on the image to follow the given instruction? Give me the seg" to extend LISA into visual masking task. However, even with larger 13B model and extensive tuning of input prompt, masks generated by LISA consistently result in severe performance degradation on all tasks.Evaluation on Real Robotic Control. We also plug the IVM model into robot control tasks to help robot model improve generalization. Specifically, we evaluate a language-conditioned behavior cloning (LCBC) robot agent trained with or without IVM masked images. Figure 9 clearly demonstrates that without IVM assistance, the LCBC robot agent suffers from severe performance drop when noticeable distractions are applied. With IVM assistance, however, the agent consistently pays close attention to correct instruction-related image regions, performing robustly against diverse distractions such as human disturbances and numerous task-irrelevant objects of various colors and shapes. This demonstrates promising potentials of using IVM to enhance embodied agents to follow complex instructions in unseen scenes with plenty distractions.We ablate the key components of IVM and report overall accuracy improvement(%) of IVM-Enhanced GPT4-V, evaluated on V* bench [58] due to its high demand on precise visual grounding abilities.Training Data. We investigate the impact of IVM-Mix data characteristics on IVM performance from two key perspectives: 1) Large machine-annotated data volume clearly enhances IVM model performance, as illustrated by the progressive improvement in Figure 10 (a) with increased machineannotated data volume (red, blue and yellow line). This demonstrates the effectiveness of our proposed LLM-empowered Mixture-of-Expert pipeline in generating reliable data for IVM training.2) Figure 10 (a) also reveals that incorporating human annotations significantly boosts training DWSL Framework. We also explore the efficacy of the DWSL framework in Figure 10 (a) by comparing IVM training using: 1) DWSL (red line), 2) traditional Supervised Learning (SL) without DWSL (blue line), and 3) SL on limited human data (gray line). The results demonstrate that DWSL effectively leverages both human-and auto-annotated data, particularly as the volume of machine-annotated data increases, enjoying higher asymptotic performances. This is expected as machine-annotated data often contain inaccuracies and training naively using all these data can lead to suboptimal results. Meanwhile, the limited human data alone cannot provide satisfactory outcomes. DWSL, however, addresses these challenges by dynamically prioritizing good samples and discarding misleading ones, resulting in stable and improved results. This is further illustrated in Figure 10 (b) which visualizes the outputs of the discriminator for each sample, where the discriminator can correctly retain good samples (e.g. Human) and filter out low-quality data with lower weights. Mask Deployment Strategy. We investigate the impact of mask deployment strategy on downstream applications. While more complex solutions such as visual search algorithms [58] can be employed, our investigation focuses solely on simpler approaches to understand the intrinsic capabilities of IVM model. Specifically, we examine four basic masking methods: overlay, blur, grayscale, and cropping, as illustrated in Figure 11 . In particular, for the crop method, we find the smallest area that retains all the activated (>0) values in the heatmap and crop it. Table 3 demonstrates that IVM maintains robustness across all simple post-processing methods, where overlay+crop enjoys the most performance enhancement and thus is used as our default mask deployment method. 
conclusion: 
 We introduce Instruction-guided Visual Masking (IVM), a generic and powerful visual grounding method that enhances broad multimodal instruction following tasks in a plug-and-play way. By masking out all instruction-irrelevant image regions, IVM effectively injects superior visual grounding ability to downstream LMMs non-intrusively, significantly boosting both commercial and opensourced LMMs and achieving state-of-the-art results across numerous challenging multimodal benchmarks. Real robot experiments further demonstrate the versatility of IVM, showcasing the potential to deploy IVM to embodied robotic tasks where failures caused by distractions are longstanding challenges. For further improvement, one promising direction is to finetune LMMs using IVM-generated heatmap as an additional input channel to reduce suboptimal heuristics caused by mask deployment methods. Due to resource limitation, we leave this for future work. We open source the IVM checkpoint and the IVM-Mix-1M dataset to help the community further explore relevant directions * . More discussion on limitations and future directions can be found in Appendix A. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] Justification: All data are collected from open-sourced and peer-reviewed dataset. The models used for annotations are also open-sourced and peer-reviewed.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: N/A Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
 
 Appendix: Here, we discuss our limitations, potential solutions and interesting future works.1. Computational Overhead. Note that IVM introduces additional parameters and computational overhead to directly enhance visual grounding ability of LMMs, which in turn indirectly improve the VQA performances. However, more VQA performance gains can be obtained if the same amount of additional parameters are end-to-end trained directly on VQA data (LLaVA-13B v.s IVM-Enhanced LLaVa-7B in Table 2 ). Solution and future work: Nevertheless, this is quite reasonable because IVM primarily focuses on improving the visual grounding ability, but accurate VQA also requires other abilities which can be learned through end-to-end training. End-to-end training, however, requires tremendous VQA data to implicitly and slowly improve the visual grounding ability, which is quite data-intensive. Both Table 1 and Figure 1 can show that even trained on billions of data, GPT4-V still performs subpar on tasks that require strong visual grounding ability. IVM, instead, can significantly boost the visual grounding ability of GPT4-V using just 7B parameters and less computations. One promising and interesting future direction is to include some auxiliary tasks to directly absorb the strong visual grounding ability in the IVM-Mix-1M dataset through end-to-end training like [59] .2. Data Quality. Due to task complexity, the machine-annotated data in IVM-Mix-1M inevitably includes wrong labels that mistakenly exclude instruction-sensitive image regions or suboptimal labels that not fully mask out all instruction-irrelevant areas. These inaccuracies may lead to suboptimal IVM model. We propose a DWSL framework to tackle this. However, the DWSL framework relies on a learned discriminator and a human-designed f (x) function, which may not exclude all inaccuracies. Solution and future work: We have clearly demonstrated in Figure 10 that with a simple f (x) and a lightweight discriminator, DWSL consistently outperforms the naive Supervised Learning (SL), doing pretty well on prioritizing good samples and meanwhile identifying inaccurate labels. To further enhance this, one can use other advanced techniques such as Reinforcement Learning from Human Feedback (RLHF) [46, 5, 26] to provide more fine-grained judgement on annotation qualities, or resort a theoretical-soundness f (x) [60] to achieve better results. In addition, one can also use our pretrained IVM model to directly generate high-quality heatmaps to enhance the machine annotations.3. Mask Deployment Methods. In this paper, we directly use the simple post-processing method to apply the IVM generated heatmaps on images, which then are fed into LMMs to perform downstream tasks. However, these post-processing methods introduce some heuristics, which may be suboptimal for downstream LMMs. In addition, LMMs may not see many masked images during pretraining, thus some distributional shift may occur. Solution and future work: Although these limitations exist, IVM still obtain consistent improvements using diverse mask deployment methods, as shown in Table 3 , which showcases the great versatility of IVM to inject visual grounding abilities. To further improve this, one strategy is employ some task-specialized visual search method [58] , but will bring many computational load during inference and limit the versatility on embodied agents. Another promising direction is directly using the IVM generated heatmaps as an additional input channel to finetune the LMMs like [52] , which can fully eliminate the heuristics of post-process methods, may bring larger performance gains. Due to resources limits, we leave this for a future work.4. Fine-grained Heatmaps. Note that the IVM generated heatmaps cannot provide exact semantic object segmentation with clear contours like reasoning segmentation [31] offers. Discussions: We want to clarify that this is an advantage of the IVM model rather than a limitation. This is because of the ambiguous nature of the visual masking task. For this task, the ground truth heatmaps are mostly less semantic-meaningful for annotations as discussed in 3.1. So, we ensemble the annotation proposals from different visual grounding methods for data annotation, which will make the trained IVM model robust to include instructionrelevant image areas, rather than being aggressive to exclude some instruction-sensitive pixels like reasoning segmentation [31] does illustrated in Figure 2 .Overall, although some limitation exist, we have thoroughly discussed potential solutions to these limitations. Moreover, in this paper, we have demonstrated the superior effectiveness and versatility of IVM to directly inject strong visual grounding ability to downstream LMMs or embodied agents, representing a pioneer effort to extend traditional visual grounding methods towards a more complex and generic setting that covers diverse multimodal instruction following tasks.This paper aims to advance the field of artificial intelligence, where no significant negative social impact is observed in this paper. The IVM-Mix-1M may contain some potential privacy issues and biases. However, in this paper, nearly all data are collected from open-sourced data, which have been well peer-reviewed, thus resolved this ethical concern.In this section, we primarily focus on the architectural design of the lightweight generator and discriminator, as both the Language Model Multitask (LMM) and the vision backbone are derived from the powerful foundation models (LLaVA & SAM). Both the generator and discriminator utilize the same transformer-based decoder block, as depicted in Figure 12 . We employ two such blocks for both the generator and discriminator. Specifically, the generator produces dense predictions by upscaling the output features of the decoder block through a straightforward upsampling operation.In contrast, the discriminator first employs a two-layer convolutional downsampling network to encode segmentation labels. This network, in conjunction with the decoder block and a simple MLP (multi-layer perceptron) head, outputs the weights. We adopt 8 NVIDIA 80G A100 GPUs and take 4 days to train our IVM model. The training scripts are based on deepspeed [4] engine and the training hyperparameters can be found in Table 4 .We evaluate our IVM on diverse multimodal benchmarks, including general VQA (VQAv2 [18] , GQA [27] , MME [16] ), first-person perspective QA (EgoThink [10] ), scientific QA (SQA [44] ), [34] ) and V* [58] , a recently proposed challenging benchmark with high-resolution and complex visual input.Our evaluation employs a two-stage inference pipeline: the image is firstly simplified by IVMgenerated heatmap and mask deployment methods; Subsequently, the simplified image is fed into downstream LMMs(GPT4-V [1] , LLaVA [38] ) without finetuning. We adhere to the official procedures of each benchmark to evaluate the output of LMMs and report the results.Task descriptions. The real robot experiments evaluate several pick and place manipulation tasks that require strong visual grounding abilities. Specifically, we evaluate on 4 tasks as shown in Table 5 , following the task definitions in DecisionNCE [35] . For each task, we collect around 100 demonstrations using the demonstration collection system in BridgedataV2 [56] . We take both a side camera view and a wrist camera view as the vision inputs, as shown in Figure 13 . For each demonstration, the environmental steps are around 50 steps. During data collection, the object and robot locations are randomly initialized, and the scene also has lots of randomly located distractors with varied shape and color. Training details. Here, we train Language-Conditioned Behavior Cloning (LCBC) policies using DDPM [24] loss since diffusion policies are good at fitting complex data distributions [67, 56, 2] , especially human demonstrations [11] . For model architecture, the side and wrist images are augmented and then passed through a shared ResNet50 [22] image encoder and get an image embedding for each camera view, following [56] . As the downstream data is quite limited, we load the ImageNet [13] -pretrained ResNet50 image encoder and further train it on the small robot data. Meanwhile, the language instruction is passed through a frozen T5 text-encoder [48] , which is fused into the image encoder via Film conditioning layers [47] . Then, this language-conditioned image embedding is passed through a MLPs with residual connections similar to IDQL [21] , which then outputs the predicted noise in DDPM [24] . To obtain smoothed policy rollouts, we adopt Action Chunking and Temporal Ensemble from ACT [66] with a chunking size 4 rather than 100 in [66] because the episode horizons in this paper are only around 50. The LCBC policies are trained either on the original side camera view (without IVM assistance) or on the IVM-masked side camera view (with IVM assistance) for 200K steps with a batch size of 64. The training can be completed on 2 NVIDIA RTX4090 GPU in 17h. All hyperparameters are summarized in Table 6 . Yes [56] Evaluation details. We first evaluate the trained LCBC policies without strong distractions, where no or only small distractors appear in the image. Then, we add lots of distracting objects with varied shapes and colors, and even introduce strong human disturbance to attack the LCBC policies. For each score reported in Figure 9 , we evaluate 10 episodes and report the success rates.For labeled visual grounding data, We provide the following prompt to drive GPT-4 [1] to generate more complex instructions based on given language annotations.[Image Description] %s[System] You are an AI visual assistant, and you are seeing a single image. What you see are part of the image and are provided with a simple phrase. Please generate any instructions that can be executed based on the content of the picture described, including simple queries about the content of the picture, such as the object types, counting the objects, object actions, relative positions between objects, etc. Also consider more complex questions that require reasoning. For unlabeled visual instruction following data, we first try to simplify complex instructions. Specifically, we employ GPT-4 to infer the necessary object for executing the given instructions based on these instructions and a simple image caption. If the dataset lacks captions, they can be generated using an existing caption model like BLIP-Caption [33] . Below, we outline the prompts specifically designed for GPT-4.[Image Caption] %s[Instruction] %s[System] You are an helpful AI assistant. I need to reply to the previous instruction based on an image, and I have a simple caption for the image. Please note that there may be objects in the image that I did not detect. Since you cannot view the image, please list any potential objects that might influence my responses, separated by semicolons, in a single line without any additional output. If you believe that the number of objects could be too extensive and might hinder my judgment, print 'None'.With the simplified instruction, we can adopt existing visual grounding models to generate the candidate label. Specifically, we utilize four models: AlphaCLIP [52] , LISA [31] , OwVIT [20] and Grounding-SAM [49] and the inference pipelines are provided in the official implementation of these models.As IVM is an extension of traditional visual grounding task, we also evaluate our IVM on RefCoCo, RefCoCo+ and RefCoCog [64] . We reported the accuracy (IOU-50%) on the validation split in Table 7 .As a generalist model capable of handling complex instructions, our IVM achieves performance comparable to that of state-of-the-art (SOTA) specialist models. In this section, we provide more visualization result in VQA-type data as shown in Figure 14 .Failure Case. Although we observe numerous successful instances, our IVM still faces significant challenges, as illustrated in Figure 15 . We summarize these challenges into three categories: missing target, misguided target, and insufficient reasoning.(a) Missing Target: Challenges arise when target objects are relatively small and scattered around many separate image corners. In this case, accurately detecting all of the targeted objects is quite difficult. Even specialized open vocabulary detection models struggle with this task. For example, the cup on the right in the image is masked by the IVM mistakenly. However, we still observe that the IVM-generated heatmap for the right cup is partially activated, meaning that IVM have partially focus this regions. We believe by providing more training data, IVM can handle this better. (b) Misguided Target: Accurately Localizing tiny target objects is a recognized challenge [58] , especially when similar but more obvious objects are present. For instance, IVM incorrectly focuses on the more centrally located shoes of another man, instead of the shoes of the man wearing the red hat at the edge of the picture. However, this instruction is pretty challenging that at first glance, even a human might struggle to spot the man with the red hat in the left corner. We will leave challenge scenarios like this for future research.(c) Insufficient reasoning: The objective of the IVM task is to assist LMMs in extracting visual features more effectively to better follow instructions. Thus, the demands on the model's reasoning capabilities extend far beyond mere object localization. Although IVM demonstrates strong performance, it sometimes overlooks additional image content necessary for accurately following instructions after correctly locating the target object. For instance, while IVM successfully identified the braking motorcycle, it failed to recognize that answering the question requires knowledge of the positions of both motorcycles simultaneously. We attribute this issue to biases in the training data. By incorporating more complex instructions and diversified labels, we anticipate that our model will achieve improved performanceHere, we provide more evaluation rollouts of the IVM-assisted LCBC agents under strong distractions. Figure 16 clearly demonstrates that even under strong distractions like the background are full of distracting objects with similar colors or shapes to the targeted objects, and strong human disturbances that adversarially attack the robots, the IVM-assisted LCBC agents can still complete the tasks pretty well, enjoying high-level of generalization and robustness thanks to the superior visual grounding ability injected by IVM. More videos can be found in the supplementary materials. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.* https://github.com/2toinf/IVM 


Decision: Thank you for the detailed rebuttal. After reviewing the reviewers' comments and the authors' responses, I appreciate the strong contribution of this paper to the field of multimodal instruction following. The proposed Instruction-Guided Visual Masking (IVM) model effectively enhances multimodal models by enabling precise focus on task-relevant regions, showing significant promise for improving performance across various downstream tasks. Given the strengths and the thorough validation of the proposed methods, I recommend accepting this paper.

Reviews: review: paper_summary: paper_summary: For the purpose of precise instruction following performance in LLM, this paper proposes a versatile grounding model that is compatible with diverse multi-modal models. Leveraging the LLM, a visual masking data generation pipeline is built and 1 million image-instruction pairs are constructed. On top of it, an Instruction-Guided Visual Masking (IVM) model is trained to focus on task-relevant regions that align with complex instructions.
strength_weakness: strength_weakness: 1. This work proposes a visual grounding model. However, it seems that the comparison with state-of-art visual grounding methods (VG) is missing. I understand this task involves slight differences with reasoning segmentation (RS) tasks. But will the VG task evaluation be more direct?
2. This approach relies heavily on the human manually labeled data, without which the performance will significantly drop. Therefore, it is more like a dataset creation work. Finetuning on this dataset, other compared works might also achieve similar or even better performance.
3. The derived framework is a little bit heavy, with LLM and heavy visual rendering head involved . There might be a more efficient network framework to achieve similar performance.
questions: questions: Refer to the questions in weakness section.

Other questions related to technical details.
1. How does this approach adapt to images with distinct image resolution? Is there any specifically designed relative position embedding?
2. Is there any plan to extend this work to more complicated robot action? The pick and put are very simple action. The real world application will involve more convoluted operations such as visual navigation or sequential manipulation.
limitations: limitations: Authors adequately addressed the limitations.
paper_summary: paper_summary: This paper presents IVM (Instruction-guided Visual Masking). The key idea is that we could mask out the instruction-irrelevant regions in the given image. The trained model is tasked to mask out the irrelevant regions, enforcing the multimodal model to focus on the task-related visuals. Such grounding-centric approach is effective in enhancing different multimodal models.

The paper also details their solution to create a large number of reliable pixel-level labels. The paper presents a MoE pipeline with various visual grounding models to collect reliable labels.

The paper further introduces DWSL for IVM training. DWSL helps IVM training to focus more on higher quality training data.
strength_weakness: strength_weakness: I don't find significant problems in the paper. One possible improvement for this paper is that it would be interesting to provide some typical failure cases of the proposed method.
questions: questions: see weakness
limitations: limitations: N/A
paper_summary: paper_summary: This paper introduces the Instruction-guided Visual Masking (IVM), a versatile visual grounding model designed to improve alignment between textual instructions and specific image regions. It outlines the development of a visual masking data generation pipeline and a new learning technique, Discriminator Weighted Supervised Learning (DWSL), which prioritizes high-quality data samples to enhance performance on multimodal tasks.
strength_weakness: strength_weakness: 1. The IVM model architecture shown in Figure 6 is not conducive to understanding the approach.
2. It is recommended to compare more methods on a multimodal benchmark.
questions: questions: 1. The paper mentions that the IVM-based model does not show significant gains on the GQA, SQA, and VQAv2 benchmarks. It suggests that these benchmarks may not depend heavily on grounding abilities, which raises some doubts for me. Intuitively, the results for simple visual inputs related to instructions should be better.
2. Additionally, I am concerned that the improved performance of the proposed method over comparative models might largely be attributed to the use of a larger Visual Grounding dataset. I am interested in seeing how other models would fare if they were trained or fine-tuned using the same dataset.
limitations: limitations: Limitations and broader impact have been discussed.
paper_summary: paper_summary: In this paper, the author introduce Instruction-guided Visual Masking (IVM), a generic and powerful visual grounding method that enhances broad multimodal instruction following tasks in a plug-and-play way. By masking out all instruction-irrelevant image regions, IVM effectively injects superior visual grounding ability to downstream LMMs non-intrusively, significantly boosting both commercial and open-sourced LMMs and achieving state-of-the-art results across numerous challenging multimodal benchmarks.
strength_weakness: strength_weakness: 1. The routine of retraining models by preparing new datasets, as described in this paper, can be considered somewhat old-fashioned. 
2. The technical insight of this paper is relatively weak, and no strong novelty technical methods are proposed.
questions: questions: I have a few minor concerns that I would like the author to address.

1. The routine of retraining models by preparing new datasets, as described in this paper, can be considered somewhat old-fashioned. Recent literature has proposed similar grounding datasets, such as Ferret (GRIT) [1] and Kosmos-2 (GRIT) [2]. However, these two works are not cited. There should be some new discussion on motivation about the dataset in this paper.

2. Why does the instruction-related area between 90% and 100% in Figure 5 increase abnormally significantly? It is necessary to consider whether these data needs to be cleaned.

3. Formulas 1 and 2 in the paper are somewhat deliberately mystifying. I understand that the purpose is to learn an optimal model from the hand-labeled data and the auto-generated data. This process is somewhat similar to the pseudo-label curriculum learning proposed in CLIP-VG [3]. It is suggested to increase relevant discussion and citations.

4. I have some confusion regarding Figure 6 of the paper, which requires modification. Specifically, (1) the discriminator and generator should be two separate processes, while in Figure 6, the image, text, and attention are depicted as being fed to the model simultaneously; (2) It is unclear whether the LLM in the figure represents one model or two models since there are two LoRAs shown but only one LMM is illustrated.

5. A framework diagram of the inference model used in the downstream experiment should be drawn according to Figure 6, so as to illustrate how the IVM assists the model in inference.

6. Since the paper claims that IVM is a plug-and-play method for assisting LLM, so I am curious about how this paper performs on the RefCOCO/+/g dataset after incorporating IVM. After all, the motivation behind this paper stems from the visual grounding task, and there has been a number of research on grounding multimodal large language models, such as Ferret [1], Kosmos-2 [2], and LION [4].

7. Other writing issues, such as some vocabulary in the paper should be consistent. For example, 
(a) line 135 RefCoCo should be changed to RefCOCO to maintain uniformity; 
(b) Lora is used in Figure 5, while LORA is used in the main text, and it is recommended to use LoRA uniformly in order to be consistent with the original article; 
(c) LMM and LLM are used in this paper, however LLM has not the full vocabulary,  and it is suggested maybe better to use LLM and MLLM uniformly. These rudimentary errors should not appear in NeurIPS submission papers.

On the whole, this paper is a relatively solid work, the overall presentation is good, literature review is relatively sufficient, so I currently give a positive scores. Hope the author can address my concerns, and I will decide whether to raise down or raise up my rating according to the author's rebuttal reply.

--

[1] You, Haoxuan, et al. "Ferret: Refer and Ground Anything Anywhere at Any Granularity." The Twelfth International Conference on Learning Representations. 2024

[2] Peng, Zhiliang, et al. "Kosmos-2: Grounding multimodal large language models to the world." arXiv preprint arXiv:2306.14824 (2023).

[3] Xiao, Linhui, et al. "CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Grounding." IEEE Transactions on Multimedia (2023).

[4] Chen, Gongwei, et al. "Lion: Empowering multimodal large language model with dual-level visual knowledge." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
limitations: limitations: see weaknesses.

Comments: title: Thanks for the further comments
I am not convinced by reviewer bwCu's comments and rating, as it seems that the reviewer bwCu was aware of the author's identity and deliberately gave high marks.
Thanks for the authors' responses. That's all.
Thanks for the responses!
Thanks for increasing the score!
The author's response is polite, and I appreciate the author's efforts during this rebuttal process.
comment
comment: Hi! The discussion is approaching to close. So, hope our additional responses successfully address your remaining concerns. If not, please do not hesitate to point them out. We would really appreciate any further comments that can improve the quality of our paper!
We thank the reviewer for the efforts and engagement in the discussion phase. Regarding the remaining concerns, we provide additional responses as follows:

>**Additional responses to Q1**
- For Q1, we provide the analysis of the relationships about data quantities w.r.t different `instruction-relevant visual ratios (IVR)` （the ratios of preserved pixels in the original images） generated by IVM on different multimodal benchmarks including the simple GQA, SQA, VQAv2 (due to time limits, we evaluated on 1/10 subset of the VQAv2 dataset, if the reviewer would like to see results on full datasets, we will try our best to finish the experiments by the author rebuttal deadline) and the more complex V* and EgoThink benchmarks.

|Benchmark|<20% (IVR) |20%-40% (IVR)|40%-60% (IVR)|60%-80% (IVR)|>80% (IVR)|
|---|---|---|---|---|---|
|V*|100%|0%|0%|0%|0%|0%|
|Egothink| 67% | 14% | 4% | 3% | 12%|
|VQAv2 (40K samples)| 13% | 16% | 12% | 10% | 49%|
|GQA| 17% | 14% | 8% | 4% | 57% |
|SQA| 0% | 5%| 11% | 7% | 77% |

- These statistics demonstrate that most visual contents in GQA, SQA and VQAv2 benchmarks are instruction-relevant (>80% IVR). Existing base MLLMs can easily focus on the correct image areas to follow the instructions, rather than be distracted by the minor visual distractors (for example, see Figure b in the PDF in **General Response**).
- On the contrary, only a small ratio of visual contents in the challenging V* and Egothink benchmark are instruction-relevant (<20% IVR). In this case, MLLMs are more likely to be distracted by a lot of irrelevant visual content if the MLLMs' visual grounding ability is not that strong. With the IVM assistance, however, the performance can be greatly enhanced via the explicit surgical visual grounding.

>**Additional responses to Q2**
- For Q2, `LISA is the only comparable baseline in our setting`. Specifically, most MLLMs cannot be directly finetuned or trained in our setting as they are not primarily designed to predict dense image heatmaps given complex instructions that require strong reasoning ability. Among all these MLLMs, LISA is the most relevant work that can be fairly compared, as LISA is specifically tailored for reasoning segmentation, which shares some similarity with our setups.

Indeed, we have empirically compared the baseline method LISA (please check the Human+machine label (SL) baseline in Figure 9 (a) in our paper for details). Here, we summarize the results more directly (detailed comparisons of diverse data quantities and data components can be found in Figure 9 (a)).

|Training object & data|IVM on IVM-Mix-1M (Ours)|LISA on IVM-Mix-1M|
|---|---|---|
|Improvements on V* benchmark|**+26.2**|+16.2|

`If the reviewer has any further detailed concerns or requires any other experimental supports, please do not hesitate to point them out. We will be more than happy to address them and improve the quality of our paper.`
I agree with reviewer zLHv's perspective on the concerns regarding reviewer bwCu's comments and rating. 
Compared to the comments from other reviewers, reviewer bwCu's comments seem overly positive and lack the professional suggestions expected. 
This raises my curiosity about the reason behind the reviewer bwCu's strong accept rating.
After reading the comments of the other reviewers, I am not convinced by reviewer bwCu's comments and rating. To be specific,

- (a) The reviewer bwCu claims that his confidence in this paper is "absolutely certain about your assessment. You are very familiar with the related work." However, the reviewer bwCu did not give some valuable and in-depth comments to this paper. I think, if the reviewer bwCu is familiar with this field, he should be able to find the defects of this paper and make some valuable comments.
- (b) Reviewer bwCu gave nothing but praise for this paper;
- (c) Judging from the comments of the other three reviewers, there are still more or less problems in this paper. However, reviewer bwCu directly gave the extreme rating of "Strong Accept".

To sum up, it seems that the reviewer bwCu was aware of the author's identity and deliberately gave high marks.
Thanks for the authors' responses. That's all.
I feel that my concerns were not fully addressed. The explanations provided seem more intuitive and lack solid theoretical or experimental support, such as Q1 and Q2. Given this, I decided to lower my rating to weak accept.
Thanks to the reviewer again for the efforts and engagements, and open to any further comments!
We really thank the reviewer for increasing the score, and sorry for the confusion in the rebuttal phase! Regarding the remaining concerns, we provide the following detailed responses.

>**Explanation about Eq. (1) and Eq.(2)**

- Due to the space limits in the rebuttal phase (6000 characters), we did not include very detailed discussions on Eq. (1) and Eq. (2) in the rebuttal. The introduction of Eq.(1) and Eq. (2) is inspired by recent offline IL work [2] that tries to learn a good model jointly from a high-quality near-expert dataset (human data) and a mixed-quality suboptimal dataset (machine-generated data), which typically consists of two stages:

   - `Discriminator Training`. Eq. (1) tries to train a discriminator $d$ to distinguish between human data $\mathcal{D}_E$ and the machine-generated data $\mathcal{D}_D$. By doing so, the output of the discriminator $d$ becomes a confidence value for data quality ($d$ will output near 1 or near 0 if the data can be clearly identified as human or machine data; $d$ will output around 0.5 if the annotation quality is hard to judge), see Figure 9 (b) for discriminator output statistics. 

   - `Discriminator-weighted Supervised Learning`. After training the discriminator, the $d$ value can be an adaptive weight function to reprioritize the training annotations in Eq. (2), where high-quality annotations will be more fitted than low-quality ones. In this sense, the side-effect of bad annotations in the mixed-quality machine data can be largely filtered out, see Figure 9 (a) for detailed experimental evidence.

Here, all annotations are pre-collected rather than gradually updated during training as CLIP-VG does. So, we're more insensitive to the compounding errors than CLIP-VG.

>**Additional discussions on Q6**

- We are really sorry for the confusion here. Due to time limits, we tested IVM only on the validation split rather than all refcoco splits.  Now, we're trying to implement IVM-13B by scaling IVM on more human and machine data. After that, we will conduct more thorough experiments in the future.

>**I expect the authors to make corresponding revisions regarding the aforementioned issues in the final version.**
- Sure! We will consider all these helpful suggestions when revising our paper!

Thanks to the reviewer again for the efforts and engagement in the discussion phase, and open to any further comments!

[2] Discriminator-weighted offline imitation learning from suboptimal demonstrations. ICML 2022
I noticed that the author did not fully respond to some of my questions, such as explaining equations 1 and 2 in Q3, etc. Additionally, the authors' results in Q6 appear to be insufficient compared to several baselines. Nevertheless, Considering the author's response is polite, and I appreciate the author's efforts during this rebuttal process. Therefore, I would like to raise my rating by 1 point to `weak accept`. I expect the authors to make corresponding revisions regarding the aforementioned issues in the final version.
Thank you. I have no further comment at this point.