Multimodal Inplace Prompt Tuning for Open-set Object Detection

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 11:08:16

Abstract: The integration of large language models into open-world detection frameworks significantly improves versatility in new environments. Prompt representations derived from these models help establish classification boundaries for both base and novel categories within open-world detectors. However, we are the first to discover that directly fine-tuning language models in detection systems results in redundant attention patterns and leads to suboptimal prompt representations. In order to fully leverage the capabilities of large language models and augment prompt encoding for detection, this study introduces a redundancy assessment metric to identify uniform attention patterns. Furthermore, in areas with high redundancy, we incorporate multimodal inplace prompt tuning (MIPT) to enrich the text prompt with visual clues. Experimental results validate the efficacy of our MIPT framework, achieving a notable increase across benchmarks, e.g. elevating GLIP-L from 22.6% to 25.0% on ODinW-35, and 9.0% improvement on LVIS.

Main paper: 
 introduction: 
 Detection systems with large language models mark pivotal advancements in the open world [55] . As text prompts and visual regions are aligned through contrastive learning [17] , new visual regions can be distinguished between classification edges shaped by linguistic world knowledge. This multimodal perception approach decouples the understanding of world knowledge from the only perspective of vision (e.g., traditional single-modal detection [10, 45] ), facilitating the development of detection systems towards greater scale and enhanced capabilities.Recent studies use two different methods to overcome the closedvocabulary limitations. First, using fixed semantic representations from the frozen language model can make full use of the smoothness of the text space for categorizing objects [1, 44, 64] . The classification edge brought by the plain-text-pretrained large language model [4, 37] , shows advances in the visual field. While innovative, such methods still have limitations in alignment for detection problems of long-tail and fine-grained categories. This is because alignment can only be unilaterally conducted from the visual backbone, but the classification boundaries of the text still lack visual discriminativeness. Transitioning from frozen to tunable language models can offer higher flexibility. Finetuning allows the language model to adapt its pretrained representations to the specific visual domain [8, 61, 69] . Through closer alignment between text prompts and visual regions, language world knowledge recalibrated by visual contrast can lead to better prompt representation, thus providing more precise classification boundaries. Despite the pivotal role of language models, the nuances of how these models learn and adapt within such multimodal frameworks have been largely overlooked.Initially, language models were designed to understand and generate contextualized texts. When integrated into detection systems [25, 34] , text embeddings provide a novel view to define visual classification boundaries. Recent fully-finetuned detection models [25, 34, 67] concatenate categories into a template and finetune the language model to obtain the prompt representation of the textual template. However, our research reveals a degradation in the efficiency of the attention mechanisms for language models when directly fully-finetuned, as shown in the Section 4.2.3. This redundancy can result in model's inability to effectively encode the categories for rich semantics. Thus in turn, falls to detect such fine-grained or long-tail categories, as shown in the left column of Figure 1 . Meanwhile, continuing full-scale fine-tuning on such categories would require substantial training costs. As depicted in the Figure 1 , comparing to the full-finetuning (e.g GLIP-T [25] with more than 200 million parameters trainable), we aim to propose an efficient method that utilizes a small number of trainable parameters to improve the detection effectiveness for long-tail and fine-grained categories.To this end, given the language model adaptation for detection that have been previously overlooked, we introduce a novel Multimodal Inplace Prompt Tuning Framework (MIPT) for open-set object detection. As shown in the right column of Figure 1 , our MIPT is for the first time designed to identify redundancy within the weight distribution of the language model, and enrich the textual representation through a light-weight integration of visual exemplars. Consequently, text priors imbued with world knowledge are effectively augmented with visual cues, resulting in an enhanced classification boundary. Specifically, we introduce a Jensen-Shannon Redundancy evaluation to detect redundancy by assessing the variance of attention patterns intra-and inter-heads within transformer layers. Upon identifying redundant patterns, we apply multimodal inplace prompt tuning in these layers to refine textual prompts using visual cues, thereby augmenting the semantic depth of text encodings and improving model learning efficacy. Inplace tuning is achieved by performing parameter-efficient fine-tuning [15] for language models, thereby allowing it to reparameterizing internal encoding patterns without changing the original model structure. Furthermore, we develop cross-modal self-distillation to refine multimodal alignment between visual features and the augmented textual features in the fusion layers. Our experimental outcomes on GLIP-T, GLIP-L, and G-DINO-T showcase the remarkable efficacy of the (a) Fully pretrained large detection models, such as GLIP, experience degradation in the language model, which makes encoding rich semantics for downstream categories challenging (especially failing to detect the fine-grained categories, such as various breeds of cats.). Resolving this issue through full parameter fine-tuning would entail substantial costs. (b) By diagnosing the redundant parts of the pretrained language model within the detector, we introduce visual clues into these redundant layers and employ inplace tuning with minimal parameters. This enhances text encoding and achieves precise fine-tuning with few samples for downstream tasks.MIPT framework, yielding substantial performance enhancements. Notably, GLIP-L (MIPT) demonstrates an impressive average precision improvement of 9.0% on the LVIS Val v1.0, reaching 25.0% on the ODinW-35, thus affirming our approach's capability to significantly elevate detection accuracy across diverse datasets.Our contributions can be summarized as:(1) We are the first to identify redundancy in attention distribution when integrating large language models into detection frameworks, which emphasizing the need for novel methods to improve fine-tuning and maximize their detection capabilities. (2) We introduce the Jensen-Shannon Redundancy metric to efficiently evaluate and pinpoint attention pattern redundancies in language models during fine-tuning for detection tasks, enhancing optimization across transformer heads in multimodal settings. (3) We propose Multimodal Inplace Prompt Tuning, a novel approach that recalibrates prompt representations using visual cues for language models. This adjustment is achieved without altering the original model structure, thus maintaining the integrity and generalizability of the model while significantly improving its performance in detection tasks. 
related work 2.1 open-set object detection: 
 The domain of open-set object detection has rapidly evolved with the integration of large language models and multimodal learning frameworks [19, 30, 54, 57, 58, 60, 66, 71, [73] [74] [75] [76] . MDETR [17] introduces a model that aligns text phrases with visual regions using a DETR-like architecture. This method improves object detection by using textual queries to guide the detection, resulting in a more versatile understanding of visual content. GenerateU [31] , which aims to identify and name objects within images without relying on predefined categories, thus extending beyond the limitations of traditional object detection methods. Works such as GLIP [25] and Grounding DINO [34] have demonstrated the potential of grounding object detection in language, formulating detection as a problem of language grounding to learn instancelevel visual representations with deep language-aware fusion. APE [48] combines detection and grounding into a unified model that can handle diverse tasks simultaneously. While previous work has significantly advanced open-set object detection, training largescale multimodal detection frameworks still requires substantial computational power and data. Therefore, efficiently fine-tuning existing detection frameworks to better adapt to new downstream data presents a cost-effective solution. In this paper, we introduce multimodal inplace prompt tuning. This approach involves freezing the original model and fine-tuning a small number of added parameters to enhance detection performance for better adaptation.By freezing most of the pretrained parameters and only updating a subset, models can achieve improved performance on downstream tasks [2, 3, 9, 18, 20, 29, 40, 49, 70, 72] . Adapter Tuning [14] have paved the way for more nuanced and efficient adaptation strategies with customized adapters. Since large language models are sensitive to prompt, Prefix-Tuning [27] and P-Tuning [35, 36] enable fine-tuning of models in a more efficient manner, focusing on the Figure 2 : An illustration of our proposed methods. To analyze the redundancy in the language model within the detector and precisely identify which parameters need fine-tuning, we consider both the variance within the attention distribution of the attention heads and between heads, based on the JSR. This approach helps to assess the level of redundancy in an attention layer. Then, by ranking the degrees of redundancy, we can accurately pinpoint the locations that require fine-tuning. After adding a small number of trainable parameters through LoRA in the language model, we concatenate pre-selected visual exemplars with the original category prompt as input. This allows the original language model to learn to refine the classification boundaries through visual exemplars via the MIPT approach.adaptation of prompt rather than the parameters within the model. The existing methods for adapting large language models, such as adapters, often result in unwanted side effects like inference latency, and the optimization of prompt tuning can be particularly challenging. LoRA [15] employs a low-rank decomposition approach to simulate parameter changes, where the updated weights of the trained low-rank matrices are simply added together to complete the update process. However, the wide variety of above methods and tasks makes choosing the appropriate one a challenge. UniPELT [41] further incorporates various methods into the transformer architecture. However, the fine-tuning methods mentioned above primarily involve adjustments from unimodality and do not concern the learning for multimodal data. To this end, our work aim to aggregate the multimodal information by tuning the text prompt with visual clues through low-rank adaptation.As model sizes increase, especially with large language models, a single set of weights is sufficient to tackle tasks across multiple modalities [5, 6, 12, 28, 33, 43, 51-53, 62, 63, 65, 68] . Inspired by this, MQ-Det [59] proposes modulated pre-training, enabling the detection models to utilize both textual and visual queries without extensive re-training. DINOv [22] delves deeper into visual incontext prompting for open-set segmentation, enabling the model to utilize provided bounding boxes and points as visual prompts. T-Rex2 [16] further addresses the challenge of long-tail categories in open-set detection through the use of visual in-context instructions.Constructing detection boundaries for open-set detection often requires the assistance of large language models, but few work has been done to examine whether changes in these language model are adequately adapted to visual tasks. Our approach investigates the redundancy issue of language models in a multimodal framework.It utilizes visual clues to address the suboptimal encoding of text prompts caused by this redundancy. 
method: 
 Training large-scale detection models from scratch incurs considerable computational and data costs [17, 33, 60] . Although previous studies have sought efficient strategies to enhance performance within existing detection frameworks [59, 67] , there has been scant research on the suitability of language model modifications for detection tasks. As illustrated in the half bottom of Figure 2 , our methods start by evaluating the positions of redundancy within the language model using Jensen-Shannon Redundancy (Sec. 3.2). Then, at the redundant positions, we introduce information from the visual counterpart by multimodal inplace prompt tuning(Sec. 3.3). Furthermore, we can use the features of visual regions and text embeddings after fusion module for further cross-modal distillation and alignment.(Sec. 3.4).Our methods mainly based on the grounding paradigm detection framework [25, 34] . As illustrated on the top of Figure Examining the entire pipeline of open-set multimodal detection, large language models play vital role within it. World knowledge pre-trained in large language models aids in constructing effective classification boundaries. Via full-scale training, although it enables large language models with world knowledge to better align with detection tasks, it can lead to degradation of attention patterns. If used directly in downstream task inference, it might not perform well, especially for long-tail and fine-grained categories. Continuing to resolve this issue through full-model training would further consume substantial training resources. MQ-Det [59] is the first to integrate visual queries into an established language-only query detector. However, MQ-Det introduces visual information into text encoding through a cascaded attention module, then empirically determines the optimal placement of this new module within the language model. This hinders the effective utilization of added parameters and the cost of model inference. Our study addresses redundancy in language model detectors by using metrics to efficiently introduce learnable parameters and improve performance through inplace prompt tuning without raising costs.We first introduce the Jensen-Shannon Redundancy (JSR). This novel approach aims to identify redundancy within the attention mechanisms of large language models. JSR applies the Jensen-Shannon divergence to evaluate and minimize redundancy within the attention mechanisms of large language models. This method quantifies divergence in attention distribution both within individual transformer heads (intra-head redundancy) and across different heads (inter-head redundancy). By pinpointing where attention patterns are overly uniform or duplicated, it identifies the most redundant parts of the model.Intra-head Redundancy is calculated to measure the redundancy of attention distributions within each individual head of a transformer layer. Given a head ℎ with token length 𝐿, let 𝐷 𝑖,ℎ represent the attention distribution for the 𝑖th token, corresponding to the sequence positions. The average attention distribution Dℎ for head ℎ is computed as the mean of the attention distributions across all token positions:EQUATIONSubsequently, the Jensen-Shannon divergence 𝐽𝑆 (𝐷 𝑖,ℎ || Dℎ ) is calculated for each token distribution 𝐷 𝑖,ℎ relative to the average distribution Dℎ . The intra-head redundancy 𝑈 intra,ℎ for head ℎ is then defined as the mean Jensen-Shannon divergence across all tokens, which is:EQUATION3.2.2 Inter-head Redundancy. Inter-head Redundancy evaluates the consistency of attention distributions across the different heads within the same layer. For each token position 𝑖, the average attention distribution D𝑖 across all heads 𝐻 , is calculated as:EQUATIONThe Jensen-Shannon divergence 𝐽𝑆 (𝐷 𝑖,ℎ || D𝑖 ) is then computed for the distribution of each head ℎ compared with the average distribution D𝑖 . The inter-head uniformity 𝑈 inter for the layer is determined by averaging these divergences across all token positions, yielding:EQUATIONThe final metric for assessing the importance of a layer, 𝐼 layer , is computed as a sum of the weighted intra-head uniformity for all heads and the inter-head uniformity, adjusted by a hyperparameter 𝜆 to ensure both components contribute on a comparable scale:EQUATIONwhere 𝜆 is selected to balance the contributions of intra-head and inter-head redundancy, facilitating a comprehensive assessment of the significance of each layer within the language model. By employing the JSR, we effectively pinpoint superfluous attention patterns and determine the optimal subset of parameters for fine-tuning. This strategy not only reduces the computational overhead but also ensures that performance enhancements are substantial by strategically targeting areas of the model that contribute the least to information diversity. Further details will be elaborated in the experimental section.Building upon our proposed Jensen-Shannon Redundancy metric, the next stage of model refinement is implemented through Multimodal Inplace Prompt Tuning. This process aims to enrich the model's text prompts with visual context, thereby addressing the challenge of redundant attention patterns and enhancing prompt representations inplace for improved object detection in open-world scenarios. (3) def __init__(self, config):(4)super().__init__(config) (5) self.register_parameter("vision_gate", nn.Parameter(torch.zeros(1))) (6) self.Q.add_lora() (7) self.V.add_lora()# Forward pass with text and vision inputs (9) def forward(self, text, vision, text_attention_mask, vision_attention_mask): (10) # The total number of visual exemplars in the vision clues bank # QKV operations (15) query_layer, key_layer, value_layer = self.QKV(hidden_states) (16) attention_scores = calc_attention_score(query_layer, key_layer)# Separate visual and textual information # Calculate context layer by weighting value layer (23) text_context_layer = calc_context(text_attention_scores, text_attention_mask, text_value_layer) (24) vision_context_layer = calc_context(vision_attention_scores, vision_attention_mask, vision_value_layer) (25) # Modal merge using vision gate (26) context_layer = torch.tanh(self.vision_gate) * vision_context_layer + text_context_layer (27) # Reshape and return output (28) outputs = (context_layer, text_attention_scores) if output_attentions else (context_layer,) (29) return outputs [59] , we expand the coordinates of all bounding boxes by a factor of 1.5, allowing us to capture the contextual backdrop of each target object. Subsequently, the raw images are processed through a visual encoder and a FPN [32] to extract multi-scale feature representations. ROI Align [46] is then employed to project the expanded bounding boxes onto the corresponding feature scale, extracting the target features from the feature map. These extracted object features are cataloged, forming a comprehensive visual clue bank that serves as a reservoir of candidates for subsequent utilization in our MIPT framework.Tuning via Visual Clues. Leveraging JSR, next step begins with a thorough evaluation of redundancy across various layers of the language model. This assessment facilitates an ordered layer selection based on the degree of redundancy, identifying those that are most amendable to enhancement, as shown in Figure 4 . Subsequently, we freeze the attention parameters of the original redundant layers. As shown in the line 6 and 7 of the Algorithm 1, a LoRA [15] bypass is then integrated into the query and value linear layers of the attention mechanism. The mathematical expression is as follows:EQUATIONIn equation 6, W 0 represents the frozen pretrained weight matrix of the model. B and A are the trainable rank decomposition matrices, specifically introduced during the adaptation process. x denotes the input to the weight matrix. By refining the model in this fashion, we facilitate the language model's ability to construct nuanced classification boundaries enriched with visual context. This not only streamlines the integration process but also preserves computational efficiency, as the core structural elements of the language model are left unaltered, thereby avoiding the need for additional inference modules. The core of Multimodal Inplace Prompt Tuning lies in the fusion of self-attention results (context layers) derived from both visual and textual streams. This fusion is guided by the learnable vision gate 𝛾, initialized in the line 5 of the Algorithm 1, which modulates the influence of visual information on the merged attention result (context layer).During the forward pass, the process starts by projecting and normalizing visual inputs. Visual counterparts are first passed throughcos 𝜃 = a linear transformation and normalization process. Specifically, given a set of visual embeddings, we apply a linear transformation to match the dimensionality of the text embeddings. This transformed visual information is then normalized to ensure stability and consistency in the scale of features, as shown in the line 11. The process is followed by concatenating these visual features with textual inputs to create a unified feature set, as shown in line 13. The purpose of this is to keep the computational process of the original text encoding unchanged. This combined feature set is then processed through a modified attention mechanism, first by performing Query, Key, Value (QKV) operations, as shown in line 15, and then by calculating attention scores to establish relevancies across the concatenated multimodal data, as shown in line 16. Distinctly, this method separates visual and textual features, as illustrated from the line 18 to line 21 in the Algorithm 1, computes attention results for each modality by applying their respective attention scores. The textual self-attention results (𝐶 𝑇 ), as shown in the line 23, is obtained directly from the textual stream's attention mechanism, without further modification. This preserves the integrity and richness of textual information, serving as a robust foundation for subsequent fusion. Meanwhile, to obtain the visual self-attention results (𝐶 𝑉 ), the attention of the text token will be calculated to assign the weight of several visual examples of the same class, thus weighting these visual examples to obtain a visual counterpart 𝐶 𝑉 optimized for this text token, as shown in line 24.A pivotal aspect of this method is inplace tuning for the visual self-attention results using the shared attention from the text, which is achieved through the application of Low-Rank Adaptation [15] to fine-tune the existing attention weights. The modal merge operation, as shown in line 26, computes the final modal-merged results 𝐶 by combining both modalities:EQUATIONThis equation illustrates how the model dynamically adjusts the contribution of visual information, ensuring that the final attention result leverages complementary cues from both modalities. The addition of 𝐶 𝑇 directly, without scaling, ensures that textual information retains its foundational role, while tanh(𝛾) • 𝐶 𝑉 introduces a modulated visual perspective.Continuing with MIPT, we further delve into the Cross modal Selfdistillation method. Take the fusion module of GLIP for example, which contains three submodules:EQUATIONEQUATIONEQUATION)where 𝐿 is the number of fusion block, 𝐶𝑟𝑜𝑠𝑠𝐴𝑡𝑡𝑛 denotes the fusion operation for text and image. 𝐼𝑚𝑎𝑔𝑒𝐸𝑛𝑐𝑜𝑑𝑒𝑟𝐿𝑎𝑦𝑒𝑟 takes the image features 𝐼 𝑖 and image features reconstructed from text 𝐼 𝑖 𝑡 2𝑖 as inputs. In a similar way, 𝑇 𝑒𝑥𝑡𝐸𝑛𝑐𝑜𝑑𝑒𝑟𝐿𝑎𝑦𝑒𝑟 takes the text features 𝑇 𝑖 and text features reconstructed from image 𝑇 𝑖 𝑖2𝑡 as inputs. CMSD uses the output from 𝐶𝑟𝑜𝑠𝑠𝐴𝑡𝑡𝑛 as input for the cosine similarity loss, with the goal of further aligning the semantic gap between image regions and category prompts. Concretely, as depicted in Figure 3 , firstly, text tokens and corresponding target image undergo 𝐶𝑟𝑜𝑠𝑠𝐴𝑡𝑡𝑛 fusion, where their features are combined. After fusion, the text features are detached from the computation graph to serve as targets, and then, based on the cosine distance, the gap between the image features and the text features is narrowed. We freeze the original parameters of 𝐶𝑟𝑜𝑠𝑠𝐴𝑡𝑡𝑛 in the fusion block and then introduce a trainable LoRA module to encourage 𝐶𝑟𝑜𝑠𝑠𝐴𝑡𝑡𝑛 to better distill cross-modal knowledge. This optimizes the congruence between the text token and corresponding ground truth region. This is mathematically expressed as:EQUATIONIn this formula, 𝑁 num_gt denotes the number of ground truth regions in the image, 𝐼 𝑖+1 symbolizes the pooled visual features, and 𝑇 𝑖+1 represents the targeted text embeddings. The Cross modal Self-distillation seeks to minimize this alignment loss, ensuring that the visual features are finely tuned to reflect the textual data, thus fostering a deep, semantic synergy between the visual and textual information. This process improves the model's ability to recognize and categorize objects based on both visual and textual cues, bridging the gap between the abstract semantic knowledge in language models and the concrete visual information captured by the detection system. 
experiment 4.1 experimental setup: 
 4.1.1 Benchmarks. In our research, we have assessed the effectiveness of our proposed methods across two distinctive benchmarks to demonstrate their robustness and versatility in object detection tasks.The first benchmark utilized is the LVIS dataset [11] , which contains a wide array of detection targets spanning more than 1200 categories. This dataset is notably leveraged for evaluating detection outcomes on long-tail distributed data. GLIP, for directly transferred evaluation, owing to its extensive category range where not all classes can be included in a single text prompt, we segmented the categories into multiple prompts. Each prompt encapsulated 40 classes, and the model was queried multiple times using varying prompts. The performance was reported on both the LVIS Minival subset, comprising 5000 images, and the comprehensive LVIS V1.0 validation set. The second benchmark is the ODinW dataset [21] , which aggregates 35 public object detection datasets to evaluate the comprehensive performance of pretrained large-scale detection models.To further evaluate core model performance, we utilized its less noisy subset, ODinW-13, which provides a concentrated measure of a model's fundamental detection capabilities.In the implementation details of our experiments, we employed GLIP [25] and G-DINO [34] as our baseline detectors. Firstly, through the Jensen-Shannon Redundancy analysis, we pinpointed the layers with the highest redundancy within the large language models, selecting the top six for further optimization. For instance, as illustrated in the Figure 4 , we utilized Jensen-Shannon Redundancy to assess the redundancy of the 12 attention layers in the BERT [4] within GLIP-T. Based on these results, we selected the 7-12 layers of the BERT model for subsequent fine-tuning. We then incorporated the LoRA module (via MIPT) serving as tunable parameters to adjust original attention weights for different modalities. Pre-extracted visual exemplars were utilized. For each category, five random images were chosen and processed through the vision encoder and ROIPool [46] to extract specific regional features [59] . For pre-training on Ob-ject365 [47] , we conducted a single epoch with all original model parameters frozen, adjusting only the newly introduced parameters. During this stage, the learning rate for LoRA was set to 0.00001, and the learning rate for the gates was 0.0005. The pre-training utilized 8 V100 GPUs for GLIP-T and G-DINO-T, and 16 V100 GPUs were employed for GLIP-L.The Baselines. In the comparison of our methods, we benchmarked against several leading approaches in open-vocabulary detection as shown in the Table 1 , including MDETR [17] , GLIP [25] , G-DINO [34] , BARON [56] , and OWL-ViT [42] , which utilize various backbones like ResNet101 [13] , Swin Transformers [38] , and Vision Transformers [7] . Our experiments revealed that the MIPT-enhanced versions of G-DINO-T, GLIP-T, and GLIP-L significantly outperformed their original counterparts. For example, G-DINO-T (MIPT) improved its Average Precision (AP) on the ODinW-13 benchmark by 3.8%, and GLIP-T (MIPT) showed an increase of 4.4% on the LVIS MiniVal dataset. Most notably, GLIP-L (MIPT) demonstrated exceptional performance, not only in comparison with its baseline model-achieving an AP increase of 7.7% on LVIS MiniVal-but also outshining all other models across every benchmark dataset, including the highest AP on ODinW-35 at 25.0%. This comprehensive performance uplift underscores the advantage of the MIPT framework in enhancing detection systems. MIPT's fine-tuning of under 1% of the model's parameters underscores the framework's efficiency, achieving state-of-the-art results with minimal parameter increase, a testament to the efficacy of incorporating visual clues and targeted parameter updates. 2 , GLIP-T model fine-tunes across all its parameters, totaling 233.4 million, which is resource-intensive and less adaptable to rapid deployment scenarios. In contrast, our MIPT-enhanced GLIP-T involves fine-tuning less than 1% of the total parameters, a stark reduction that still results in superior performance. In the 3-shot and 5-shot settings on the ODinW-13 benchmark, GLIP-T (MIPT) not only demonstrates a notable increase in accuracy (57.2% and 58.2%, respectively) but does so with an incredibly marginal trainable parameters (only 0.8% compared with original GLIP-T). This illustrates the efficiency of MIPT in achieving high precision in object detection with minimal computational overhead, making it particularly suitable for applications requiring quick adaptation to new environments with limited examples.Trainable params ODinW-13 (%) 3-shot 5-shot GLIP-T [25] 233.4M 54.9 56.4 GLIP-T (MITP)1.94M 57.2 58.2 5 represent the attention mechanisms within certain heads of a transformer model, specifically focusing on the redundancy of these mechanisms. Each vertical panel depicts the attention distribution from one particular head (e.g., Head 0-6 denotes head 6 in layer 0, Head 3-11 denotes head 11 in layer 3, etc.) to prompt concatenated by various category tokens such as 'ball', 'ballet skirt', and 'banana'. The strength and pattern of attention, indicated by the thickness and the number of lines connecting tokens, highlight the model's focus. For instance, denser clusters suggest concentrated attention, which could signal redundancy if such patterns do not contribute to diverse semantic understanding. These visualizations are critical for identifying and rectifying redundant attention, thereby improving the model's efficiency and accuracy. This process forms an integral part of optimizing language models for detection tasks, as explored in our experiments. 3 , clearly demonstrate the incremental improvements achieved by integrating these components into our framework. Initially, without any of these techniques, the model (original GLIP-T) achieves baseline performances of 41.9% on ODinW-13 and 26.0% on LVIS MiniVal. Implementing MIPT to the first 6 layers of BERT improves these scores to 43.2%  
conclusion: 
 In this paper, we present a novel strategy for enhancing open-set object detection by integrating large language models with multimodal inplace prompt tuning Framework. Our approach starts with the utilization of Jensen-Shannon Redundancy (JSR) to pinpoint and mitigate redundant attention patterns in language models adapted for detection tasks. This targeted fine-tuning improves computational efficiency and detection effectiveness. Further incorporation of MIPT, which enriches text prompts with visual clues, significantly boosts performance on the LVIS and ODinW datasets, demonstrating notable accuracy and adaptability improvements, especially in few-shot scenarios. Our findings highlight the transformative potential of combining metric-based redundancy evaluations with multimodal fine-tuning to optimize large language models for complex detection frameworks. Future work could expand this approach to include more modalities and applications, propelling further advancements in multimodal learning and object detection.  
 
 Appendix:  


Decision: The paper introduces a novel framework called Multimodal Inplace Prompt Tuning (MIPT) for enhancing open-set object detection by integrating large language models (LLMs) with detection systems. The primary focus is on addressing redundancy in attention patterns of LLMs when applied to object detection tasks. Reviewers appreciated the innovative introduction of the Jensen-Shannon Redundancy (JSR) metric and the significant performance improvements demonstrated through experiments. However, they noted the pseudocode's complexity, the need for more detailed ablation studies, and a lack of discussion on potential limitations. Overall, the reviewers leaned towards acceptance, recognizing the framework's efficiency and effectiveness while suggesting areas for further refinement.

Reviews: review: paper_summary: paper_summary: This paper investigates how to efficiently leverage LLMs for fine-tuning in order to enhance open-set object detection.  The contribution of this work is introducing LoRA to this task. It introduces a method to assess redundancy within language models using Jensen-Shannon Redundancy and proposes a multimodal inplace prompt tuning approach to improve prompt representations in detection systems. Experimental results show the  the effectiveness of the proposed framework.
strength_weakness: strength_weakness: 1. Compared to full training, the proposed method can improve detection performance by fine-tuning a small number of parameters through LoRA.
2. The writing in the paper is clear and easy to understand.
3. The experimental results show performance improvements.
limitations: limitations: 1. The pseudocode in the paper is overly complicated and lengthy. I suggest the authors use an abstract algorithm instead to facilitate research and analysis.
2. One of the main contributions of the paper is introducing LoRA, a fine-tuning method for LLMs, into multimodal scenarios to enhance the visual open-set object detection task. However, more analysis and ablation studies should be provided on the design for reducing fine-tuning parameters in this context to illustrate the value of this contribution.
3. Lack of discussion on the potential limitations of the proposed method, which is crucial as it contributes to a comprehensive understanding of the method.
paper_summary: paper_summary: The first to discover that directly fine-tuning language models in detection systems results in redundant attention patterns and leads to suboptimal prompt representations. Thus, in turn, falls to detect such fine-grained or long-tail categories, as shown in their figures. To this end, given the language model adaptation for detection that has been previously overlooked, we introduce a novel Multi-modal Inplace Prompt Tuning Framework (MIPT) for open-set object detection. In this paper, it first identify redundancy in attention distribution when integrating large language models into detection frameworks. It introduces the JSR metric to efficiently evaluate and pinpoint attention pattern redundancies in language models during fine-tuning for detection tasks.  The MIPT works by recalibrating prompt representations using visual cues for language models.
strength_weakness: strength_weakness: 1. This paper is the first to discover that directly fine-tuning language models in detection systems results in redundant attention patterns which is novel and interesting and it provides empirical experiments on showing this phenomenon. The introduction of the Jensen-Shannon Redundancy metric to measure the redundant attention is convincing.

2. The experiment results outcomes on GLIP-T, GLIP-L, and G-DINO-T showcase the remarkable efficacy of the MIPT framework, yielding substantial performance enhancements which indicates the good performance of MIPT.

3. The part for Visualization of Redundant Attention Mechanisms is very interesting and this contributes for further research.
limitations: limitations: In the figure 4 and figure 5, the author illustrates the redundancy evluation in different layer. Would there exist a way in visualizeing the MIPT can reduce this redundancy?
paper_summary: paper_summary: This paper introduces a novel framework called Multimodal Inplace Prompt Tuning (MIPT) for enhancing open-set object detection by integrating large language models (LLMs) with detection systems. The primary focus is on addressing the redundancy in attention patterns of LLMs when applied to object detection tasks. The authors propose a Jensen-Shannon Redundancy (JSR) metric to identify and mitigate redundant attention patterns. By incorporating visual clues into text prompts through MIPT, the study demonstrates significant improvements in detection performance across various benchmarks. Experimental results highlight substantial performance gains with minimal parameter tuning, showcasing the framework's efficiency and effectiveness.
strength_weakness: strength_weakness: Innovative Approach: The introduction of the Jensen-Shannon Redundancy (JSR) metric and Multimodal Inplace Prompt Tuning (MIPT) is innovative, providing a new method to enhance the alignment of language and visual features in detection tasks.

Performance Improvement: The experimental results demonstrate significant improvements in detection accuracy across multiple benchmarks, such as GLIP-L’s performance increase on ODinW-35 and LVIS.

Parameter Efficiency: The proposed method achieves notable performance gains with less than 1% of the model's parameters being fine-tuned, highlighting the efficiency of the approach.
limitations: limitations: While the paper includes ablation studies, more detailed experiments isolating the effects of individual components (such as different layers and heads in the LLM) would strengthen the claims.