Zero-shot Generalizable Incremental Learning for Vision-Language Object Detection

Authors: jieren  deng
haojian  zhang
kun  ding
jianhua  hu
xingxuan  zhang
yunkuan  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-27 05:10:13

Abstract: This paper presents Incremental Vision-Language Object Detection (IVLOD), a novel learning task designed to incrementally adapt pre-trained Vision-Language Object Detection Models (VLODMs) to various specialized domains, while simultaneously preserving their zero-shot generalization capabilities for the generalized domain. To address this new challenge, we present the Zero-interference Reparameterizable Adaptation (ZiRa), a novel method that introduces Zero-interference Loss and reparameterization techniques to tackle IVLOD without incurring a significant increase in memory usage. Comprehensive experiments on COCO and ODinW-13 datasets demonstrate that ZiRa effectively safeguards the zero-shot generalization ability of VLODMs while continuously adapting to new tasks. Specifically, after training on ODinW-13 datasets, ZiRa exhibits superior performance compared to CL-DETR and iDETR, boosting zero-shot generalizability by substantial $\textbf{13.91}$ and $\textbf{8.74}$ AP, respectively. Our code is available at https://github.com/JarintotionDin/ZiRaGroundingDINO.

Main paper: 
 introduction: 
 Object detection has achieved remarkable strides in recent years [2, 44] . However, most object detection models are typically trained to predict a predefined, closed set of categories [32, 2] , constraining their prospect in real-world applications. Recent research endeavors [12, 27, 45, 19, 22] focus on developing intelligent systems capable of detecting objects specified by natural language inputs, giving rise to the field of Vision-Language Object Detection (VLOD) (also named Open Vocabulary Object Detection), also referred to as Open Vocabulary Object Detection. In this framework, the models are termed Vision-Language Object Detection Models (VLODMs). By incorporating natural language inputs, VLODMs can recognize a much broader set of visual concepts beyond a fixed category set, resulting in exceptional zero-shot generalizability.Despite VLODMs' great zero-shot recognition ability in the general domain, VLODMs often exhibit suboptimal performance in more specialized domains, such as identifying aquatic organisms in aquariums or interpreting remote sensing images from aerial drones. In real-world scenarios, the necessity to adapt VLODMs to various unforeseen downstream tasks remains crucial in order to attain the desired accuracy, as highlighted in recent research [19, 45] .A straightforward idea is to adapt different individual VLODMs to different downstream tasks. However, a general agent often needs to simultaneously recognize objects from diverse tasks, each Figure 1 : Incremental Vision-Language Object Detection (IVLOD) aims to enhance VLODMs' performance across specialized domains via incremental learning, while also preserving their zero-shot generalization capability, enabling them to handle both known and unknown objects simultaneously and effectively. learned at distinct times. Moreover, in a dynamically changing environment, equipping VLODMs with the ability of incremental learning is paramount [20, 31] , considering that downstream tasks typically arrive in a streaming manner and incremental learning paradigm better aligns with the learning and cognition processes of a general intelligent agent.To endow VLODMs with the incremental learning ability, a naive idea is applying existing methods [10, 23] designed for detector on closed set directly. However, from our observations, it will weaken the excellent zero-shot generalization ability of VLODMs to detect unseen categories. To highlight the distinctiveness of incremental learning for VLODMs, we believe it is necessary to introduce a new task, Incremental Vision-Language Object Detection (IVLOD), which stresses the importance of maintaining the zero-shot generalization ability of VLODMs while adapting to various downstream tasks incrementally.To further demonstrate the uniqueness of the IVLOD task, we have compared three different variants of adapting VLODMs to multiple downstream tasks: zero-shot learning (i.e., no adaptation), conventional incremental learning (applying CL-DETR [23] to VLODM), and zero-shot generalizable incremental learning (i.e., the IVLOD task) in Fig. 1 . Unlike general incremental object detection (general IOD, ref. second column), IVLOD is characterized by preserving the original zero-shot performance of VLODM while also performing incremental learning. From this comparison, IVLOD faces two major challenges. One is the issue known as catastrophic forgetting [29] , where the performance on previously learned tasks may sharply decline when new tasks are introduced. The other challenge is to maintain the zero-shot generalizability of VLODMs while learning new downstream tasks.From a unified perspective, both challenges can be attributed to the forgetting problem. Technically, two kinds of methods can tackle this issue: selecting exemplar samples from past datasets for replaying [31, 23] or maintaining a duplicate model to facilitate knowledge distillation [20, 23] . In order to prevent forgetting, the replaying-based approach necessitates preserving enough examples to ensure the representativeness of the samples for replaying, but it is hard to do this with a limited-size memory, specifically when the model is pre-trained on a large-scale vision-language dataset. At the same time, current knowledge distillation techniques do not adequately focus on preventing the forgetting of pre-training knowledge, rendering them suboptimal for IVLOD tasks. Furthermore, they often need to store an entire model copy, which requires a large memory budget. Such storage demands are often untenable, particularly in scenarios that use resource-constrained edge devices.To effectively address IVLOD's challenges while improving memory efficiency, we introduce a novel approach named Zero-interference Reparameterizable Adaptation (ZiRa). ZiRa retains the parameters of the original model and introduces a parallel dual branch structure named Reparameterizable Dual Branch (RDB) designed for efficient tuning on downstream tasks. The RDB structure is reparameterizable, allowing the model to adapt to downstream tasks without increasing memory usage. Furthermore, the RDB structure serves a dual purpose-it achieves the branch labor division for downstream continual learning, protecting the learned knowledge from being excessively overwritten, and more crucially, it lays the structural foundation for the key novel element of ZiRa named Zero-interference Loss (ZiL).ZiL simultaneously penalizes the output norms of both the entire RDB and a high-learning-rate branch within the RDB, guiding the RDB to learn in a direction that protects both knowledge learned from pre-training and downstream tasks. The idea of ZiL comes from the following insight: During pre-training, the model is trained with visual-language inputs that contain significant noise, and the model has learned how to handle this noise. As a result, the pre-trained VLODMs have a certain degree of robustness to the input, and they can handle a certain range of interference without their performance being affected. ZiL ensures that the fine-tuned residual input of RDB is of small norm, thereby guaranteeing that the model's original performance. At the same time, for adaptation, only a small adjustment to the input is needed to learn new concepts. Although ZiL constrains the input norm of the RDB, it still allows the RDB to acquire sufficient downstream knowledge. As a result, ZiL effectively addresses the two challenges of IVLOD at the same time. Crucially, ZiRa accomplishes this without necessitating the duplication of the entire model or any exemplars, thereby ensuring efficient utilization of memory.Compared to existing methods [34, 23] , ZiRa leverages the advantages offered by VLODMs more fully through a principled design (i.e., the ZiL based on RDB), offering a superior and more effective solution without the need for excessive additional memory. The contributions of this paper can be summarized as follows:• We introduce Incremental Vision-Language Object Detection (IVLOD), a novel learning task that incrementally adapts VLODMs to multiple specialized domains while preserving zero-shot generalization ability.• We present a novel approach called ZiRa to tackle IVLOD challenges. Based on the reparameterizable parallel tuning structure of RDB, ZiRa introduces ZiL to simultaneously minimize interference with zero-shot performance and prevent forgetting on downstream tasks. Notably, ZiRa makes these achievements in a memory-efficient manner.• We conduct comprehensive experiments on COCO and ODinW-13 datasets, showcasing the effectiveness of our method. For instance, ZiRa outperforms CL-DETR [23] and iDETR [10] by a significant margin, making a 13.91 and 8.74 improvement in zero-shot AP on COCO, respectively, while offering competitive results on downstream tasks incremental learning. 
related work: 
 Vision-Language (or Open Vocabulary) Object Detection. Vision-Language (or Open Vocabulary) Object Detection Models combine natural language descriptions with object detection in images.Early research in this area primarily focused on grounding textual phrases to image regions, such as grounding referring expressions [28, 39, 4] and visual question answering [1, 25, 42] . More recent studies have developed unified and scalable approaches for simultaneous object detection and grounding tasks. For instance, GLIP [19] unifies phrase grounding and object detection tasks by treating object detection as context-free phrase grounding and treating phrase grounding as a contextualized object detection task. This approach enables the integration of multi-layered textual information into the detection model. Grounding DINO [22] , based on the DINO [44] , employs a DETR-like [2] architecture that uses a Transformer as the detector, which allows for unified processing of image and language data and offers the outstanding capacity to utilize large-scale datasets. Based on Grounding DINO, our work further addresses the newly introduced IVLOD tasks.Incremental Learning and Incremental Object Detection. Incremental learning, also known as continual learning or lifelong learning, tackles the challenge of learning new tasks without forgetting previous ones. Incremental learning has given rise to two primary strategies, memory replay, and knowledge distillation, to tackle its inherent challenges [20, 31, 3, 5, 38, 37, 40] . For example, Learning without Forgetting (LwF) [20] employs knowledge distillation to maintain performance on old tasks while learning new ones. Another notable approach, iCaRL [31] , combines exemplar memory and a nearest-mean-of-exemplars classification method for incremental learning. Incremental object detection, a specialized subfield within incremental learning, focuses on improving the adaptability of object detection models to new object categories while maintaining performance on previously learned categories [34, 17, 30, 41, 11, 10, 23] . ILOD [34] was the first to introduce incremental object detection. More recently, CL-DETR [23] effectively combined knowledge distillation and exemplar memory techniques to enhance incremental object detection using Transformer-based models. Our work distinguishes itself by performing incremental learning on VLODMs, which are more favorable for open-world problems. Besides the need to continually adapt across multiple specialized tasks, it is also important to preserve their zero-shot generalization ability. To the best of our knowledge, our work is the first to pose and tackle IVLOD tasks.Open-World Object Detection. Open-World Object Detection (OWOD) [16, 13, 26 ] also targets the detection of both seen and unseen objects, but it involves simultaneous learning of new objects and detecting unknown objects during the incremental learning process. This means that as new tasks are introduced, the model is trained to recognize new objects while also developing its ability to detect unknown objects. In contrast, Incremental Vision-Language Object Detection (IVLOD) pre-trains models to detect unknown objects before the incremental learning process begins. The focus of IVLOD during incremental learning is on preserving the model's ability to detect these unknown objects while adapting to new tasks. Given the superior zero-shot generalization capabilities of current vision-language models, we believe that starting with a pre-trained vision-language model for IVLOD is a more effective approach compared to the traditional OWOD paradigm.Reparameterization. Reparameterization techniques optimize model inference speed by transforming the training-time architecture into an equivalent, simplified form for inference [8, 9, 7] . RepVGG [9] exemplified this approach by employing 3×3 and 1×1 convolutional branches for training, which are streamlined to 3 × 3 convolutional branches and ReLU activations during inference. RepMLPNet [7] introduced "Locality Injection" to merge trained parameters of parallel convolutional kernels into fully connected layers, enhancing MLP-based model performance. Our work builds upon the Reparameterizable Dual Branch (RDB) concept initially proposed by [43] . The key innovations introduced in our approach include the use of differentiated learning rates for the High Learning Rate Branch and Low Learning Rate Branch, and the implementation of Zero-interference Loss (ZiL). These enhancements allow for a more effective balance between learning new tasks and preserving previously acquired knowledge, distinguishing our method from prior work. 
methodology: 
 In this section, we systematically present our approach. We begin with an overview (Sec. 3.1), and then introduce Zero-interference Reparameterizable Adaptation (ZiRa), which includes the Reparameterizable Dual Branch (RDB) (Sec. 3.2) and the Zero-interference Loss (ZiL) (Sec. 3.3).The framework of our model for addressing the IVLOD task is illustrated in Fig. 2 , with the selected VLODM being Grounding DINO [22] . Other VLODMs with a similar structure are also suitable. Specifically, the VLODM takes image features and category text prompts as inputs and produces class-agnostic proposals whose features are closely aligned with the text prompts. These text prompts are also refined to obtain text classification features by fusing the image features. Subsequently, the model classifies the class-agnostic proposals based on the cosine similarity between the visual features of proposals and the text classification features. These VLODMs are pre-trained on extensive grounding datasets annotated with detailed human language and object boxes, which equips them with great zero-shot recognition capabilities. To address the challenges of IVLOD, we propose ZiRa which is based on the structure of RDB and constrains the output of RDB by ZiL.Adaption on Both Language and Vision Sides. Fine-tuning the entire model is a common method for adapting the original model to downstream tasks. However, it has been demonstrated that tuning only a few newly introduced parameters in a pre-trained model is more effective [14, 15] , while full fine-tuning often leads to significant forgetting. In view of this, we introduce additional parallel branches on both the language and vision sides of the VLODM to adapt the model for sequential downstream tasks.As shown in Fig. 2 , these additional branches are termed the Reparameterizable Dual Branch (RDB).On the language side, we integrate an RDB alongside a pre-trained linear layer positioned between the detector and the language backbone network, aiming to learn the new high-level semantic concepts.On the visual side, we also connect a supplementary RDB in parallel with a pre-trained convolutional layer located between the visual backbone and the detector, aiming to learn the visual features of new classes. It's pivotal to adapt on both sides, as these two sides capture distinct structures of vital knowledge and the lack of either will lead to insufficient adaptation to downstream tasks. Dual Branch Structure within the RDB. Within the RDB, illustrated in Fig. 3 , our design features a dualbranch structure, comprising the Low-learning rate Branch (LLRB) and the High-learning rate Branch (HLRB). The HLRB is set to a high learning rate for rapid task adaptation, whereas the LLRB employs a more conservative learning rate. Specifically, the LLRB is set at η (0 < η < 1) times the learning rate of the HLRB. This differentiation in learning rates brings division of labor between LLRB and HLRB, especially when combined with our next strategy that reparameterizes the HLRB into the LLRB after learning each task. The LLRB's low learning rate can help maintain downstream task knowledge, while the HLRB's high learning rate can help swift adaptation to new tasks. This division protects the knowledge stored in the LLRB from being excessively overwritten when incrementally learning downstream tasks, achieving a better balance between stability the plasticity, while the single branch or naive dual branch structure can not take advantage of this division mechanism.Specifically, the collective output of the RDB x rdb , is formulated as:EQUATION)where x is the input and s is a learnable scaling factor.For the language side, the RDB is structured as two parallel linear layers, and its output x l rdb is expressed as:EQUATIONwhere W l hlrb and W l llrb represent the weights of HLRB and LLRB, respectively, while b l hlrb and b l llrb denote their corresponding biases.For the vision side, the RDB is structured as two parallel convolution layers, and its output x v rdb is illustrated as:EQUATIONHere, W v hlrb and W v llrb stand for the kernel of HLRB and LLRB, respectively, while b v hlrb and b v llrb represent their corresponding biases. ⊗ denotes the convolution operation.Reparameterization from the HLRB to the LLRB. With each new task, ZiRa resets the HLRB parameters to zero and, after completing learning on a new task, merges the trained parameters from the HLRB into the LLRB. This approach not only allows the LLRB to incrementally acquire new knowledge but also avoids the linear increase in the number of additional branches as the number of tasks increases, effectively managing memory consumption. Moreover, it offers a structural prerequisite for ZiL to prevent forgetting on downstream tasks by penalizing the output norm of HLRB, which will be detailed in the next section.On the language side, ZiRa combines the HLRB and LLRB into a consolidated LLRB with new weights W l hlrb • s + W l llrb and new biases b l hlrb • s + b l llrb , given by:EQUATIONSimilarly, on the vision side, the HLRB and LLRB are merged into a singular LLRB, as depicted by:EQUATIONwhereW v hlrb • s + W v llrb and b v hlrb • s + b vllrb can be considered as the new kernel and new biases. Single Branch for Inference. Sub-branches inside the RDB like the LLRB and HLRB can increase computational overhead. We efficiently address this by reparameterizing all sub-branches in RDB and the frozen pre-trained branch into a new one for inference. Since the frozen pre-trained branch is also a linear structure, after learning all the tasks, we can create a new branch that merges the frozen pre-trained branch and the RDB via reparameterization. During inference, we exclusively use this newly consolidated branch, but we still retain the parameters frozen pre-trained branch and the RDB for future learning. This approach significantly reduces the computational cost during inference, as only one branch is required for predictionProtect Zero-shot Generalizability with ZiL. Preventing performance degradation on the original pre-training domain can be viewed as preventing the VLODM from forgetting the original pre-training domain knowledge. Two common methods to avoid forgetting is to reserve some old samples for replaying or distill the current model from the previous ones. However, preserving old samples and storing the previously trained model requires significant additional memory. On the contrary, Zero-interference Loss (ZiL) can prevent forgetting in a memory-efficient manner. Specifically, ZiL penalizes the norm of the whole RDB's output (ref. Fig. 3 ), aiming at safeguarding the zero-shot performance of VLODM. The ZiL for the RDB, L rdb , is defined as follows:EQUATIONHere, L 1 corresponds to the L 1 norm.Why ZiL works. The effectiveness of ZiL can be explained as follows. First, the VLODM's input (What we fine-tuned in this paper) is robust to small-norm random perturbations, which have little impact on the model's performance. To verify this, we add Gaussian noise to the input of the pretrained VLODM's detector and observe the model's performance. As shown in Fig. 4 , the model's performance is not significantly affected by the addition of small-norm random noise to the input.Second, the ZiL ensures that the fine-tuned input of the VLODM has a small norm additional term.To further investigate the effectiveness of ZiL, we plot curves illustrating the evolution of the RDB's output norm throughout the IVLOD process, as shown in Fig. 5 . It is evident that training the RDB with ZiL leads to a substantial reduction in the output norm of the RDB on COCO, compared to training the RDB without ZiL. This preliminary observation strongly suggests that ZiL effectively preserves the zero-shot generalizability of VLODMs. Examining the evolution curves in Fig. 5 more closely, we observe that in the initial stages, ZiL plays a relatively minor role in curbing the growth of the RDB's output norm. However, as the process of accumulating new knowledge progresses, the influence of ZiL becomes more significant, leading to stronger constraint effects. This increase in ZiL's impact leads to a dynamic balance: the interference caused by integrating new knowledge is counteracted by the interference reduction achieved by ZiL, which is reflected in the fact that as the number of tasks increases, the norm of the RDB's output doesn't rise unboundedly but rather stabilizes after learning a certain number of tasks. This dynamic balance is crucial for maintaining the zero-shot generalizability of VLODMs.Prevent forgetting on downstream tasks with ZiL. The L rdb serves primarily to preserve the zero-shot generalization capability, leaving the challenge of downstream task forgetting unresolved. By exploiting the advantage of RDB, we address this challenge by further applying ZiL on HLRB, which is computed as:EQUATIONwhere s is a learnable scaling parameter.The L hlrb can effectively protect the knowledge that is learned from previous tasks and encompassed within the combined LLRB and the pre-trained branch. Specifically, given that the knowledge learned from the previous task has already been reparameterized into LLRB and LLRB learns at a lower learning rate, the total output of the branches previously trained can be regarded as LLRB(x) + PTB(x), where PTB is the pre-trained branch. The total output of the current branches is LLRB(x) + HLRB(x) • s + PTB(x), which has an additional component HLRB(x) • s compared to the output from the branches trained on the previous task. Hence, optimizing L hlrb equates to penalizing the discrepancy in the outputs between the models trained on the previous and the current task, effectively guiding the HLRB to learn new tasks without much downstream task forgetting.Total Optimization Objective. For a general VLODM, we continue to employ its original training loss. To prevent forgetting, we incorporate an additional loss term, L zil . Taking Grounding DINO as an example, the total loss function becomes:EQUATIONwhere L zil = L rdb + L hlrb is the proposed ZiL, λ is its loss weight, and L cls and L loc are the contrastive classification loss and localization loss of Grounding DINO. Throughout the training process, the entire original VLODM is frozen, except for the RDB. 
setup: 
 Datasets. We conduct our experiments on the COCO [21] datasets and the "Object Detection in the Wild (ODinW)" [18] benchmark. ODinW is a more challenging benchmark designed to test model performance under real-world scenarios. It comprises numerous sub-datasets from various domains for evaluation, such as Thermal (to detect objects in heat map images) and Aquarium (to detect different marine animals). Following GLIP [19] , we use ODinW-13 datasets, they are labeled as Ae (Aerial Maritime Drone), Aq (Aquarium), Co (Cottontail Rabbits), Eg (Egohands), Mu (Mushrooms), Pa (Packages), Pv (Pascal VOC), Pi (Pistols), Po (Pothole), Ra (Raccoon), Sh (Shellfish), Th (Thermal Dogs and People), Ve (Vehicles). The 13 sub-datasets of ODinW-13 are trained sequentially, one by one, and are tested after all sub-datasets have been trained.As COCO is a dataset within the realm of everyday life, we use zero-shot COCO performance as a metric of the model's zero-shot generalization ability across general domains. Additionally, we provide the average performance on the downstream ODinW-13 benchmark, with performance calculations made only after the model has been sequentially trained on all sub-datasets. We denote the zero-shot mAP on COCO as "ZCOCO" and the average mAP on ODinW-13 downstream tasks as "Avg".Implementation Details. In this paper, unless otherwise specified, we use the Grounding DINO [22] with the Swin-Tiny [24] backbone as the default original VLODM in our experiments. This Grounding DINO is pre-trained on Objects365 [33] , GoldG [19] , and Cap4M [19] . We also use OV-DINO [35] with the Swin-Tiny backbone as an additional validation to demonstrate ZiRa's ability to generalize effectively across different DETR-based architectures. This OV-DINO model is pre-trained on the Objects365 [33] , GoldG [19] , and Cap1M [35] datasets. The learning order of the ODinW-13 sub-datasets is randomly shuffled. All results are obtained after running experiments three times with different random seeds. The hyperparameter experiments are also conducted on the ODinW-13 dataset and COCO, and the results are shown in the appendix.Our proposed method is implemented with PyTorch and trained on two Nvidia RTX 3090 GPUs. Each downstream task is trained for a total of two epochs with a batch size of 2. For Grounding DINO, we employ an initial learning rate of 10 -3 , which decays to 0.1 times the original value after the first epoch to ensure effective convergence. For OV-DINO, we employ an initial learning rate of 10 -4 , which also decays to 0.1 times the original value after the first epoch to ensure effective convergence. AdamW is used as the optimizer, and the weight decay is 10 -4 . At the beginning of IVLOD, we initialize the parameters of the total RDB as zero. In our evaluation, we compare ZiRa with existing Incremental Object Detection (IOD) methods, with a specific focus on methods based on Detection Transformers (DETR) [2] , as our approach falls into this category. The baseline methods we consider include:• TFA [36] : A classical linear-probing-based incremental few-shot object detection baseline.• OW-DETR [13] : An open-world object detection approach based on DETR, primarily addressing forgetting through exemplar replaying, which necessitates additional memory to store exemplars.• CL-DETR [23] : A DETR-based method employing refined knowledge distillation and exemplar replaying. This method requires a complete model copy and exemplars, incurring significant extra memory costs.• iDETR [10] : An incremental few-shot object detection method specifically focusing on tuning the projection layer of DETR-like detectors via knowledge distillation and selfsupervision.• Adapting-tuning (AT) [14] : A parameter-efficient adaptation method that has demonstrated its effectiveness in incremental few-shot object detection [6] .We first compare the above methods implemented on the Grounding DINO model. Experiments are conducted under both few-shot and full-shot settings. Since CL-DETR and OW-DETR are not designed for few-shot IOD, we only compare them under the full-shot setting. The results presented in Tab. 1 demonstrate that ZiRa consistently outperforms existing IOD methods in terms of "Avg" on downstream tasks. Moreover, ZiRa exhibits remarkable performance in preserving the zero-shot generalization ability of VLODMs under both the few-shot and full-shot settings. In particular, ZiRa surpasses CL-DETR and iDETR by substantial margins, with improvements of 13.91 and 8.74 AP under the full-shot setting in terms of "ZCOCO", respectively.We then compare ZiRa with existing methods implemented on the OV-DINO model. The results in Tab. 2 show that ZiRa outperforms existing IOD methods in terms of "Avg" on downstream tasks. ZiRa also demonstrates excellent performance in preserving the zero-shot generalization ability of VLODMs under the full-shot setting. Specifically, ZiRa surpasses CL-DETR and iDETR by significant margins, with improvements of 14.55 and 11.36 AP in the full-shot setting for "ZCOCO," respectively.These results emphasize ZiRa's effectiveness in maintaining the zero-shot AP and its potential as a superior solution for IOD. Distinctively, ZiRa, unlike CL-DETR and OW-DETR, requires minimal extra memory for a few branches without storing image exemplars and the model copy, making it more memory-efficient than other IOD methods. Main Components of ZiRa. Our approach combines the ZiL and the RDB to counteract forgetting in both the general domain and downstream tasks. In this study, we analyze the impact of these components. We utilize "Rep+" to denote whether to reparameterize the HLRB into LLRB after each task as well as using differentiated learning rates. Please note that even when we do not use "Rep+", we still retain the dual-branch structure like RDB. ZiL encompasses the RDB's ZiL (L rdb ) and HLRB's ZiL (L hlrb ), here we separate them and study their impact separately. We also evaluate performance with the "hAP" metric, which is the harmonic mean of "ZCOCO" and "Avg".The results in Tab. 3 provide valuable insights. First, the comparison between the first and second rows demonstrates that "Rep+" can mitigate forgetting on both pre-training and downstream tasks. This effect comes from the labor division between branches that are built on differentiated learning rates and reparameterization. Second, the comparison between the first row and third rows highlights that ZiL (L rdb + L hlrb ) significantly enhances the "ZCOCO". However, it cannot address forgetting on downstream tasks without "Rep+", and it even reduces the "Avg" since it also limits the plasticity of the RDB. The best results lie in the fourth row, underscoring that combining "Rep+" and ZiL can effectively mitigate forgetting on both pre-training and downstream tasks to a greater extent. Conversely, the results in the last row illustrate that using (L rdb ) alone does not optimally address forgetting on downstream tasks. Comparing the second row and the last row, we can find that L hlrb can improve both "ZCOCO" and "Avg", showcasing that L hlrb can mitigate forgetting both on downstream tasks and pre-training, but without L rdb , using L hlrb alone can not achieve the best "hAP".Learning on Different Modalities with ZiRa. We carried out a series of experiments to investigate the effects of learning only on the vision side using parallel convolution layers (indicated as "V") and only on the language side using parallel linear layers (denoted as "L"). The results in Tab. 4 demonstrate that ZiRa can function independently on either the language or vision side, and it can also effectively function when simultaneously applied to both modalities, highlighting the generality of ZiRa's impact. Since the language and vision side learning acquires knowledge in distinct structures, neglecting either can hinder optimal learning. Consequently, simultaneously learning on both sides can outperform individual tuning on either the vision or language sides when employing ZiRa to avoid forgetting. Branch Structure. To study the impact of the additional branch structure introduced in VOLDM, we compared the results of introducing a single branch (denoted as SB), dual branches (denoted as DB), and RDB (DB + Rep+). Besides RDB, both the output norm of SB and DB are also penalized with ZiL. As we can see in Tab. 5, the SB structure, due to its inability to introduce inter-branch division of labor, falls short in its ability to prevent forgetting in downstream tasks compared to the RDB. Moreover, we can see that merely introducing DB with ZiL cannot prevent forgetting on downstream tasks, on the contrary, it brings worse forgetting due to increased plasticity. RDB not only incorporates the mechanism of branch labor division but also fully utilizes l hlrb to learn to minimize the interference of HLRB, thereby more effectively preventing the forgetting on downstream tasks. 
conclusion: 
 This paper presents a novel learning task, Incremental Vision-Language Object Detection (IVLOD), which aims to continually adapt Vision-Language Object Detection Models (VLODMs) to multiple specialized domains while preserving VLODMs' zero-shot generalization ability. To solve this new task in a memory-efficient way, we introduce Zero-interference Reparameterizable Adaptation (ZiRa). ZiRa inserts Reparameterizable Dual Branche (RDB) on the both language side and vision side of the VLODM and constrains the RDB by Zero-interference Loss (ZiL) to protect the original generalizability of VLODMs and prevent forgetting on downstream tasks at the same time. Notably, ZiRa eliminates the need for saving the entire model copy for distillation or maintaining exemplars for replaying, which makes it a highly memory-efficient method. Extensive experiments conducted on the COCO and ODinW-13 datasets showcase the superiority of ZiRa for IVLOD.Limitations. While ZiRa has demonstrated its effectiveness in IVLOD tasks, there are still some limitations. First, the current implementation of ZiRa is based on the DETR architecture, which may not be optimal for all VLODMs. Second, the current ZiRa method is designed for the IVLOD task, and its generalization to other incremental learning tasks remains to be explored. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: The paper specifies all the training and test details.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The paper does not report error bars or statistical significance.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [No] Justification: Each experiment will take 3 hours on 2 Nvidia RTX 3090 GPUs.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA]Justification: The paper does not use existing assets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Impact of λ. The hyperparameter λ modulates the influence of ZiL within the overall loss function.In this study, we investigate the impact of varying values of λ on model performance, as presented in Tab. 6. The results show that a small λ value (e.g., 0.01) tends to cause the model to overfit new tasks, resulting in reduced performance in the general domain. Conversely, a large λ value (e.g., 1.0) overly constrains the model, hindering its adaptation to downstream tasks. The optimal λ value falls between these extremes, with the highest hAP observed when λ = 0.05, which strikes a balance between "ZCOCO" and "Avg". Differentiated Learning Rate. We explore the effects of hyperparameter η. The results are detailed in Tab. 7. On the one hand, setting the LLRB's learning rate as zero equivalently freezes it. Although this results in the highest "Avg" by preserving downstream task knowledge, the "ZCOCO" performance suffers considerably. This degradation is because HLRB, which has a limited parameter size, endures all the optimization pressure from L zil and L cls +L loc . Despite HLRB's attempts to find a harmonious optimization path, it tends to emphasize downstream task learning over maintaining initial zero-shot generalizability, especially when the predominant loss L cls + L loc outweighs L zil . On the other hand, equating the LLRB's learning rate to the HLRB's is equivalent to creating two HLRBs inside the RDB. The enhanced plasticity that comes from the two-HLRB structure makes it challenging for ZiL to prevent it from gravitating toward directions detrimental to zero-shot generalizability, leading to diminished "ZCOCO" scores. Moreover, a significantly high LLRB learning rate can also precipitate a swift decline in downstream task knowledge retention. Optimal η lies between these two cases, like 0.1 or 0.2, striking a balance between efficient new task adaptation and zero-shot generalizability protection. Impact of Different Initial Values of the Scaling Factor. The scaling factor s is a learnable parameter that affects the IVLOD's performance. Its initial value could shape the model's efficiency and adaptation capabilities. As documented in Tab. 8, contrary to initial expectations, a balanced scaling factor for both language and vision components, set at 1.00, does not lead to the best outcomes. Instead, an asymmetrical approach where the vision component's scaling factor is reduced to 0.10 while maintaining the language component's scaling factor at 1.00, or vice versa, appears to enhance overall performance metrics, including "ZCOCO", "Avg", and "hAP". Remarkably, the configuration where both language and vision scaling factors are set to 0.10 achieves superior results, yielding a slight improvement in "ZCOCO" and the highest "hAP" score among the tested scenarios. This suggests that a more conservative initial scaling factor may encourage the model to rely more on the base knowledge from pre-training, thus improving its generalization before adapting further through incremental learning. By default, we utilize the L1 norm to regulate ZiL. However, the choice of norm type for ZiL significantly impacts performance. As demonstrated in Tab. 9, L2 excels in ZCOCO, L1 strikes a balance between zero-shot and downstream performance, and Smooth L1 performs exceptionally well in downstream tasks while maintaining competitive zero-shot performance, even outperforming the L1 norm. Regardless of the norm type used, ZiRa remains effective, showcasing its versatility across different norm types. To more vividly showcase the effectiveness of the proposed ZiRa method, we also performed a visualization experiment. As shown in Fig. 6 , using the pre-trained Grounding DINO directly to leverage its zero-shot detection capabilities allows for detecting many unseen objects, but fails to detect some. Upon incrementally fine-tuning with iDETR using training samples (with five classes per phase and one sample per class), the model will be able to recognize objects that are undetected by zero-shot detection. However, some objects that can be initially recognized by zero-shot detection are no longer detected. In contrast, after incremental fine-tuning with ZiRa, the images reveal that both familiar and novel objects are detected. This demonstrates ZiRa's dual ability to prevent forgetting both pre-trained knowledge and information from downstream tasks. Justification: The code is released at https://github.com/JarintotionDin/ ZiRaGroundingDINO. Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.Justification: The authors have reviewed the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Justification: There is no societal impact of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: This paper was reviewed by four experts in the field. The authors' rebuttal resolved most of the concerns. Reviewers liked the new task proposed by the paper and unanimously agreed to accept the paper. The reviewers did raise some valuable suggestions in the discussion that should be incorporated in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes to the best of their ability.

Why not higher: The reviewers still have shared concerns about the paper's presentation (positioning among related works) and discussions about related works.

Reviews: review: paper_summary: paper_summary: This paper deals with the object detection problem in a zero-shot and incremental learning setting. Specifically, Incremental Vision-Language Object Detection (IVLOD) task is proposed to incrementally adapt the pre-trained Vision-Language Object Detection Models (VLODMs) to various specialized domain.   
The technical design is Zero-interferences Reparametrizable Adaptation (ZiRA), which includes low learning rate and high learning rate adaptation branches for both vision and language encoder. In addition, an L1-norm is applied on top of the adaptation branch outputs. 

Experiments are conducted on the Zero-shot COCO (ZCOCO) and ODinW-13 benchmarks. A series of baselines such as TFA, OW-DETR, CL-DETR, Adapting-tuning (AT), and iDETR were compared. Ablation study on modules, vision or language finetuning were also conducted.
strength_weakness: strength_weakness: ### 1. The new task of IVLOD is not well desribed and compared with existing one 
- Incremental Vision-Language Object Detection (IVLOD) is claimed to be a new task. However, it is unclear what are the differences between this new task and existing works such as CL-DETR, Ground DINO, iDETR etc. 
- How is the language specially used in IVLOD compared with existing method? 
- Claimed as the first contribution, but no separate detailed description/subsection to compare IVLOD with existing task setting, either verbally or visually. Fig. 1 only compares with the final performance, which is less intuitive. 
### 2. The main contribution is the different learning rates 
- For the Reparameterizable Dual Branch (RDB), although the name is special, the module design is no much different with existing approaches such as LoRA, Adapter, [6], etc. 
- The structure design is not quite novel, but leveraging different learning rates can be a novel point. 
- It is encoraged to have a better survey of relevant adaptation-based methods for incremental learning. 
### 3. Experimental section have flaws
- ODinW was adopted, but only 13 datasets were selected. How are those 13 datasets selected? Other methods used much more datasets for evaluation, e.g. 35. 
- In Table 1, AT was not used for Full evaluation. CL-DETR/OW-DETR were not used for lower shots evaluation. What is the reason? 
- The prompt in Fig. 6 looks strange. How is the prompt leveraged by the proposed model? 
- In Table 3, why the model performance on zero-shot COCO underperforms Vanilla one by 1.31%? 
- There are many hyper-parameters such as $\lambda$ for loss, $\eta$ for learning rate, scaling $s$. How are those parameters selected and tuned? It seems in Table 5 and 6 that the hyper-parameters matters. 
- In Table 8, how about no norm is adopted? 
### 4. Other minors
- 1st not 1-th in Fig. 1
- Incorrect usage of double quotes in this paper, e.g.  “Locality Injection" 
- Line 209 lacks a period
questions: questions: Please refer to the above Weaknesses for questions.
limitations: limitations: N/A
paper_summary: paper_summary: This paper proposes the new problem of incremental visual-language object detection (IVLOD), which aims to preserve zero-shot generalization performance of VLMs, while also adapting to new concepts over time. Authors address IVLOD by proposing the zero-interference reparameterizable adaptation (ZiRa), a light weight branch added to the text and image encoders of VLMs to address this task. Authors demonstrate that their proposed approach outperforms prior work by a considerable margin.
strength_weakness: strength_weakness: Differences in Baseline Pre-Training. Although authors benchmark their approach against relevant prior work, many of these methods are not trained on the same scale of pre-training data as GroundingDINO. Given that prior works typically operate in a closed-world, and are pre-trained on much smaller datasets, it is unclear how to make an "apples-to-apples" comparison between methods. One strategy might be to reimplement prior works (e.g. TFA and iDETR) using GroundingDINO.

Demonstrate ZiRa with Other Architectures. Although authors claim that their proposed approach can work with any DETR based architecture, they only showcase its performance with GroundingDINO. Given that this work establishes a new problem, it would be useful to extensively evaluate different architectures to provide a "lay of the land".
questions: questions: Impact of Pre-Training Data on IVLOD Performance. The performance of VLMs is significantly impacted by the data used for pre-training. How might the specific pre-training data used impact the effectiveness of incremental learning? 

Adapting to a Large Number of Tasks. As shown in Figure 5, the L1 norm steadily increases as the number of downstream tasks increases. Does this suggests that the proposed approach does not scale well to a large number of tasks?

Adapting to More Than One Class at a Time. Although each class is considered a new "task", one may want to adapt to multiple classes at once. How would the proposed approach perform when learning more than one class at a time? 

Runtime and Memory Usage. A key benefit of the proposed approach is that adapting to a new task only marginally increases runtime and memory usage. It would be useful to quantify this such that future works can benchmark on this axis as well. 

Writing Quality. Although the approach is clear, the grammar and writing quality hinders comprehension. For example "inputted" is not a word, and "input end" is a confusing phrase. I would encourage authors to further polish the manuscript. In addition, authors should include more descriptive captions in their tables. Notably, I don't believe the full class names in Table 1, 2 (e.g. Ae, Aq, Co) are provided.
limitations: limitations: Yes, authors highlight that the current implementation is based on the DETR architecture, which may not work for all VLODMs. In addition, authors state that ZiRa is designed for IVLOD, and should be evaluated in the context of other incremental learning tasks.
paper_summary: paper_summary: In order to extend the application of VLODM in a broader domain, the authors propose to combine incremental learning and zero-shot generalization to solve the problem. However, the widespread catastrophic forgetting and maintaining zero-shot generalization capability in incremental learning are two important issues to be considered. To address the challenges in incremental learning, the authors propose to construct a Zero-interference Reparameterizable Adaptation using Reparameterizable Dual Branch.
strength_weakness: strength_weakness: 1. The authors do not highlight the need for this work and its similarities and differences with other work in the introduction and related work.
2. Although the authors have designed an innovative Reparameterizable Dual Branch structure, it is not very interpretable for the problem to be solved.
3. Although the authors give very rich experimental results in Table 1, the description of the metrics. is not clear enough. What is " the zero-shot performance on COCO as “ZCOCO" "? Is it mAP on the base class or the novel class, or all classes? Is it mAP or AP50, or AP75?
questions: questions: 1. The authors introduce the names VLOD and VLODMs in the introduction and cite the literature [11, 17, 20, 24, 39] as the source for the derivation of these names. In these works, tasks such as open vocabulary/open set object detection, grounding, etc. using Vision-Language Model are included. Is the VLOD proposed by the authors consistent with or similar to the task setting of open vocabulary/open set object detection? If it is consistent, why use the term VLOD? If it is similar, the difference between them should be highlighted in the introduction.
2. In GLIP, the authors give the statement, i.e., " without seeing any images in COCO during pre-training". However, in this work, the authors do not discuss or clarify this. Considering the task setting of zero-shot generalization, how do the authors ensure that the images/objects in the test set has not appeared in the pre-training phase?
3. Based on question 1, where the author claims "Our work distinguishes itself by performing incremental learning on VLODMs, which are more favorable for open-world problems. " in related work, is the author's work incremental or zero-shot learning, or a combination of both? What are the similarities and differences between that work and the open-world object detection?
4. How should the scaling factor “s” be determined in Equation 1? The Reparameterizable Dual Branch structure process contains both HLRB and LLRB, does the size of the scaling factor imply which weight is dominant? If HLRB is dominant, how to ensure that no knowledge forgetting occurs? If LLRB is dominant, how can generalization ability be ensured? Table 7 does not allow a reasonable interpretation of this question.
5. The authors give Figure 4 and analyze it to prove the validity of ZiL, but how to define " small-norm " and why do the authors claim " the model’s performance is not significantly affected by the addition of small-norm random noise to the input " when “The average AP on COCO” drops more than 30%?
6. The authors use the model structure of the Grounding DINO work, in which text representation is an important part of the work. The text template used by the authors is quite different from the one used in Grounding DINO and the authors have not analyzed or explained it, why?
limitations: limitations: See Waeknesses and Questions.
paper_summary: paper_summary: This paper presents Incremental Vision-Language Object Detection (IVLOD), a novel learning task designed to incrementally adapt pre-trained Vision-Language Object Detection Models (VLODMs) to various specialized domains, while simultaneously preserving their zero-shot generalization capabilities for the generalized domain. To this end, Zero-interference Reparameterizable Adaptation (ZiRa) is proposed to tackle IVLOD without incurring a significant increase in memory usage. Experiments on COCO and ODinW-13 datasets demonstrate that ZiRa effectively safeguards the zero-shot generalization ability of VLODMs while continuously adapting to new tasks.
strength_weakness: strength_weakness: 1. The key contribution of this paper is RDB and ZiL. However, RDB has been proposed by [1]. Therefore, I would owe the novelty to the differentiated learning rate and ZiL (basically L1 norm). From this point of view, the novelty is somewhat incremental. Importantly, the authors should cite [1] and highlight the distinction.

2. The paper only evaluates the general zero-shot performance after incremental learning. Is zero-shot specific-domain inference possible after learning on few examples in this specific domain?

3. It's suggested to include more comprehensive incremental object detection literature, including but not limited to [2-3].


[1] Zhang, Chang-Bin, et al. "Representation compensation networks for continual semantic segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
[2] Qiao, Limeng, et al. "Defrcn: Decoupled faster r-cnn for few-shot object detection." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.
[3] Yang, Ze, et al. "Efficient few-shot object detection via knowledge inheritance." IEEE Transactions on Image Processing 32 (2022): 321-334.

Minors:
L178 between stability and plasticity
L342 typo VOLDM
questions: questions: 1. Is TFA also pre-trained on Objects365 [30], GoldG [17], and Cap4M? I have some doubts about the TFA ZCOCO performance in Table 1 if it's pretrained on COCO. To my best knowledge, TFA only finetunes the last classifier layer for the incremental steps and has demonstrated strong performance on the base classes.

2. How is general IOD (e.g. TFA) tested on Zero-shot COCO? General IOD generally does not support zero-shot inference. Is there any adaptation to achieve it?

3. What learning rate is applied to w/ Rep+ (the first line of Table 2)? Is it the high or low one?
limitations: limitations: No negative societal impact is expected.

Comments: title: Official Comment by Reviewer 8RRL
comment: We’d like to express our gratitude for the reviewer’s invaluable assistance in improving our work.
The authors' rebuttal has addressed my concerns. I raise the score to 6.
Thanks for the further clarification.   
With the detailed explanation, I can understand the difference with other setting of open-world/open-vocabulary/open-set. A gentle suggestion would be illustrate this with an intuitive and motivative Figure 1. This will also improve the impact of this paper and the focused new setting.
Thank you very much for your positive feedback and recommendation for acceptance. We appreciate your recognition of the contributions of our work, particularly the introduction of the new task of Incremental Vision-Language Object Detection (IVLOD) and the establishment of extensive baselines. 

We also noticed that while you have recommended our paper for acceptance, the final score was not mentioned. We would be grateful if you could kindly update your review to reflect this. We have taken into account all the suggestions from you and other reviewers and incorporated the relevant background information in the revised manuscript (ref. response to Reviewer NuJX).

Thank you once again for your thoughtful review and support.
Thank you for increasing your score. We appreciate the opportunity to clarify the differences between Incremental Vision-Language Object Detection (IVLOD) and other related tasks, including open vocabulary/open set object detection and open-world object detection (OWOD).

1. **IVLOD vs. Open Vocabulary/Open Set Object Detection:**
   - **Open Vocabulary/Open Set Object Detection** focuses on detecting both seen and unseen objects by leveraging pre-trained models, typically without the need for incremental learning. These models are designed to recognize and localize objects that were not explicitly included in the training data but can be identified through language prompts or other contextual information.
   - In contrast, **IVLOD** not only aims to detect unseen objects but also involves the **incremental adaptation of Vision-Language Object Detection Models (VLODMs)** to new tasks or domains while preserving their zero-shot generalization capability. Unlike open vocabulary detection, which does not involve updating the model after the initial training phase, IVLOD requires the model to continuously adapt to new tasks without forgetting previously learned knowledge.

2. **IVLOD vs. Open-World Object Detection (OWOD):**
   - **OWOD** also aims to detect both seen and unseen objects, but it involves the simultaneous learning of new objects and the detection of unknown objects during the incremental learning process. This means that as new tasks are introduced, the model is trained to detect new objects and maintain its ability to recognize unknown objects in real time.
   - **IVLOD**, on the other hand, **pre-trains models to detect unknown objects before the incremental learning process begins**. The focus during incremental learning is on maintaining the model’s ability to detect these pre-trained unknown objects while adapting to new tasks. Additionally, while OWOD does not further classify unknown objects beyond detection, IVLOD leverages VLODMs to classify unknown objects using language prompts, enabling more precise identification and categorization of previously unseen objects.

   - Given the superior zero-shot generalization capabilities of current vision-language models, we believe that starting with a pre-trained vision-language model for IVLOD represents a more effective approach compared to the traditional OWOD paradigm, which involves incrementally learning new objects while simultaneously learning open-set detection.

In summary, while there are similarities between IVLOD and both open vocabulary/open set object detection and OWOD, IVLOD introduces the unique challenge of **incrementally adapting models** to new tasks while **preserving their zero-shot capabilities**, distinguishing it from these related tasks. We will ensure that these distinctions are clearly articulated in the revised manuscript to provide a comprehensive understanding of our contributions.
The rebuttal addresses some of my concerns regarding a new task and contribution, part of the experimental questions are addressed.   I would like to encourage the authors to add the new task discussion in the revision.   
After taking into account the rebuttal and considering other reviewers's comments, I would like to increase my score.
Thank you for your positive feedback and recommendation for acceptance. We appreciate your recognition of our efforts in proposing a new problem and establishing extensive baselines. Additionally, we will carefully consider the suggestions made by other reviewers and incorporate the relevant background information to further strengthen the manuscript.
Thank you for your thorough review and positive feedback. We are pleased that our work on Incremental Vision-Language Object Detection (IVLOD) and the proposed differentiated learning rates and Zero-interference Loss (ZiL) have been well-received. We appreciate your suggestion to include literature on continual learning. We will ensure that these references are included in the revised manuscript to provide a more comprehensive background and context for our work.
The authors' rebuttal has addressed my concerns. I believe this work should be credited for introducing a new and practical task, i.e., incremental vision-language object detection (IVLOD), where the goal is to continuously adapt VLODMs to various unforeseen downstream tasks while not sacrificing the general zero-shot capability. Plus, the proposed differentiated learning rates and zero-interference loss (ZiL) are well-tailored for this specific task. One small suggestion is to also include continual learning literature, for instance, but not limited to [1-3] (continual object detection to be added as well), since the proposed task is related to the continual learning concept. Overall, I finalize my rating to **7 Accept**.


[1] Cermelli, Fabio, et al. "Modeling the background for incremental learning in semantic segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

[2] Zhang, Chang-Bin, et al. "Representation compensation networks for continual semantic segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[3] Yang, Ze, et al. "Label-guided knowledge distillation for continual semantic segmentation on 2d images and 3d point clouds." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
Authors have sufficiently addressed my questions. I recommend this paper should be accepted. Although other reviewers point out that there is limited novelty in the method, I think that this paper should get credit for proposing a new problem and establishing extensive baselines. I would encourage authors to add relevant background information suggested by other reviewers.