Zero-shot Generalizable Incremental Learning for Vision-Language Object Detection
Authors: Jieren Deng
Haojian Zhang
Kun Ding
Jianhua Hu
Xingxuan Zhang
Yunkuan Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Vision-language object detection
incremental learning
Primary area: machine_vision
TL;DR: This paper presents a novel method, Zero-interference Reparameterizable Adaptation (ZiRa), for incremental vision-language object detection that efficiently adapts models to new tasks while preserving zero-shot generalization capabilities.

Abstract: This paper presents Incremental Vision-Language Object Detection (IVLOD), a novel learning task designed to incrementally adapt pre-trained Vision-Language Object Detection Models (VLODMs) to various specialized domains, while simultaneously preserving their zero-shot generalization capabilities for the generalized domain. To address this new challenge, we present the Zero-interference Reparameterizable Adaptation (ZiRa), a novel method that introduces Zero-interference Loss and reparameterization techniques to tackle IVLOD without incurring a significant increase in memory usage. Comprehensive experiments on COCO and ODinW-13 datasets demonstrate that ZiRa effectively safeguards the zero-shot generalization ability of VLODMs while continuously adapting to new tasks. Specifically, after training on ODinW-13 datasets, ZiRa exhibits superior performance compared to CL-DETR and iDETR, boosting zero-shot generalizability by substantial $\textbf{13.91}$ and $\textbf{8.74}$ AP, respectively. Our code is available at https://github.com/JarintotionDin/ZiRaGroundingDINO.

Zero-shot Generalizable Incremental Learning for Vision-Language Object Detection
Jieren Dengh?
Haojian Zhang +1 Kun Ding', Jianhua Hu', Xingxuan Zhang', and Yunkuan Wang'
'nstittte of Automation, Chinese Academy Sciences (CAS)
(dengjieren2019, 1angiiren201.jjaahuuaauu jianhua.hu, zhanghaojian2014, yunkuan. wang @ iaa.ac.cn
School of Artificial Intelligence, University of Chinese Academy of Science, UCAS
Shanggaii Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine zhangxingxuan @ sjtt..ddu.cn
Abstract
This paper presents Incremental Vision-Language Object Detection (IVLOD), a novel learning task designed to incrementally adapt pre-trained VisguneL Object Detection Models (VLODMs) to various specialized domains, while simul- taneously preserving their zero-shot generalization capabilities for the generalized domain. To address this new challenge, present the Zero-interference Repa- rameterizable Adaptation (ZiRa), novel method that introduces Zero-interference Loss and reparamcterization techniques to tackle IVLOD without incurring significant increase in memory usage. Comprehensive experiments on COCO and ODinW-13 datasets demonstrate that ZiRa effectively safeguards the zero- shot generalization ability VLODMs while continuously adapting to new tasks. Specifically, after training ODin W-13 datasets. ZiRa exhibits superior per- formance compared to CL-DETR and iDETR, boosting zero-shot generalizabil- ity by substantial 13.91 and 8.74 AP, respectively. Our code is available https: thub. com/Jarintotionniiiii.i wtt..onnnn.....aaaaannnnnnnn.l.
Introduction
Object detection has achieved remarkable strides in recent years [2, 44). However, most object detec- tion models are typically trained predict a predefined, closed set of categories [32, 2], constraining their prospect in real-world applications. Recent research endeavors [12, 27, 45, 19, 22] focus developing intelligent systems capable of detecting objects specified by natural language inputs, giving rise to the field of Vision-Language Object Detection (VLOD) (also named Open Vocabulary Object Detection), also referred to as Open Vocabulary Object Detection. In this framework, the models are termed Vision-Language Object Detection Models (VLODMs). By incorporating natural language inputs, VLODMs can recognize a much broader set of visual concepts beyond a fixed category set, resulting in exceptional zero-shot generalizability.
Despite VI.OMMs' great zero-shot recognition ability in the general domain, VLODMs often exhibit suboptimal performance in more specialized domains, such as identifying aquatic organisms in aquariums or interpreting remote sensing images from aerial drones. real-world scenarios, the necessity to adapt VLODMs to various unforeseen downstream tasks remains crucial in order to attain the desired accuracy, as highlighted in recent research [19, 45].
A straightforward idea is to adapt different individual VLODMs to different downstream tasks. However, general agent often needs to simultaneously recognize objects from diverse tasks, each
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

1-th Task: Rabbits
Zero-shot
General IOD
IVLOD
1-th Task AP: 64.72
1-th Task AP: 72.01
1-th Task AP: 72,55
2-nd Task: Aquarium
Zero-shot
General IOD
IVLOD
1-th Task AP: 64.72
1-th Task AP: 71.98
1-th Task AP: 71.72
2-nd Task AP: 20.75
2-nd Task AP: 46.1/
2-nd Task AP: 47.73
Zero-shot COCO
Zero-shot
General 1OD
IVLOD
Zero-shot AP: 47.37
Zero-shot AP: 32.15
Zero-shot AP: 46.06
Figure I: Incremental Vision- AssonnLanguugg Object Detection (IVLOD) aims to enhance VLODMs' per- formance across specialized domains via incremental learning, while also preserving their zero-shot generalization capability, enabling them to handle both known and unknown objects simultaneously and effectively.
learned at distinct times. Moreover, in dynamically changing environment, equipping VLODMs with the ability of incremental learning is paramount [20, 31], considering that downstream tasks typically arrive in streaming manner and incremental learning paradigm better aligns with the learning and cognition processes of general intelligent agent.
To endow VLODMs with the incremental learning ability, naive idea is applying existing methods [10, 23] designed for detector on closed set directly. However, from our observations, will weaken the excellent zero-shot generalization ability of VLODMs to detect unseen categories. To highlight the distinctiveness of incremental learning for VLODMs, we believe it is necessary to introduce a new task, Incremental Fision-Languag Object Detection (IVLOD), which stresses the importance of maintaining the zero-shot generalization ability VLODMs while adapting to various downstream tasks incrementally.
To further demonstrate the uniqueness of the IVLOD task, we have compared three different variants of adapting VLODMs to multiple downstream tasks: zero-shot learning (i.e., no adaptation), con- ventional incremental learning (applying CL-DETR [23] to VLODM), and zero-shot generalizable incremental learning (i.e., the IVLOD task) in Fig. 1. Unlike general incremental object detection (general IOD, ref. second column), IVLOD characterized by preserving the original zero-shot performance of VLODM while also performing incremental learning. From this comparison, IVLOD faces two major challenges. One is the issuc known as catastrophic forgetting [29], where the perfor- mance previously learned tasks may sharply decline when new tasks are introduced. The other challenge to maintain the zero-shot generalizability VLODMs while learning new downstream tasks.
From a unified perspective, both challenges can be attributed the forgetting problem. Technically, two kinds of methods can tackle this issue: selecting exemplar samples from past datasets for replaying [31, 23] or maintaining a duplicate model facilitate knowledge distillation [20, 23]. In order to prevent forgetting, the replaying-hand based approach necessitates preserving enough examples to ensure the representativeness the samples for replaying, but it is hard to do this with a limited-size memory, specifically when the model is pre-trained the same time, current knowledge distillation techniques do not adequately focus on preventing the forgetting of pre-training knowledge, rendering them suboptimal for IVLOD tasks. Furthermore, they often need to store an entire model copy, which requires a large memory budget. Such storage demands often untenable, particularly in scenarios that essource-constraine edge devices.
large-scale vision-language dataset. At
To effectively address IVLOD's challenges while improving memory efficiency, we introduce a novel approach named Zero-interference Reparameterizable Adaptation (ZiRa). ZiRa retains the parameters of the original model and introduces parallel dual branch structure named Reparameterizable Dual Branch (RDB) designed for efficient tuning downstream tasks. The RDB structure is

---

reparameterizable. allowing the model to adapt to downstream tasks without increasing memory usage. Furthermore, the RDB structure serves a dual purpose- it achieves the branch labor division for downstream continual learning, protecting the learned knowledge from being excessively overwritenn, and more crucially, it lays the structural foundation for the key novel element of ZiRa named Zero-interference Loss (ZiL).
ZiL simultaneously penalizes the output norms of both the entire RDB and a high-learning-rate branch within the RDB, guiding the RDB to learn in a direction that protects both knowledge learned from pre- trainininng and downstream tasks. The idea of ZiL comes from the following insight: During pre- tre-training, the model is trained with visual- risual-language inputs that contain significant noise, and the model has learned how to handle this noise. As a result, the pre-trained VLODMs have a certain degree of robustness to the input, and they can handle a certain range of interference without their performance being affected. Zil. ensures that the fine-tuned residual input of RDB is of small norm, thereby guaranteeing that the model's i original performance. At the same time, for adaptation, only small adjustment to the input is needed to learn new concepts. Although ZiL. constrains the input norm of the RDB, it still allows the RDB to acquire sufficient downstream knowledge. As result, ZiL effectively addresses the two challenges of IVLOD at the same time. Crucially, ZiRa accomplishes this without necessitating the duplication of the entire model or any exemplars, thereby ensuring efficient utilization of memory.
Compared to existing methods [34, 23], ZiRa leverages the advantages offered by VLODMs more fully through a principled design (i.e., the ZiL based on RDB). offering a superior and more effective solution without the need for excessive additional memory. The contributions of this paper can be summarized as follows:
We introduce Incremental Vision-Language Object Detection (IVLOD). a novel learning task that incrementally adapts VLODMs multiple specialized domains while preserving zero-shot generalization ability.
We present a novel approach called ZiRa to tackle IVLOD challenges. Based on the reparameterizable parallel tuning structure of RDB, ZiRa introduces Zil. to simultaneously minimize interference with zero-shot performance and prevent forgetting on downstream tasks. Notably, ZiRa makes these achievements in a memory-efficient manner.
We conduct comprehensive experiments on COCO and O-in 11 datasets, showcasing the effectiveness of our method. For instance, ZiRa outperforms CL-DETR [23] and iDETR [10] by significant margin, making 13.91 and 8.74 improvement in zero-shot AP on COCO, respectively, while offering competitive results on downstream tasks incremental learning.
2 Related Work
Vision-Language (or Open Vocabulary) Object Detection. Vision-Language (or Open Vocabulary) Object Detection Models combine natural language descriptions with object detection in images. Early research in this area primarily focused on grounding textual phrases to image regions, such as grounding referring expressions [28, 39, 4] and visual question answering [1, 25, 42]. More recent studies have developed unified and scalable approaches for simultaneous object detection and grounding tasks. For instance, GLIP [19] unifies phrase grounding and object detection tasks by treating object detection as context-free phrase grounding and treating phrase grounding as a contextualized object detection task. This approach enables the integration of multi-layered textual information into the detection model. Grounding DINO [22], based on the DINO [44], employs DETR-like [2] architecture that uses Transformer as the detector, which allows for unified processing of image and language data and offers the outstanding capacity to utilize large-scale datasets. Based Grounding DINO, our work further addresses the newly introduced IVLOD tasks.
Incremental Learning and Incremental Object Detection. Incremental learning, also known as continual learning or lifelong learning, tackles the challenge of learning new tasks without forgetting previous ones. Incremental learning has given rise to two primary strategies, memory replay, and knowledge distillation, to tackle its inherent challenges [20, 3, 5, 38, 37, 40]. For example, Learn- ing without Forgetting (LwF) [20] employs knowledge distillation to maintain performance on old tasks while learning new ones. Another notable approach, iCaRL [31], combines exemplar memory
3

---

and nearest-mean-of of-exepllarss classification method for incremental learning, Incremental object detection, specialized subfield within incremental learning, focuses on improving the adaptability of object detection models to new object categories while maintaining performance on previously learned categories [34, 17, 30, 41, 11, 10, 23J. HLOD [34] was the first to introduce incremental object detection. More recently, CL-DETR [23] effectively combined knowledge distillation and exemplar memory techniques to enhance incremental object detection using Transformer-bar bbsse models. Our work distinguishes itself by performing incremental learning VLODMs, which are more favorable for open-world problems. Besides the need to continually adapt across multiple specialized tasks, is also important to preserve their zero-shot generalization ability. To the best of our knowledge, work is the first to pose and tackle IVLOD tasks.
Open- World Object Detection. Open- World Object Detection (OWOD) [16. 13, 26] also targets the detection of both seen and unseen objects, but it involves simultaneous learning of new objects and detecting unknown objects during the incremental learning process. This means that as new tasks are introduced, the model is trained to recognize new objects while also developing its ability to detect unknown objects. In contrast, Incremental Vision-Language Object Detection (IVLOD) pre-trains models to detect unknown objects before the incremental learning process begins. The focus of IVLOD during incremental learning is on preserving the model's ability to detect these unknown objects while adapting to new tasks. Given the superior zero-shot generalization capabilities of current vision- -asonnlan models, believe that starting with a pre-trained vision- language model for IVLOD is a more effective approach compared to the traditional OWOD paradigm.
Reparameterization. Reparameterization techniques optimize model inference speed by transform- ing the training-time architecture into an equivalent, simplified form for inference [8, 9, 7]. Rep [9] exemplified this approach by employing 3 x 3 and 1 x convolutional branches for training, which are streamlined to 3 3 convolutional branches and ReLU activations during inference. RepMLPNet [7] introduced "Locality Injection" to merge trained parameters of parallel convolutional kernels into fully connected layers, enhancing MLP-based model performance. Our work builds upon the Reparameterizable Dual Branch (RDB) concept initially proposed by [43]. The key innovations introduced in our approach include the use of differentiated learning rates for the High Learning Rate Branch and Low Learning Rate Branch, and the implementation of cero-interference Loss (ZiL). These enhancements allow for a more effective balance between learning new tasks and preserving previously acquired knowledge, distinguishing method from prior work.
3 Methodology
In this section, we systematically present our approach. We begin with an overview (Sec. 3.1), and then introduce Zero-interference Reparameterizable Adaptation (ZiRa), which includes the Reparameterizzale Dual Branch (RDB) (Sec. 3.2) and the Zero-interference Loss (ZiL) (Sec. 3.3).
3.1 Overview
The framework of our model for addressing the IVLOD task is illustrated in Fig. 2, with the selected VLODM being Grounding DINO [22]. Other VLODMs with a similar structure are also suitable. Specifically, the VLODMM takes image features and category text prompts as inputs and produces class-agnostic proposals whose features closely aligned with the text prompts. These text prompts are also refined to obtain text classification features by fusing the image features. Subsequently, the model classifies the class-agnostic proposals based on the cosine similarity between the visual features of proposals and the text classification features. These VLODMs are pre-trained on extensive grounding datasets annotated with detailed human language and object boxes, which equips them with great zero-shot recognition capabilities. To address the challenges of IVLOD, we propose ZiRa which is based on the structure of RDB and constrains the output of RDB by ZiL.
3.2 Reparameterizable Dual Branch
Adaption on Both Language and Vision Sides. Fine-tuning the entire model is a common method for adapting the original model to downstream tasks. However, it has been demonstrated that tuning only a few newly introduced parameters in a pre-trained model is more effective [14, 155, while full fine-tuning often leads to significant forgetting. In view of this, we introduce additional parallel
4

---

Lineur RDB
00000000
Tunable
Cat.
Frozen
Dog.
Text Backbone
Pre-trained Linear
beta
Flower.
Vision-
Language
Text
Prompts
Object Detector
Text Classification
Features
Image
Pre-trained Project Conv
Backbone
x
Conv RDB
Image
Ligt
www. Class-agnostic Proposals
Figure 2: Our framework, features two Reparameterizable Dual Branch with Zero-interference Loss on both the vision and language sides.
branches on both the language and vision sides of the VLODM to adapt the model for sequential downstream tasks.
As shown in Fig. 2, these additional branches are termed the Reparameterizable Dual Branch (RDB). On the language side, we integrate an RDB alongside pre-trained linear layer positioned between the detector and the language backbone network, aiming learn the new high-level semantic concepts. On the visual side, we also connect supplementary RDB in parallel with a pre-trained convolutional layer located between the visual backhone and the detector, aiming to learn the visual features of new classes. It's pivotal to adapt both sides, as these two sides capture distinct structures of vital knowledge and the lack of either will lead insufficient adaptation to downstream tasks.
Dual Branch Structure within the RDB. Within the RDB, illustrated in Fig. 3, our design features dual- branch structure, comprising the Low-learning rate Branch (LLRB) and the High-learning rate Branch (HLRB). The HLRB is set to high learning rate for rapid task adap- tation, whereas the LLRB employs more conserva- tive learning rate. Specifically, the LLRB is set at !) (0 < <n < 1) times the learning rate of the HLRB. This differentiation in learning rates brings division of labor between LLRB and HLRB, especially when combined with our next strategy that reparameterizes the HLRB into LLRB after learning each task. The LLRB's ' low learn- ing rate help maintain downstream task knowledge, while the HLRB's s high learning rate can help swift adap- tation to tasks. This division protects the knowledge stored in the LLRB from being excessively overwritten when incrementally learning downstream tasks, achieving a better balance between stability the plasticity, while the single branch or naive dual branch structure not take advantage of this division mechanism.
ZiL RDB
Add
Scaling
ZiL for HIRB
Frozen
Pre-
trained
LLRB
Branch
HLRB
Reparameterable Dual Brnnh (RDB)
Reparameterization each
Reperameterization after all tasks
Figure 3: The structure of the Reparam- eterizable Dual Branch (RDB).
Specifically, the collective output of the RDB Frdbb is formulated as:
Frdhh HLRB(()) (x) LLRB(r)),
where r is the input and s learnable scaling factor.
For the language side, the RDB structured as two parallel linear layers, and its output of expressed as:
rab (Wh + blar)) s Wh x + blr
(2)
where W hirs and WT Wrb represent the weights of HLRBB and L.RBB, respectively, while tharo and Ulrrb denote their corresponding biases.
5

---

For the vision side, the RDB is structured as two parallel convolution layers, and its output 2/40 is illustrated as:
Frdb (Whrre hdrb & + bhrb)) + Wiirs
itrb @ x Birbb.
(3)
Here, Whrs and Wird stand for the kernel of HLRB and LLRB, respectively, while hard and birb represent their corresponding biases. @ denotes the convolution operation.
Reparameterizattin from the HLRB to the LLRBB With each new task, ZiRa resets the HLRB parameters to zero and, after completing learning on new task, merges the trained parameters from the HLRB into the LLRB. This approach not only allows the L.RBB to incrementally acquire new knowledge but also avoids the linear increase in the number of additional branches the number of tasks increases, effectively managing memory consumption. Moreover, it offers a structural prerequisite for ZiL to prevent forgetting downstream tasks by penalizing the output norm of HLRB, which will be detailed in the next section.
On the language side, ZiRa combines the HLRB and LLRB into a consolidated LLRB with new weights W kdrb $ + W Urb and new biases b/aro + blrvs given by:
ai =
(Whers htrs s +Wirn) + (Bhers s + b(tr))
(4)
Similarly, on the vision side, the HLRB and LLRB are merged into singular LLRB, as depicted by:
rib = WWhrrs s Würb) & x + (bhtro 8 + b(rr)); where W kimb W Hirb and herb bürb can be considered as the new kernel and new biases.
(5)
Single Branch for Inference. Sub-branches inside the RDB like the LLRB and HLRB can increase computational overhead. We efficiently address this by reparameterizing all uub-branches in RDB and the frozen pre-trained branch into a new one for inference. Since the frozen pre-trained branch is also a linear structure, after learning all the tasks, we can create new branch that merges the frozen pre-trained branch and the RDB via reparameterization. During inference, we exclusively use this newly consolidated branch, but we still retain the parameters frozen pre-trained branch and the RDB for future learning. This approach significantly reduces the computational cost during inference, as only one branch is required for prediction
3.3 Zero-interference Loss
Protect Zero-shot Generalizability with ZiL. Preventing performance degradation on the original pre-training domain can be viewed as preventing the VLODM from forgetting the original pre-training domain knowledge. Two common methods to avoid forgetting is to reserve some old samples for replaying distill the current model from the previous ones. However, preserving old samples and storing the previously trained model requires significant additional memory. On the contrary, Gero-interference Loss (ZiL) can prevent forgetting in memory efficient manner.
Specifically, ZiL penalizes the norm of the whole RDB's output (ref. Fig. 3), aiming at safeguarding the shoo performance of VLODM. The Zil. for the RDB, Lerab, is defined as follows:
Lidb = (z(ra))).
(6)
Here, L, corresponds to the L, norm.
Why ZiL works. The effectiveness of Zil, can explained as follows. First, the VLODM's input (What we fine-tuned in this paper) is robust to small-norm random perturbations, which have little impact on the model's performance. To verify this, we add Gaussian noise to the input of the pre- trained VLODM's detector and observe the model's performance. As shown in Fig. 4, the model's performance not significantly affected by the addition of small-norm random noise to the input.
Second, the ZiL ensures that the fine-tuned input of the VLODM has small norm additional term. To further investigate the effectiveness of ZiL, we plot curves illustrating the evolution of the RDB's output norm throughout the IVLOD process, shown in Fig. 5. It is evident that training the RDB with ZiL leads to a substantial reduction in the output norm of the RDB on COCO, compared to training the RDB without Zil. This preliminary observation strongly suggests that Zil. effectively preserves the zero-shot generalizability of VLODMs. Examining the evolution curves Fig. more closely, we observe that in the initial stages, ZiL plays relatively minor role in curbing the growth

---

downstream
The sverage Ll narm of RDB' camput (log)
COCO
47.
70000
33
10
K2.5
average nome RDB's
25
90.0
avegge
35.0
223
30.0
sannead deviatioon skiid VLODM
With ZIL (SRDOW-L3)
With AL (CUCC)) Witheu (E0CO))
Wither (01HW--17)
Number of learned downstream tasks
Number of leamed downstream tasks
Figure The performance of the pre-trained VLODM with different levels of Gaussian noise added to the input of VLODM's detector.
Figure 5: The average L1 norm curve of the RDB's s output overall sequentially learned downstream tasks, computing the output norm on both language and vision sides. The longitu- dinal axis logarithmically scaled for better visualization.
of the RDB's output norm. However, as the process of accumulating new knowledge progresses, the influence of ZiL. becomes more significant, leading stronger constraint effects. This increase in ZiL's impact leads to a dynamic balance: the interference caused by integrating new knowledge is counteracted the interference reduction achieved by ZiL., which is reflected in the fact that as the number of tasks increases, the norm of the RDB's output doesn't It rise unboundedly but rather stabilizes after learning certain number of tasks. This dynamic balance is crucial for maintaining the zero-shot generalizability of VLODMs.
Prevent forgetting downstream tasks with ZiL. The Lyab serves primarily to preserve the zero-shot generalization capability, leaving the challenge of downstream task forgetting unresolved. By exploiting the advantage of RDB, address this challenge by further applying ZiL on HLRB, which is computed as:
Lutrb = L1 (HLRB(x)) 8),
where s a learnable scaling parameter.
The Liiro can effectively protect the knowledge that learned from previous tasks and encompassed within the combined LLRB and the pre-trained branch. Specifically, given that the knowledge learned from the previous task has already been reparameterized into LLRB and LLRB learns at lower learning rate, the total output of the branches previously trained can be regarded as LLRB(x) + PTB(z), where PTB is the pre-trained branch. The total output of the current branches LLRB(x)) + HL)B(()) s (T)( which has additional component (1)B(xx) compared to the output from the branches trained on the previous task. Hence, optimizing Lutrb equates to penalizing the discrepancy in the outputs between the models trained the previous and the current task, effectively guiding the HLRB to learn new tasks without much downstream task forgetting.
Total Optimization Objective. For general VLODM, we continue to employ its original training loss. To prevent forgetting, we incorporate an additional loss term, Lust- Taking Grounding DINO as an example, the total loss function becomes:
(8) where Lua Lydb + Lptrb is the proposed ZiL, X is its loss weight and L,ts and Lioe are the contrastive classification loss and localization loss of Grounding DINO. Throughout the training process, the entire original VLODM is frozen, except for the RDB.
Lnoaal Lets + Istoc + XLzit,
Experiments
4.1 Setup
Datasets. We conduct our experiments on the COCO [21] datasets and the "Object Detection in the Wild (ODinW)" [18] benchmark. ODinW a more challenging benchmark designed to test model performance under real-world scenarios. It comprises numerous sub-datasets from various domains evaluation, such as Thermal (to detect objects in heat map images) and Aquarium (to detect different marine animals). Following GLIP [19],
ODinW-13 datasets, they are labeled Ac
7

---

(Aerial Maritime Drone), Ag (Aquarium), Co (Cottontail Rabbits), Eg (Egohands), Mu (Mushrooms), Pa (Packages), Pv (Pascal VOC), Pi (Pistols), Po (Pothole). Ra (Raccoon), Sh (Shellfish), Th (Thermal Dogs and People), Ve (Vehicles). The 13 sub-datasets of DDinW-13 are trained sequentially, one by one, and are tested after all sub-datasets have been trained.
Metries. As COCO is a dataset within the realm of everyday life, we use zero-shot COCO perfor- mance as metric of the model's zero-shot generalization ability across general domains. Additionally, we provide the average performance on the downstream ODinW-13 benchmark, with performance calculations made only after the model has been sequentially trained on all sub-dataset We denote the zero-shot mAP on COCO as "ZCOCO" and the average mAP on ODin W-1 downstream tasks as "Avg".
Implementation Details. In this paper, unless otherwise specified, we use the Grounding DINO [22] with the Swin-Tiny [24] backbone the default original VLODM in our experiments. This Grounding DINO is pre-trained on Objects365 [33], GoldG [19]. and Cap4M [19]. We also use
OV-DINO [35] with the Swin-Tiny backbone as an additional validation to demonstrate ZiRa's ability to generalize effectively across different DETR-based architectures. This OV-DINO model pre-trained on the Objects365 [33], GoldG [19], and CapIM [35] datasets. The learning order of the ODin W-13 sub-datasets is randomly shuffled. All results are obtained after running experiments three times with different random seeds. The hyperparameter experiments are also conducted on the 1D dataset and COCO, and the results are shown in the appendix.
Our proposed method is implemented with PyTorch and trained on two Nvidia RTX 3090 GPUs. Each downstream task is trained for a total of two epochs with a batch size of 2. For Grounding DINO, we employ initial learning rate of 10-3, which decays to 0.1 times the original value after the first epoch to ensure effective convergence. For OV-DINO, we employ an initial learning rate of 10-4, which also decays 0.1 times the original value after the first epoch ensure effective convergence. AdamW is used as the optimizer, and the weight decay is 10-4, At the beginning of IVLOD, we initialize the parameters of the total RDB as zero
Table I: IVLOD results on 1D and ZCOCO. All results are based on the same Grounding DINO. "Shots" means how many samples are used when adaptation training.
Shots
Methods
ZCOCO
Avg
Ae
Aq
Co
Eg
Mu
Pa
Pv
P4
Po
Ra
Sh
Th
Ve
0
Original Model
47.37
46.70
19.
10
20.75
64.72
56.96
25. 39
54.49
54,80
65.97
22.08
62
22
32.84
70.62
57.15
TFA
18.84
39.50
18.25
63.90
50.79
28.47
50.37
29.49
59.16
21
50.67
19.86
43,97
iDETR
44.61
49.82
22.81
23,24
69.75
61.43
31.73
56.27
55.40
62.44
28.45
60.33
43.33
73.64
58.84
AT
44.11
46.23
21.55
23.62
66.60
58.96
27.68
53.97
54.58
62.47
26.94
53.17
20.37
70.31
60.71
ZiRa
45.25
50.20
19.91
24.84
68.82
63.02
37.02
60.20
55.11
63.37
28.26
66.35
38.97
69.27
57.43
TFA
26.43
45.76
21.92
22.30
67.40
60.72
30.63
53.56
46.80
63.60
26.88
56
28.00
64.28
52.49
5
IDETR
43.51
51.65
25.69
25.53
70.42
62.98
49.98
50.54
54.85
64.80
33.24
57.64
42.36
76.51
56.92
AT
43.67
47.16
14.63
24.97
66.56
64.19
38.85
42.03
55.49
65.20
27.48
52.68
32.21
71.23
57.54
ZiRa
45.53
54.19
23
30,92
72.46
66.34
51.23
56.27
60.35
67.4
36.22
60.18
42
77.18
59.55
TEA
34.35
46.61
21.17
22.16
62.82
60.63
32.
50.15
55.34
64.98
27.59
57
28.14
66.82
52.11
10
IDETR
43.54
53.29
25.39
27.70
65.62
67.58
47.99
60.20
56.32
63.93
35.18
59.37
53.63
74.70
55.12
AT
43.06
47.34
18.73
25.42
69.77
66
35 84
48.25
53.39
64.07
28.89
50.33
32
66.57
55.37
ZiRa
45.70
54.86
25.01
29, 44
69.59
68.44
54.72
61.68
59.69
67.14
36.05
61.77
48.30
73.92
57.47
TFA
30.97
47.93
23.80
30.65
67.21
61.77
30.52
50,23
47.73
60.91
29.25
61.72
31.42
66.23
61.61
iDETR
37.32
58.71
32.64
46.65
70.99
68.56
55.32
58.88
64.48
71.01
50.33
63.30
39.19
77.12
64.80
Full
AT
42.30
51.14
23.62
39.90
72.32
65.51
31.47
50.48
60.51
66.07
39,09
53.50
34,04
68.07
60.23
OW-DETR
31.22
55.58
28, 46
43.78
70.54
67.7
43. 84
56.75
63
69.51
45.16
58
36
36.99
74.42
63.20
CL-DETR
32.15
57.26
29.35
69.90
65.10
71.68
60.83
76.74
ZiRa
46.06
59.73
32.81
48.19
70.33
69.67
59.33
58.05
64.04
70.67
50.06
67.49
45.51
76.76 63.54
Table 2: IVLOD results on ODin W- 13 and ZCOCO. All results are based on the same OV-DINO.
Shots
Methods
ZCOCO
Avg
Ae
Ag
Co
Eg
Mu
Pv
Pi
Po
Ra
Sh
Th
Zero
Orginnal Model
50.22
26.64
15.69
19.37
11.79
40.67
1.23
59.26
50.78
12.19
33.81
8.50
43.68
46.86
Pull
IFA
44.98
47.50
22.27
35.36
63.58
56.68
19.84
64.26
68.07
50.01
25.99
56.31
33.70
56.96
65.20
Full
CLEDERR
34.52
45.28
23.07
27.90 4681
67.72
25.39
66.56
59.12
28.04
29.48
64.71 25.58
68.28
56.05
Full
AT
36.80
44.33
22.46
27.42
45.78
60
24.10
65.08
58
26.65
28.88
64.46
24.60
67.30
54.27
Full
IDETR
37.71
46.91
24.09
30.56
49.16
68.49
26.05
67.45
61.85
30.75
29.50
66,83
27.11
71.27
56.69
Full
ZiRa
49.07
50.21
23.37
40.43
69.02
66.21
20.74
58.51
67.74
46.71
34.13
64.62
39.44
58.01
63.81
4.2
Comparison with Existing Methods
In our evaluation, we compare ZiRa with existing Incremental Object Detection (IOD) methods, with
a specific focus on methods based on Detection Transformers (DETR) [2], as our approach falls into this category. The bascline methods we consider include:
8

---

TFA [36]: A classical linear linear-probing probing- based incremental few-shot object detection baseline.
OW-DETR [13]: An open-world object detection approach based on DETR, primarily addressing forgetting through exemplar replaying, which necessitates additional memory to store exemplars.
CL-DETR [23]: A DETR-basee method employing refined knowledge distillation and exemplar replaying- This method requires a complete model copy and exemplars, incurring significant extra memory costs.
iDETR [10]: An incremental few-shot object detection method specifically focusing on tuning the projection layer of DETR-like detectors via knowledge distillation and self- supervision.
Adapting-tunine (AT) [14]: A parameter-efficient adaptation method that has demonstrated effectiveness incremental few-shot object detection [6].
We first compare the above methods implemented the Grounding DINO model. Experiments are conducted under both few-shot and full-shot settings. Since CL-DETR and OW-DETR are not designed for few-shot IOD, only compare them under the full-shot setting. The results presented in Tab. 1 demonstrate that ZiRa consistently outperforms existing IOD methods in terms of "Avg" on downstream tasks. Moreover, ZiRa exhibits remarkable performance in preserving the zero-shot generalization ability of VL.OMMs under both the few-shot and full-shot settings. In particular, ZiRa surpasses CL-DETR and iDETR by substantial margins, with improvements of 13.91 and 8.74 AP under the full-shot setting in terms of "ZCOCO", respectively.
We then compare ZiRa with existing methods implemented the OV-DINO model. The results in Tab. show that ZiRa outperforms existing IOD methods in terms of "Avg" on downstream tasks. ZiRa also demonstrates excellent performance in preserving the zero-shot generalization ability of VLODMs under the full-shot setting. Specifically, ZiRa surpasses CL-DETR and iDETR by significant margins, with improvements of 14.55 and 11.36 AP in the full-shot setting for "ZCOCO," respectively.
These results emphasize ZiRa's effectiveness in maintaining the zero-shot AP and its potential as superior solution for IOD. Distinctively, ZiRa, unlike CL-DETR and OW-DETR, requires minimal extra memory for a few branches without storing image exemplars and the model copy, making more memory-efficient than other IOD methods.
4.3
Ablation Study
Table 3:
Main ablation results.
Ropt
Lo
LAsA
ZCOCO
Avg
hAP
Ae
Aq
Co
Eg
Mu
Ra
Th
x
x
x
39.72
58.12
47.19
30.26
47.11
69.30
66.16
56.82
58.20
64.34
69.08
48.48
63.35
41.53
76.59
64.37
x
x
42.11
59.98
49.48
32.17
48.37
70.97
69.28
59.32
58.05
66.39
71.50
51.35
67.46
42.01
78.15
64.71
X
46.09
54.24
49.83
30.14
33.84
68.80
65.29
42.27
60.20
61.42
69.54
34.79
64.99
39.31
74.41
60.12
/
46.06
59.73
52.01
32.81
48.19
70.33
69.67
59.33
58.05
64.04
70.67
50.06
67.49
45.51
76.76
63.54
X
46.01
58.43
51.48
30.77
48.91
69.69
68.93
51.98
58.05
61.80
70.58
45.71
67.72
43.35
78.19
63.95
X
V
44.93
60.83
51.68
32.62
47.82
71.57
70.06
60.13
58.05
66.33
71.03
52.72
68.59
47.42
78.28
66.20
Main Components of ZiRa. Our approach combines the ZiL and the RDB to counteract forgetting in both the general domain and downstream tasks. In this study, we analyze the impact of these components. We utilize "Rep+" to denote whether to reparameterize the HLRB into LLRB after each task as well as using differentiated lcarning rates. Please note that even when we do not use "Rep+", we still retain the dual-branch structure like RDB. ZiL encompasses the RDB's s ZiL (Lyas) and HLRB's s ZiL (Larr)), ), here we separate them and study their impact separately. We also evaluate performance with the "hAP" metric, which is the harmonic mean of "ZCOCO" and "Avg".
The results in Tab. 3 provide valuable insights. First, the comparison between the first and second rows demonstrates that "Rep+" can mitigate forgetting on both pre- training and downstream tasks. This effect comes from the labor division between branches that are built on differentiated learning rates and reparameterization. Second, the comparison between the first row and third rows highlights that Zil. (Trra + Thars) significantly enhances the "7COCO". However, cannot address forgetting on downstream tasks without "Rep+", and it even reduces the "Avg" since also limits the plasticity

---

of the RDB. The best results lie in the fourth row, underscoring that combining "Rep+" and ZiL can effectively mitigate forgetting on both pre-training and downstream tasks greater extent. Conversely, the results in the last row illustrate that using (Trrdb) alone does not optimally address forgetting on downstream tasks. Comparing the second row and the last row, we can find that Lhtrb can improve both "ZCOCO" and "Avg", showcasing that Late can mitigate forgetting both on downstream tasks and pre-training, but without Lydb, using Liteb alone can not achieve the best "hAP".
Learning on Different Modalities with ZiRa. We carried out a series of experiments investigate the effects of learning only on the vision side using parallel convolution layers (indicated as "V") and only on the language side using parallel linear layers (denoted as "L"). The results in Tab. demonstrate that ZiRa can function independently on either the language or vision side, and it can also effectively function when simultaneously applied both modalities, highlighting the generality of ZiRa's impact. Since the language and vision side learning acquires knowledge in distinct structures, neglecting either can hinder optimal learning. Consequently, simultaneously learning on both sides can outperform individual tuning either the vision or language sides when employing ZiRa to avoid forgetting.
Table 4: Comparison of learning on different modalities.
Table 5: Comparison of different additional
V
ZiRa
ZCOCO
Avg
hAP
branch structures. We compared the results of
X
47.37
46.70
47.03
introducing single branch (denoted as SB), dual branches (denoted as DB), and RDB (DB
V
X
x
39.44
58.23
47.03
+ Rep+).
X
x
45.83
54.77
49.90
J
X
Structure
ZCOCO
Avg
hAP
XXX
39.72
58.12
47.19
SB
46.16
55.07
50.22
45.96
57.64
51.14
DB
46.09
54.24
49.83
X
46.41
56.49
50.96
RDB
46.06
59.73
52.01
46.06
59.73
52.01
Branch Structure. To study the impact of the additional branch structure introduced in VOLDM, we compared the results of introducing a single branch (denoted as SB), dual branches (denoted as DB), and RDB (DB + Rep+). Besides RDB, both the output norm of SB and DB are also penalized with ZiL. As we can see in Tab. 5, the SB structure, due to its inability to introduce inter-branch division of labor, falls short in its ability to prevent forgetting in downstream tasks compared to the RDB. Moreover, we can see that merely introducing DB with ZiL cannot prevent forgetting on downstream tasks, on the contrary, it brings worse forgetting due increased plasticity. RDB not only incorporates the mechanism of branch labor division but also fully utilizes lAtrs to learn to minimize the interference of HLRB, thereby more effectively preventing the forgetting downstream tasks.
5 Conclusion
This paper presents a novel learning task, Incremental Vision- LaaguuLe Object Detection (IVLOD), which aims to continually adapt Vision-Language Object Detection Models (VLODMs) to multiple specialized domains while preserving VLODMMs' zero-shot generalization ability. To solve this new task in memory-efficient way, we introduce Zero-interference Reparameterizable Adaptation (ZiRa). ZiRa inserts Reparameterizable Dual Branche (RDB) on the both language side and vision side of the VLODM and constrains the RDB by iero-interference Loss (ZiL) to protect the original generalizability of VLODMs and prevent forgetting on downstream tasks at the same time. Notably, ZiRa eliminates the need for saving the entire model copy for distillation or maintaining exemplars for replaying, which makes a highly memory-efficient method. Extensive experiments conducted on the COCO and ODin datasets showcase the superiority of ZiRa for IVLOD.
Limitations. While ZiRa has demonstrated its effectiveness in IVLOD tasks, there are still some limitations. First, the current implementation of ZiRa is based on the DETR architecture, which may not be optimal for all VLOODMs. Second, the current ZiRa method designed for the IVL.OD task, and its generalization to other incremental learning tasks remains to be explored.
10

---

Acknowledgments
We would like to express our deepest gratitude to the anonymous reviewers and the community for their valuable feedback, which helped enhance the quality of this work. Additionally, we are grateful to our colleagues for their insightful discussions and support throughout the development of this project.
References
[1] Anderson, P., He, X., Buchler, C., Teney, D., Johnson, M., Gould, S., Zhang. L.: Bottom-up and top-down attention for image captioning and visual question answering. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018. pp. 6077-6086 6086 (2018)
[2] Carion, N., Massa, F., Synnaeve, G., Usumier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J. (eds.) Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 1. Lecture Notes in Computer Science, vol. 12346, pp. 213-229 (2020)
[3] Cermelli, F., Mancini, M., Bulô, S.R., Ricci, E., Caputo, B.: Modeling the background for incremental learning in semantic segmentation. In: 2020 IEEE/CVF Conference on Computer Vision and Pattem Recognition, CVPR 2020. Seattle, WA, USA, June 13-19, 2020. pp. 9230-9231 (2020)
[4] Deng, C., Wu, Q. Wu, Q. Hu, F., Lyu, F., Tan, M.: Visual grounding via accumulated attention. IEEE Trans. Pattern Anal. Mach. Intell. 44(3), 1670- 1670-1684 (2022)
[5] Deng, J., Hu, J., Zhang, H., Wang, Y.: Incremental prototype prompt- with pre-trained representation for class incremental learning. arXiv abs/2204.03410 (2022)
[6] Deng, J., Zhang, H., Hu, Zhang, X.. Wang. Y:: Class incremental robotic pick-and-place via incremental few-shot object detection. IEEE Robotics Autom. Lett. 8(9), 5974-5981 (2023)
[7] Ding, X., Chen, H., Zhang, X., Han, J., Ding, G.: Repmlpnet: Hicrarchical vision MLP with re- parameterized locality. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18 18-24, 2022. pp. 568-577 (2022)
[8] Ding, X., Guo, Y.. Ding, G., Han, J.: Acnet: Strengthening kernel skeletons for powerful CNN via asymmetric convolution blocks. In: 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Scoul, Korea (South). October 27 November 2, 2019. pp. 1911-1920 (2019)
[9] Ding, X., Zhang, X., Ma, N., Han, J., Ding, G., Sun, J.: Repvgg: Making vgg-style convnets great again.
In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021.
pp. 13733-13742 (2021)
[10] Dong, N., Zhang, Y. Ding. M., Lec, G.H.: Incremental-dert Incremental few-shot object detection via self-supervised learning. In: Williams, B., Chen, Y., Neville, J. (cds.) Thirty- Serrt-SSeventh AAAI Conference Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference
Innovative Applications of Artificial Intelligence, IAAI 2023, Thirtcenth Sympasium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023. pp. 543-551 (2023)
[11] Ganea, D.A., Boom, B., Poppe. R.: Incremental few-shot instance segmentation. In: IEEE Conference on Computer Vision Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021. pp. 1185- 1194 (2021)
[12] Gu, X., Lin, T., Kuo, W., Cui, Y.: Open-vocabulary object detection via vision and language knowledge distillation. In: The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 (2022)
[13] Gupta, A., Narayan, S.. Joseph, K.J., Khan, S., Khan, F.S., Shah, M.: OW-DETR: open-worle detection transformer. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. pp. 9225-9234 (2022)
[14] Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M.. Gelly, S.: Parameter-efficient transfer learning NLP. Chaudhuri, K., Salakhutdinov. R. (eds.) Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California. USA. Proceedings of Machine Learning Research, vol. 97, pp. 2790-2799 (2019)
[15] Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu Z., Li, Y., Wang, S., Wang, L., Chen, W.: LoRA: Low-rank adaptation of large language models. In: The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 (2022)
[16] Joseph, K.J., Khan, S.H., Khan, F.S., Balasubramanian, V.N.: Towards open world object detection. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021. pp. (2021)
11

---

[17] Joseph, K.J., Rajasegaran, J., Khan, S.H., Khan, F.S.. Balasubramanian, V.N.: Incremental object detection via meta-learning. IEEE Trans. Pattern Anal. Mach. Intell. 44 9209-9216 (2022)
[18] Li, C., Liu, H., Li, L.H., Zhang. P., Aneja, J. Yang, Jin, P., Lee, YJ., Hu, H., Liu, Z., Gao, J. ELEVATER:
A benchmark and toolkit for evaluating language-augmenied visual models. arXiv abs/2204.08790 (2022)
[19] Li. L.H., Zhang, P... Zhang. H., Yang. J., Li, C., Zhong. Y., Wang, L., Yuan, L.,, Zhang. L.. Hwang. J., Chang, K., Gao, J.: Grounded language-image pre-training. In: IEEE/CVT Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA,, USA, June 18-24, 2022. pp. 10955-10965 (2022)
[20] Li, Z., Hoiem, D.: Learning without forgetting. In: Leibe, B., Matas, J., Sehe, N., Welling, M. (eds.) Computer Vision - ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV. Lecture Notes in Computer Science, vol. 9908, pp. 614-629 (2016)
[21] Lin, T., Maire, M., Belongie, S.,., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: common objects in context. In: Fieet, DJJ., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) Computer Vision ECCV 2014- 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V. Lecture Notes in Computer Science, vol. 8693, pp. 740-755 (2014)
[22] Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang. J., Li, C., Yang, J., Su, H., Zhu, J., Zhang. L.: Grounding DINO: marrying DINO with grounded pre-training for open-set object detection, arXiv abs/2303.05499 (2023)
[23] Liu, Y., Schiele, B., Vedaldi, A., Rupprecht, C.: Continual detection transformer for incremental object detection. In: TEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023. pp. 23799- (2023)
[24] Liu, Z.. Lin, Y. Cao. Y.. Hu, H., Wei, Y., Zhang. Z.. Lin, S.. Guo, B.: Swin Transformer: Hierarchical vision transformer using shifted windows. In: TEFF/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021. pp. 9992-10002 (2021)
[25] Lu, J., Batra, D., Parikh, D., Lee, S.: Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-lane and-aagguge tasks. In: Wallach, H.M., Larochelle, H.,, Beygelzimer, A., d' d'Alché- Bue, F., Fox, E.B., Gamett, R. (eds.) NeurIPS. pp. 13-23 (2019)
[26] Ma, Z., Yang. Y., Wang, G., Xu, X., Shen, H.T., Zhang, M.: Rethinking open- object detection in autonomous driving scenarios. In: Magalhäes, J.. Bimbo, A.D., Satoh, S., Sebe, N., Alameda- X., Jin, Q. Oria, V. Toni, L. (eds.) MM '22 The 30th ACM International Conference on Multimedia, Lisboa, Portugal, October 10- 14, 2022. pp. 1279 1288 (2022)
[27] Maaz. M.. Rasheed, H.A., Khan, S., Khan, F.S., Anwer, R.M., Yang, M.: Class-agnostic object detection
with multi-modal transformer. In: Avidan, S., Brostow, G.J., Cissé, M., Farinella, G.M., Hassner. T. (eds.) Computer Vision ECCV 2022 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part X. Lecture Notes in Computer Science, vol. 13670, pp. 512-531 (2022)
[28] Mao, J., Huang, J., Toshev, A., Camburu, O.,, Yuille, A.L., Murphy, K.: Generation and comprehension of unambiguous object descriptions, In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016. pp. 11-20 (2016)
[29] McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: The sequential learning problem. In: Psychology of Learning and Motivation, vol. 24, pp. 109- 169-16610 (1989)
[30] Pérez-Rúa, J., Zhu, X., Hospedales, T.M., Xiang, T.: Incremental few-shot object detection. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, (2020)
2020. pp.
[31] Rebuffi, S., Kolesnikov, A., Sperl, G., Lampert, C.H.: icarl: Incremental classifier and representation
learning. In: IEEE Conference Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI. USA, July 21 21-26, 2017 pp. 5533-5542 (2017)
[32] Ren, S., Hc, K., Girshick, R.B., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1137-1149 (2017)
[33] Shao, S., Li, Z., Zhang, T., Peng, C., Yu, G., Zhang, X., Li, J., Sun, Objects365: A large-scale, high-quality dataset for object detection. In: IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019. pp. 8429-8438 (2019)
[34] Shmelkov, K., Schmid, C., Alahari, K.: Incremental learning of object detectors without catastrophic forgetting. In: IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017. pp. 3420-3429 (2017)
[35] Wang, H., Ren, P., Jie, Z., Dong, X., Feng. C., Qian, Y., Ma, L., Jiang, D., Wang. Y., Lan, X., Liang. X.: OV-DINO: unified open- vocabulayy detection with language selective fusion abs/2407.07844 (2024)
12

---

[36] Wang, X., Huang, T.E., Gonzalez, J., Darrelll T., Yu, F. Frustratingly simple few-shot object detection. In: Proceedings of the 37th International Conference Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event. Proceedings of Machine Learning Research, vol. 119, pp. 9919-9928 (2020)
[37] Wang. Y., Huang, Z., Hong. X.: S- prompts learning with pre-trained transformers: An occam's 's razor for domain incremental learning. arXiv abs/2207.12811 (2022)
[38] Wang, Z., Zhang. Z., Lee, C., Zhang. H., Sun, R.. Ren, X., Su, G., Perot, V., Dy, J.G., Pfister, T.: Learning
to prompt for continual learning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. pp. 139-149 (2022)
[39] Yang, S., Li, G., Yu, Y: eelationship-embeddee representation learning for grounding referring expressions. IEEE Trans. Pattern Anal. Mach. Intell. 43(8), 2765-2779 (2021)
[40] Yang, 7.,, Li, R., Ling, E., Zhang, C., Wang. Y., Huang, D., Ma, K.T., Hur, M., Lin, G.: Label-guided knowl- edge distillation for continual semantic segmentation on 2d images and 3d point clouds. In: IRFF/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023. pp. 18555- 18566 (2023)
[41] Yin, L., Perez-Rua, J.M., Liang. K.J.: Sylph: hypernetwork framework for incremental few-shot object detection. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. pp.
(2022)
[42] Yu, Z.. Yu, J.. Cui, Y. Tao, D., Tian, Q.: Deep modular co-attention networks visual question answering. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. pp. 6281-62999 (2019)
[43] Zhang, C., Xiao, Liu, X., Chen, Y., Cheng, M.: Representation compensation networks for continual semantic segmentation, In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. pp. 7043-7054 (2022)
[44] Zhang. H., Li, F., Liu, S., Zhang, I.., Su, H., Zhu, J., Ni, L.M., Shum, H.: DINO: DETR with improved denoising anchor boxes end-to-end object detection. In: The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 (2023)
[45] Zhang. H. Zhang, P., Hu, X., Chen, Y., Li. L.H., Dai, X., Wang, L.. Yuan, L., Hwang, J., Gao, J.: Glipv2: Unifying localization and vision- Isionnlanguuae understanding. In; NeurIPS, pp. 36067-36080 (2022)
13

---

Appendix
Effeet of Hyperparameter
Impact of A. The hyperparameter À modulates the influence Zil. within the overall loss function. In this study, we investigate the impact of varying values of on model performance, as presented in Tab. 6. The results show that a small À value (e.g., 0.01) tends to cause the model to overfit new tasks, resulting reduced performance in the general domain. Conversely, large À value (c.g.. 1.0) overly constrains the model, hindering its adaptation to downstream tasks. The optimal À value falls between these extremes, with the highest hAP observed when À 0.05, which strikes balance between "ZCOCO" and "Avg".
Table 6: Results with varying for ZiL.
ZCOCO
Avg
hAP
0.00
39.72
58.
47.19
0.01
44.58
60.86
51.46
0.05
45.65
60.54
52.05
0.10
46.06
59.73
52.01
0.20
46.25
58.13
51.51
0.50
46.60
54.27
50.14
1.00
46.94
50.08
48.46
Differentiated Learning Rate. We explore the effects of hyperparameter 1). The results are detailed in Tab. On the one hand, setting the LLRB's learning rate as zero equivalently freezes it. Although this results in the highest "Avg" by preserving downstream task knowledge, the "ZCOCO" performance suffers considerably. This degradation because HIRBB, which has limited parameter size, endures all the optimization pressure from Lit and Lete Despite HLRB's s attempts to find harmonious optimization path, tends to emphasize downstream task learning over maintaining initial zero-shot generalizability, especially when the predominant loss Lets Lioc outweighs Led. On the other hand, equating the LLRB's s learning rate to the HLRB's s is equivalent to creating two HLRBs inside the RDB. The enhanced plasticity that comes from the two-HLRB structure makes it challenging for ZiL to prevent it from gravitating toward directions detrimental to zero-shot generalizability, leading to diminished "ZCOCO" scores. Moreover, significantly high LLRB learning rate can also precipitate a swift decline in downstream task knowledge retention. Optimal lies between these two cases, like 0.1 0.2, striking balance between efficient new task adaptation and zero-shot generalizability protection.
Table 7: Results with varying 17.
7)
ZCOCO
Avg
hAP
0.00
44.97
60.23
51.49
0.10
45.93
59.79
51.95
0.20
46.06
59.73
52.01
0.50
45.87
59,46
51.79
0.70
45.65
58.78
51.39
1.00
44.82
58.64
50.81
learnable Impact of Different Initial Values of the Scaling Factor, The scaling factor parameter that affects the IVLOO's performance. Its initial value could shape the model's efficiency and adaptation capabilities. As documented Tab. 8, contrary initial expectations, a balanced scaling factor for both language and vision components, set at 1.00, does not lead to the best outcomes. Instead, an asymmetrical approach where the vision component's 's scaling factor is reduced 0.10 while maintaining the language component's scaling factor at 1 1.00, or vice versa, appears to enhance overall performance metrics, including "ZCOCO". "Avg". and "hAP". Remarkably, the configuration where both language and vision scaling factors are set to 0.10 achieves superior results, yielding a slight improvement "ZCOCO" and the highest "hAP" score among the tested scenarios. This suggests that more conservative initial scaling factor may encourage the model to rely more on the base knowledge from pre-training, thus improving its generalization before adapting further through incremental learning.
14

---

Table 8: Impact of Different Initial Values of the Scaling Factor.
Language
Vision
ZCOCO
Avg
hAP
1.00
0.10
46.06
59.73
52.01
1.00
1.00
45.39
60.40
51.83
0.10
1.00
45.20
60.46
51.73
0.10
0.10
46.03
60.16
52.15
Impact of the Norm Types of ZiL
By default, utilize the LI norm regulate ZiL. However, the choice of norm type for ZiL significantly impacts performance. As demonstrated in Tab. 9, L2 excels in ZCOCO, LI strikes balance between zero-shot and downstream performance, and Smooth LI performs exceptionally well in downstream tasks while maintaining competitive zero-shot performance, even outperforming the LI norm. Regardless of the norm type used, ZiRa remains effective, showcasing its versatility across different norm types.
Table 9: Results with Varying Norm Types of ZiL.
Norm Type
ZCOCO
Avg
hAP
LI
46.06
59.73
52.01
L2
46.26
57.32
51.20
Smooth L.1
46.10
60.07
52.17
Visualization
To more vividly showcase the effectiveness of the proposed ZiRa method, also performed visualization experiment. As shown in Fig. 6, using the pre-trained Grounding DINO directly to leverage its zero-shot detection capabilities allows for detecting many unseen objects, but fails to detect some. Upon incrementally fine-tuning with iDETR using training samples (with five classes per phase and one sample per class), the model will be able recognize objects that undetected by zero-shot detection. However, some objects that can be initially recognized by zero-shot detection are no longer detected. In contrast, after incremental fine-tuning with ZiRa, the images reveal that both familiar and novel objects are detected. This demonstrates ZiRa's 's dual ability to prevent forgetting both pre-trained knowledge and information from downstream tasks.
15

---

Seen Calegories
banana
apple
pear
yellow box
green box
cup
pipe
black_ Looc__spogee
gear
angle aluminum
Zero-shon shot
Incremental
ZIRA
Text Prompt: hhapeppar.banam..ppeeeer box.yellow box. grape..uu.ggearrellow carambola. wrrambol.lemooncherrr..blal hetpheerrrbbbbccssppeeeebbbbooow kiwi fruit. angle aluminum.strawbeery""
Figure 6: Visualization results of both seen categories and unseen categories with Grounding DINO-
based ZiRa.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work discussed in the conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper fully discloses the information needed to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification:
The code is released at http://thhb.ccm///aaintoooooooll
ZiRaGroundingDNN..
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. ec/
oonooicy))y) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. cc/
(CodeSSbmissionn ceiiiisssiopicyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
The paper does not report error bars or statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: Each experiment will take hours on 2 Nvidia RTX 3090 GPUs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : http///eerrpp..eeee httppp////arppppppppppptt/////// d ines?
Answer: [Yes]
Justification:
The authors have reviewed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "jrVAooRWlH", "reviewer": "Reviewer_NuJX", "summary": "This paper deals with the object detection problem in a zero-shot and incremental learning setting. Specifically, Incremental Vision-Language Object Detection (IVLOD) task is proposed to incrementally adapt the pre-trained Vision-Language Object Detection Models (VLODMs) to various specialized domain.   \nThe technical design is Zero-interferences Reparametrizable Adaptation (ZiRA), which includes low learning rate and high learning rate adaptation branches for both vision and language encoder. In addition, an L1-norm is applied on top of the adaptation branch outputs. \n\nExperiments are conducted on the Zero-shot COCO (ZCOCO) and ODinW-13 benchmarks. A series of baselines such as TFA, OW-DETR, CL-DETR, Adapting-tuning (AT), and iDETR were compared. Ablation study on modules, vision or language finetuning were also conducted.", "questions": "Please refer to the above Weaknesses for questions.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "### 1. The overall framework for incremental object detection is reasonable and complete. \n- 1.1 This paper designs a Reparameterizable Dual Branch module for feature adaptation, and specifically devises low learning rate and high learning rate branches to address the learning-forgetting balance. An additional mechanism to gradually merge high learning rate branch into low learning rate branch is also proposed. The idea makes sense and is verified to be effective in both zero-shot COCO and average performance.   \n- 1.2 The L1-norm is reasonable in zero-shot generalizability and seems to be effective according to Fig. 4-5. \n\n### 2. Experimental results verify the effectiveness of the model design\n- On both the zero-shot and the Avg setting, the proposed method outperforms baseline methods. \n- Ablation studies were conducted w.r.t module designs, vision/language adaptation, branches, etc.", "weaknesses": "### 1. The new task of IVLOD is not well desribed and compared with existing one \n- Incremental Vision-Language Object Detection (IVLOD) is claimed to be a new task. However, it is unclear what are the differences between this new task and existing works such as CL-DETR, Ground DINO, iDETR etc. \n- How is the language specially used in IVLOD compared with existing method? \n- Claimed as the first contribution, but no separate detailed description/subsection to compare IVLOD with existing task setting, either verbally or visually. Fig. 1 only compares with the final performance, which is less intuitive. \n### 2. The main contribution is the different learning rates \n- For the Reparameterizable Dual Branch (RDB), although the name is special, the module design is no much different with existing approaches such as LoRA, Adapter, [6], etc. \n- The structure design is not quite novel, but leveraging different learning rates can be a novel point. \n- It is encoraged to have a better survey of relevant adaptation-based methods for incremental learning. \n### 3. Experimental section have flaws\n- ODinW was adopted, but only 13 datasets were selected. How are those 13 datasets selected? Other methods used much more datasets for evaluation, e.g. 35. \n- In Table 1, AT was not used for Full evaluation. CL-DETR/OW-DETR were not used for lower shots evaluation. What is the reason? \n- The prompt in Fig. 6 looks strange. How is the prompt leveraged by the proposed model? \n- In Table 3, why the model performance on zero-shot COCO underperforms Vanilla one by 1.31%? \n- There are many hyper-parameters such as $\\lambda$ for loss, $\\eta$ for learning rate, scaling $s$. How are those parameters selected and tuned? It seems in Table 5 and 6 that the hyper-parameters matters. \n- In Table 8, how about no norm is adopted? \n### 4. Other minors\n- 1st not 1-th in Fig. 1\n- Incorrect usage of double quotes in this paper, e.g.  “Locality Injection\" \n- Line 209 lacks a period", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### 1. The new task of IVLOD is not well described and compared with existing ones\n\n   **Response:** \n   Thank you for your insightful feedback. We recognize the need to clearly define and differentiate Incremental Vision-Language Object Detection (IVLOD) from existing methods. IVLOD is distinct in that it combines the incremental learning of new tasks with the preservation of zero-shot generalization capabilities. Unlike traditional incremental object detection methods such as CL-DETR and iDETR, which focus on learning new tasks incrementally without maintaining zero-shot abilities, IVLOD ensures that models can continue to generalize to unseen categories even after adaptation. Furthermore, while Grounding DINO utilizes natural language prompts for object detection, it does not address incremental task adaptation. In IVLOD, language is used as prompts for the objects to be detected, the same as how it is utilized in Grounding DINO. To enhance clarity, we will include a dedicated subsection comparing IVLOD with these existing methods both verbally (with Descriptive Subsection) and visually (with enhanced Figure 1), thereby providing a comprehensive understanding of its unique contributions. \n\n\n### 2. The main contribution is the different learning rates\n\n   **Response:**\n   We acknowledge the importance of clearly presenting the novel aspects of our work and distinguishing it from existing methods. The key innovation in our approach lies in the use of Zero-interference Loss (ZiL) to constrain the Reparameterizable Dual Branch (RDB), which effectively balances incremental learning performance and zero-shot performance of the VLODM. While methods like LoRA and Adapter introduce additional parameters for efficient fine-tuning, they do not employ a mechanism like ZiL. Our approach uniquely combines RDB with ZiL, addressing the specific challenges of incremental vision-language object detection. We will expand our literature review to include a comprehensive survey of adaptation-based methods for incremental learning, highlighting how our approach builds upon and differentiates itself from these methods.\n\n\n### 3. Experimental section has flaws\n\n   **Response:**\n   1. The selection of the 13 datasets from the ODinW benchmark was based on the same subset used in the GLIP evaluation, ensuring consistency with prior work. We acknowledge that the full ODinW benchmark comprises 35 datasets. Due to computational resource constraints, we limited our evaluation to 13 datasets. However, we plan to extend our evaluation to include all 35 datasets in future work to provide a more comprehensive assessment.\n   \n   2. CL-DETR and OW-DETR  are not specifically designed for lower shot scenarios and do not perform well under low shot settings. These methods are more suited for settings where there is a larger amount of training data available. Comparing them with lower shots evaluation would not provide meaningful insights due to their design and intended use cases. We will provide the results of AT under full evaluation. The results are as follows:\n\n | Shots | Methods | ZCOCO | Avg  | Ae   | Aq   | Co   | Eg   | Mu   | Pa   | Pv   | Pi   | Po   | Ra   | Sh   | Th   | Ve   |\n|-------|---------|-------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|\n| Full  | AT      | 42.30 | 51.14| 23.62| 39.90| 72.32| 65.51| 31.47| 50.48| 60.51| 66.07| 39.09| 53.50| 34.04| 68.07| 60.23|\n\n\n   3. Grounding DINO requires all the prompts to be combined into a single sentence, with periods separating each prompt. For instance, if the prompts are \"dog,\" \"cat,\" and \"car,\" the combined prompt fed into the model would be \"dog. cat. car.\"\n   \n   4. When adapting to new tasks, the zero-shot performance of the original Vision-Language Object Detection Model (VLODM) typically degrades. This is a common issue in incremental learning scenarios and is the primary motivation for proposing the Incremental Vision-Language Object Detection (IVLOD) task. While our method shows a 1.31% decrease in zero-shot COCO performance compared to the Vanilla model, this reduction is significantly less than that observed with other methods. Our approach effectively minimizes the decline in zero-shot capability, demonstrating its efficacy.\n   \n   5. Thank you for highlighting the importance of hyper-parameters in our method. We acknowledge that the presence of multiple hyper-parameters is a limitation of our approach. However, this does not detract from the main contributions of our paper.\n   \n   6. We need to incorporate a norm into the final loss function to ensure effective training and optimization. Without a norm, it would be difficult to train and optimize the model properly.\n\n\n### 4. Other minors\n\n   **Response:** \n   Thank you for pointing out these minor issues. We will make the necessary revisions to address them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tPkZzE6ddU", "reviewer": "Reviewer_jMVp", "summary": "This paper proposes the new problem of incremental visual-language object detection (IVLOD), which aims to preserve zero-shot generalization performance of VLMs, while also adapting to new concepts over time. Authors address IVLOD by proposing the zero-interference reparameterizable adaptation (ZiRa), a light weight branch added to the text and image encoders of VLMs to address this task. Authors demonstrate that their proposed approach outperforms prior work by a considerable margin.", "questions": "Impact of Pre-Training Data on IVLOD Performance. The performance of VLMs is significantly impacted by the data used for pre-training. How might the specific pre-training data used impact the effectiveness of incremental learning? \n\nAdapting to a Large Number of Tasks. As shown in Figure 5, the L1 norm steadily increases as the number of downstream tasks increases. Does this suggests that the proposed approach does not scale well to a large number of tasks?\n\nAdapting to More Than One Class at a Time. Although each class is considered a new \"task\", one may want to adapt to multiple classes at once. How would the proposed approach perform when learning more than one class at a time? \n\nRuntime and Memory Usage. A key benefit of the proposed approach is that adapting to a new task only marginally increases runtime and memory usage. It would be useful to quantify this such that future works can benchmark on this axis as well. \n\nWriting Quality. Although the approach is clear, the grammar and writing quality hinders comprehension. For example \"inputted\" is not a word, and \"input end\" is a confusing phrase. I would encourage authors to further polish the manuscript. In addition, authors should include more descriptive captions in their tables. Notably, I don't believe the full class names in Table 1, 2 (e.g. Ae, Aq, Co) are provided.", "limitations": "Yes, authors highlight that the current implementation is based on the DETR architecture, which may not work for all VLODMs. In addition, authors state that ZiRa is designed for IVLOD, and should be evaluated in the context of other incremental learning tasks.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Problem Motivation. The proposed problem is of significant interest for practical applications, particularly because many vision-language models (e.g. CLIP) are pre-trained on private datasets, making incremental learning with VLMs challenging. \n\nInteresting Insight. Authors highlight that VLMs are robust to noise (cf. Fig 4 and 5), and propose an incremental learning strategy that updates VLMs weights by ensuring that the updated parameters have low L1 norm. It would be interesting to also plot the zero-shot accuracy on COCO for both modes trained with and without ZiL in Fig 5 as the number of learned downstream tasks increases. \n\nSimple and Scalable Approach. The proposed approach adds a small number of tunable parameters to GroundingDINO. Notably, the proposed approach maintains a relatively constant runtime and number of parameters even when adapting to an increasing number of tasks.", "weaknesses": "Differences in Baseline Pre-Training. Although authors benchmark their approach against relevant prior work, many of these methods are not trained on the same scale of pre-training data as GroundingDINO. Given that prior works typically operate in a closed-world, and are pre-trained on much smaller datasets, it is unclear how to make an \"apples-to-apples\" comparison between methods. One strategy might be to reimplement prior works (e.g. TFA and iDETR) using GroundingDINO.\n\nDemonstrate ZiRa with Other Architectures. Although authors claim that their proposed approach can work with any DETR based architecture, they only showcase its performance with GroundingDINO. Given that this work establishes a new problem, it would be useful to extensively evaluate different architectures to provide a \"lay of the land\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Rebuttal for Weaknesses\n\n#### 1. Differences in Baseline Pre-Training\n\n**Response:** Thank you for this important observation. We want to clarify that all compared methods, including TFA and iDETR, were indeed reimplemented using the pre-trained GroundingDINO model. This approach ensures a fair \"apples-to-apples\" comparison, as all methods were evaluated on the same scale of pre-training data. We will revise the manuscript to more clearly convey this setup and ensure there is no confusion regarding the comparability of the methods.\n\n#### 2. Demonstrate ZiRa with Other Architectures\n**Response:** We appreciate the suggestion to evaluate ZiRa with additional architectures. To address this, we have demonstrated ZiRa with OV-DINO [1]. All the method are implemented with the same OV-DINO (swin-T) pre-trained on O365, GoldG, and Cap1M datasets. The results are as follows:\n\n### Table: Performance of Various Methods based on OV-DINO\n| Shots | Methods       | ZCOCO | Avg   | HAP   | Ae    | Aq    | Co    | Eg    | Mu    | Pa    | Pv    | Pi    | Po    | Ra    | Sh    | Th    | Ve    |\n|-------|----------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|\n| Zero  | Original Model | 50.22 | 26.64 | 34.81 | 15.69 | 19.37 | 11.79 | 40.67 | 1.23  | 59.26 | 50.78 | 12.19 | 2.44  | 33.81 | 8.50  | 43.68 | 46.86 |\n| Full  | iDETR          | 37.71 | 46.91 | 41.81 | 24.09 | 30.56 | 49.16 | 68.49 | 26.05 | 67.45 | 61.85 | 30.75 | 29.50 | 66.83 | 27.11 | 71.27 | 56.69 |\n| Full  | CL-DETR        | 34.52 | 45.28 | 39.18 | 23.07 | 27.90 | 46.81 | 67.72 | 25.39 | 66.56 | 59.12 | 28.04 | 29.48 | 64.71 | 25.58 | 68.28 | 56.05 |\n| Full  | AT             | 36.80 | 44.33 | 40.21 | 22.46 | 27.42 | 45.78 | 67.10 | 24.10 | 65.08 | 58.17 | 26.65 | 28.88 | 64.46 | 24.60 | 67.30 | 54.27 |\n| Full  | ZiRa           | 49.07 | 50.21 | 49.63 | 23.37 | 40.43 | 69.02 | 66.21 | 20.74 | 58.51 | 67.74 | 46.71 | 34.13 | 64.62 | 39.44 | 58.01 | 63.81 |\n\n\nThis evaluation demonstrates that ZiRa performs effectively across different DETR-based architectures, validating its versatility. By incorporating ZiRa with OV-DINO, we show that our approach is not limited to GroundingDINO and can be broadly applied to other models.\n\n[1] OV-DINO: Unified Open-Vocabulary Detection with Language-Aware Selective Fusion. Hao Wang, Pengzhen Ren, Zequn Jie, Xiao Dong, Chengjian Feng, Yinlong Qian, Lin Ma, Dongmei Jiang, Yaowei Wang, Xiangyuan Lan, Xiaodan Liang. arXiv, 2024.\n\n### Rebuttal for Questions\n\n#### 3. Impact of Pre-Training Data on IVLOD Performance\n\n**Response:** \n\nThe specific pre-training data used can significantly impact the effectiveness of incremental learning in Vision-Language Models (VLMs). Models pre-trained on diverse and extensive datasets tend to have better generalization capabilities, which can facilitate more effective incremental learning. Conversely, models pre-trained on limited or less diverse datasets might struggle with generalization, leading to more pronounced performance degradation when learning new tasks incrementally. We will include an analysis in the revised manuscript discussing the potential impacts of different pre-training datasets on the effectiveness of incremental learning, reinforcing the importance of diverse and comprehensive pre-training data for robust IVLOD performance.\n\n#### 4. Adapting to a Large Number of Tasks\n\n**Response:** Thank you for this insightful question. Upon examining the evolution curves in Figure 5 more closely, we observe that in the initial stages, ZiL plays a relatively minor role in curbing the growth of the RDB's output norm. However, as the process of accumulating new knowledge progresses, the influence of ZiL becomes more significant, leading to stronger constraint effects. This increase in ZiL's impact leads to a dynamic balance: the interference caused by integrating new knowledge is counteracted by the interference reduction achieved by ZiL. As a result, while the L1 norm of the RDB's output does increase with the number of tasks, it does not rise unboundedly but rather stabilizes after learning a certain number of tasks. Therefore, our proposed approach does manage to scale to a large number of tasks to a certain extent, achieving a balance between the accumulation of new knowledge and the mitigation of interference.\n\n#### 5. Adapting to More Than One Class at a Time\n\n**Response:** In our experimental setup, each \"task\" corresponds to learning an entire dataset, each of which contains more than one class. Thus, our proposed approach inherently handles the learning of multiple classes simultaneously. Our results demonstrate that the proposed method can effectively manage the incremental learning of multiple classes within each dataset, preserving zero-shot generalization capabilities while adapting to new tasks. We will clarify this aspect in the revised manuscript to ensure that it is clear that our approach supports learning multiple classes at a time.\n\n#### 6. Runtime and Memory Usage\n\n**Response:** \n\nThank you for highlighting this crucial aspect. We agree that quantifying the marginal increase in runtime and memory usage when adapting to new tasks would provide valuable insights for future benchmarks. We will conduct additional experiments to measure and report the runtime and memory usage associated with our approach. These results will be included in the revised manuscript, providing clear benchmarks that can be used for comparison in future work.\n\n#### 7. Writing Quality\n\n**Response:** Thank you for your feedback regarding the writing quality. We apologize for any confusion caused by grammatical errors and unclear phrases. We will thoroughly review and polish the manuscript to improve clarity and readability. Additionally, all dataset names (not class names) (e.g., Ae, Aq, Co) in Tables 1 and 2 are fully provided in the Subsection \" Datasets\" of Section \"Experiments Setup\".", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pzn3uCQv5g", "reviewer": "Reviewer_8RRL", "summary": "In order to extend the application of VLODM in a broader domain, the authors propose to combine incremental learning and zero-shot generalization to solve the problem. However, the widespread catastrophic forgetting and maintaining zero-shot generalization capability in incremental learning are two important issues to be considered. To address the challenges in incremental learning, the authors propose to construct a Zero-interference Reparameterizable Adaptation using Reparameterizable Dual Branch.", "questions": "1. The authors introduce the names VLOD and VLODMs in the introduction and cite the literature [11, 17, 20, 24, 39] as the source for the derivation of these names. In these works, tasks such as open vocabulary/open set object detection, grounding, etc. using Vision-Language Model are included. Is the VLOD proposed by the authors consistent with or similar to the task setting of open vocabulary/open set object detection? If it is consistent, why use the term VLOD? If it is similar, the difference between them should be highlighted in the introduction.\n2. In GLIP, the authors give the statement, i.e., \" without seeing any images in COCO during pre-training\". However, in this work, the authors do not discuss or clarify this. Considering the task setting of zero-shot generalization, how do the authors ensure that the images/objects in the test set has not appeared in the pre-training phase?\n3. Based on question 1, where the author claims \"Our work distinguishes itself by performing incremental learning on VLODMs, which are more favorable for open-world problems. \" in related work, is the author's work incremental or zero-shot learning, or a combination of both? What are the similarities and differences between that work and the open-world object detection?\n4. How should the scaling factor “s” be determined in Equation 1? The Reparameterizable Dual Branch structure process contains both HLRB and LLRB, does the size of the scaling factor imply which weight is dominant? If HLRB is dominant, how to ensure that no knowledge forgetting occurs? If LLRB is dominant, how can generalization ability be ensured? Table 7 does not allow a reasonable interpretation of this question.\n5. The authors give Figure 4 and analyze it to prove the validity of ZiL, but how to define \" small-norm \" and why do the authors claim \" the model’s performance is not significantly affected by the addition of small-norm random noise to the input \" when “The average AP on COCO” drops more than 30%?\n6. The authors use the model structure of the Grounding DINO work, in which text representation is an important part of the work. The text template used by the authors is quite different from the one used in Grounding DINO and the authors have not analyzed or explained it, why?", "limitations": "See Waeknesses and Questions.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The authors have innovatively proposed a parameterizable double-branching structure.\n2. The writing logic is clear and the article is well structured.", "weaknesses": "1. The authors do not highlight the need for this work and its similarities and differences with other work in the introduction and related work.\n2. Although the authors have designed an innovative Reparameterizable Dual Branch structure, it is not very interpretable for the problem to be solved.\n3. Although the authors give very rich experimental results in Table 1, the description of the metrics. is not clear enough. What is \" the zero-shot performance on COCO as “ZCOCO\" \"? Is it mAP on the base class or the novel class, or all classes? Is it mAP or AP50, or AP75?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Weakness 1\n\n**Response:** Thank you for this valuable feedback. We acknowledge the importance of clearly stating the need for this work and situating it within the context of existing research. In the revised manuscript, we will enhance the introduction and related work sections to better articulate the motivation for our study and to delineate the similarities and differences between our approach and previous methods.\n\n### Weakness 2\n\n**Response:** Thank you for your feedback. We understand the importance of interpretability in demonstrating the effectiveness of our proposed approach. The core of our solution lies in constraining the Reparameterizable Dual Branch (RDB) with Zero-interference Loss (ZiL). ZiL serves as a regularization mechanism that minimizes interference from newly learned tasks on previously acquired tasks, ensuring that the model maintains its zero-shot capabilities while adapting to new data.\n\n### Weakness 3\n\n**Response:** Thank you for highlighting the need for clearer descriptions of the metrics used in our experiments. The \"ZCOCO\" metric refers to the mean Average Precision (mAP) on all classes in the COCO dataset. Specifically, it represents the mAP@0.5:0.95 (the average of mAP at IoU thresholds from 0.5 to 0.95). This measure provides a comprehensive evaluation of the model's zero-shot performance across all classes on the COCO dataset.\n\nIn the revised manuscript, we will update the description of the metrics in Table 1 to explicitly state that \"ZCOCO\" refers to the mAP on all classes in the COCO dataset, measured at IoU thresholds from 0.5 to 0.95. This clarification will ensure that readers have a precise understanding of the evaluation criteria used in our experiments.\n\n### Questions 1\n\n**Response:** Thank you for pointing out this important aspect. We recognize the need to clarify the relationship between Vision-Language Object Detection (VLOD) and open vocabulary/open set object detection. The VLOD task is indeed consistent with the principles of open vocabulary and open set object detection. VLOD aims to extend the capabilities of Vision-Language Models (VLMs) to detect objects from both known and unknown categories using natural language prompts. The primary reason for using the term VLOD is to emphasize the integration of vision and language in object detection tasks and to highlight the incremental learning aspect, which is a key focus of our work.\n\n### Questions 2\n\n**Response:** Thank you for pointing out the need for clarification regarding zero-shot generalization and the handling of pre-training data. We acknowledge the importance of ensuring that the test set remains unseen during the pre-training phase to validate the zero-shot capabilities of our model.\n\nTo address this, we will clarify that we use the pre-trained weights of Grounding DINO, which are trained on O365, GoldG, and Cap4M datasets. These pre-training datasets are the same as those used for GLIP-T. This ensures that our model has not seen any images from the COCO dataset during pre-training, maintaining the integrity of the zero-shot generalization task.\n\n### Questions 3\n\n**Response:** Thank you for your insightful question. Our work combines both incremental learning and zero-shot learning to address the challenges of open-world object detection (OWOD). Both OWOD and IVLOD need to detect seen and unseen objects. However, in IVLOD, models are pre-trained to detect unknown objects before the incremental learning process begins. In contrast, existing OWOD methods train the model's ability to detect unknown objects concurrently with incremental learning. In addition, OWOD does not further classify unknown objects. In contrast, IVLOD leverages VLODMs to classify unknown objects through language prompts, allowing for more precise identification and categorization of previously unseen objects. Given the superior zero-shot generalization capabilities provided by current vision-language models, we believe that starting with a pre-trained vision-language model for open-world detection represents a more effective approach compared to the traditional OWOD paradigm (which involves incrementally learning new objects while simultaneously learning open-set detection).\n\n### Questions 4\n\n**Response:** We simultaneously insert RDB in both the vision and language components for learning. We believe that the parameters of RDB in different modalities require different update speeds. Therefore, we introduce a learnable scaling factor \"s\" to adjust the parameter update speeds for different modalities. In fact, rather than adjusting the scaling factor directly, we ensure HLRB is dominant by adjusting the learning rates of HLRB and LLRB. We use ZiL to constrain HLRB, mitigating the occurrence of forgetting.\n\n### Questions 5\n\n**Response:** When we refer to \"small-norm\" random noise, we define it based on the standard deviation of the noise added to the input. Specifically, our experiments show that when the norm of the random noise is small (with a standard deviation (std) within 2), the model’s zero-shot AP on COCO remains above 45. Therefore, we claim that the model’s performance is not significantly affected by the addition of small-norm random noise because, within this range, the decrease in performance is minimal and the model retains a high level of accuracy. The observed drop in average AP by more than 30% occurs when the norm of the random noise exceeds this small-norm arrange. The ZiL can ensure that the RDB's output is within the small-norm arrange, therefore protects the the model’s zero-shot AP on COCO. In the revised manuscript, we will ensure that this definition and explanation are clearly stated to avoid any confusion.\n\n### Questions 6\n\n**Response:** We use the same text template as Grounding DINO.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "soseUuat5Y", "reviewer": "Reviewer_BCvk", "summary": "This paper presents Incremental Vision-Language Object Detection (IVLOD), a novel learning task designed to incrementally adapt pre-trained Vision-Language Object Detection Models (VLODMs) to various specialized domains, while simultaneously preserving their zero-shot generalization capabilities for the generalized domain. To this end, Zero-interference Reparameterizable Adaptation (ZiRa) is proposed to tackle IVLOD without incurring a significant increase in memory usage. Experiments on COCO and ODinW-13 datasets demonstrate that ZiRa effectively safeguards the zero-shot generalization ability of VLODMs while continuously adapting to new tasks.", "questions": "1. Is TFA also pre-trained on Objects365 [30], GoldG [17], and Cap4M? I have some doubts about the TFA ZCOCO performance in Table 1 if it's pretrained on COCO. To my best knowledge, TFA only finetunes the last classifier layer for the incremental steps and has demonstrated strong performance on the base classes.\n\n2. How is general IOD (e.g. TFA) tested on Zero-shot COCO? General IOD generally does not support zero-shot inference. Is there any adaptation to achieve it?\n\n3. What learning rate is applied to w/ Rep+ (the first line of Table 2)? Is it the high or low one?", "limitations": "No negative societal impact is expected.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The authors propose RDB and ZiL to generalize the VLODMs to specific domains without losing their general zero-shot capability.\n3. A novel learning task is designed to incrementally adapt pre-trained Vision-Language Object Detection Models (VLODMs) to various specialized domains.", "weaknesses": "1. The key contribution of this paper is RDB and ZiL. However, RDB has been proposed by [1]. Therefore, I would owe the novelty to the differentiated learning rate and ZiL (basically L1 norm). From this point of view, the novelty is somewhat incremental. Importantly, the authors should cite [1] and highlight the distinction.\n\n2. The paper only evaluates the general zero-shot performance after incremental learning. Is zero-shot specific-domain inference possible after learning on few examples in this specific domain?\n\n3. It's suggested to include more comprehensive incremental object detection literature, including but not limited to [2-3].\n\n\n[1] Zhang, Chang-Bin, et al. \"Representation compensation networks for continual semantic segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n[2] Qiao, Limeng, et al. \"Defrcn: Decoupled faster r-cnn for few-shot object detection.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n[3] Yang, Ze, et al. \"Efficient few-shot object detection via knowledge inheritance.\" IEEE Transactions on Image Processing 32 (2022): 321-334.\n\nMinors:\nL178 between stability and plasticity\nL342 typo VOLDM", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weekness 1\n**Reviewer Concern:** The key contribution of this paper is RDB and ZiL. However, RDB has been proposed by [1]. Therefore, I would owe the novelty to the differentiated learning rate and ZiL (basically L1 norm). From this point of view, the novelty is somewhat incremental. Importantly, the authors should cite [1] and highlight the distinction.\n\n**Response:** Thank you for your insightful feedback. We acknowledge the prior work on Reparameterizable Dual Branch (RDB) and the need to clearly distinguish our contributions. We will ensure that [1] is appropriately cited in our manuscript. \n\nThe primary novelty of our work lies in the differentiated learning rate and the introduction of Zero-interference Loss (ZiL), which are key enhancements over the existing RDB framework. These innovations address the specific challenges of Incremental Vision-Language Object Detection (IVLOD) by balancing incremental learning performance and zero-shot capabilities more effectively.\n\n**Revised Explanation in Manuscript:** \n\"Our work builds upon the Reparameterizable Dual Branch (RDB) concept initially proposed by [1]. The key innovations introduced in our approach include the use of differentiated learning rates for the High Learning Rate Branch (HLRB) and Low Learning Rate Branch (LLRB), and the implementation of Zero-interference Loss (ZiL). These enhancements allow for a more effective balance between learning new tasks and preserving previously acquired knowledge, distinguishing our method from prior work.\"\n\n## Weekness 2\n\n**Reviewer Concern:**  The paper only evaluates the general zero-shot performance after incremental learning. Is zero-shot specific-domain inference possible after learning on few examples in this specific domain?\n\n**Response:**  Thank you for your question. In our approach, we do not directly evaluate zero-shot inference in specific domains after learning on a few examples from those domains. Instead, our methodology involves tuning a pre-trained model on the ODinW dataset, which comprises 13 datasets, and then immediately evaluating the model on COCO to assess its zero-shot performance. This evaluation is conducted without exposing the model to any images from COCO during the incremental training phase.\n\n## Weekness 3\n\n**Reviewer Concern:** It's suggested to include more comprehensive incremental object detection literature, including but not limited to [2-3].\n\n**Response:** Thank you for your valuable suggestion. We acknowledge the importance of providing a comprehensive review of the literature on incremental object detection. In the revised manuscript, we will include a detailed survey of relevant work, including but not limited to the references [2-3] you mentioned. \n\n## Question1\n\n**Reviewer Concern:** Is TFA also pre-trained on Objects365 [30], GoldG [17], and Cap4M? I have some doubts about the TFA ZCOCO performance in Table 1 if it's pretrained on COCO. To my best knowledge, TFA only finetunes the last classifier layer for the incremental steps and has demonstrated strong performance on the base classes.\n\n**Response:**  For consistency and fair comparison, all methods, including TFA, were re-implemented and evaluated using the same pre-training datasets (Objects365, GoldG, and Cap4M) and the same pre-trained Grounding DINO model.\n\n## Question 2\n**Reviewer Concern:** How is general IOD (e.g. TFA) tested on Zero-shot COCO? General IOD generally does not support zero-shot inference. Is there any adaptation to achieve it? \n\n**Response:** Thank you for your question. We acknowledge that general Incremental Object Detection (IOD) methods, such as TFA, do not inherently support zero-shot inference. To address this, we re-implemented these general IOD methods based on the Grounding DINO framework. By leveraging Grounding DINO, which supports zero-shot inference, we adapted these methods to enable zero-shot capabilities. Since all the compared baselines support a DETR-like architecture, re-implementing them based on Grounding DINO was feasible and straightforward. This approach ensures that all methods can be evaluated consistently within the same experimental setup.\n\n## Qusetion 3\n**Reviewer Concern:** What learning rate is applied to w/ Rep+ (the first line of Table 2)? Is it the high or low one?\n\n**Response:** Thank you for your question. We used a high learning rate applied to w/ Rep+ (the first line of Table 2). We also tested using a low learning rate and found that the results were similar to those obtained with the high learning rate. We infer that the absolute value of the learning rate does not significantly impact the performance of the RDB. Instead, the differential learning rates play a crucial role, allowing the High Learning Rate Branch (HLRB) and Low Learning Rate Branch (LLRB) to share different learning burdens effectively.\n\nHere is the performance (w/ Rep+) using the low learning rate:\n\n| ZCOCO | Avg  | hAP  | Ae   | Aq   | Co   | Eg   | Mu   | Pa   | Pv   | Pi   | Po   | Ra   | Sh   | Th   | Ve   |\n|-------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|\n| 40.11 | 58.20| 47.49| 31.23| 46.85| 69.40| 66.32| 56.94| 58.35| 64.12| 68.98| 48.12| 63.77| 41.51| 77.02| 64.05|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper was reviewed by four experts in the field. The authors' rebuttal resolved most of the concerns. Reviewers liked the new task proposed by the paper and unanimously agreed to accept the paper. The reviewers did raise some valuable suggestions in the discussion that should be incorporated in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes to the best of their ability.

Why not higher: The reviewers still have shared concerns about the paper's presentation (positioning among related works) and discussions about related works.
Decision: Accept (poster)