Title: Zero-shot Generalizable Incremental Learning for Vision-Language Object Detection
Venue: NeurIPS 2024
Accepted: yes
Keywords: Vision-language object detection

incremental learning
Abstract: This paper presents Incremental Vision-Language Object Detection (IVLOD), a novel learning task designed to incrementally adapt pre-trained Vision-Language Object Detection Models (VLODMs) to various specialized domains, while simultaneously preserving their zero-shot generalization capabilities for the generalized domain. To address this new challenge, we present the Zero-interference Reparameterizable Adaptation (ZiRa), a novel method that introduces Zero-interference Loss and reparameterization techniques to tackle IVLOD without incurring a significant increase in memory usage. Comprehensive experiments on COCO and ODinW-13 datasets demonstrate that ZiRa effectively safeguards the zero-shot generalization ability of VLODMs while continuously adapting to new tasks. Specifically, after training on ODinW-13 datasets, ZiRa exhibits superior performance compared to CL-DETR and iDETR, boosting zero-shot generalizability by substantial $\textbf{13.91}$ and $\textbf{8.74}$ AP, respectively. Our code is available at https://github.com/JarintotionDin/ZiRaGroundingDINO.

Zero-shot Generalizable Incremental Learning for Vision-Language Object Detection

Abstract

This paper presents Incremental Vision-Language Object Detection (IVLOD), a novel learning task designed to incrementally adapt pre-trained Vision-Language Object Detection Models (VLODMs) to various specialized domains, while simultaneously preserving their zero-shot generalization capabilities for the generalized domain. To address this new challenge, we present the Zero-interference Reparameterizable Adaptation (ZiRa), a novel method that introduces Zero-interference Loss and reparameterization techniques to tackle IVLOD without incurring a significant increase in memory usage. Comprehensive experiments on COCO and ODinW-13 datasets demonstrate that ZiRa effectively safeguards the zeroshot generalization ability of VLODMs while continuously adapting to new tasks. Specifically, after training on ODinW-13 datasets, ZiRa exhibits superior performance compared to CL-DETR and iDETR, boosting zero-shot generalizability by substantial 13.91 and 8.74 AP, respectively. Our code is available at https://github.com/JarintotionDin/ZiRaGroundingDINO .

1 Introduction

Object detection has achieved remarkable strides in recent years [2, 44]. However, most object detection models are typically trained to predict a predefined, closed set of categories [32, 2], constraining their prospect in real-world applications. Recent research endeavors [12, 27, 45, 19, 22] focus on developing intelligent systems capable of detecting objects specified by natural language inputs, giving rise to the field of Vision-Language Object Detection (VLOD) (also named Open Vocabulary Object Detection), also referred to as Open Vocabulary Object Detection. In this framework, the models are termed Vision-Language Object Detection Models (VLODMs). By incorporating natural language inputs, VLODMs can recognize a much broader set of visual concepts beyond a fixed category set, resulting in exceptional zero-shot generalizability.

Despite VLODMs' great zero-shot recognition ability in the general domain, VLODMs often exhibit suboptimal performance in more specialized domains, such as identifying aquatic organisms in aquariums or interpreting remote sensing images from aerial drones. In real-world scenarios, the necessity to adapt VLODMs to various unforeseen downstream tasks remains crucial in order to attain the desired accuracy, as highlighted in recent research [19, 45].

A straightforward idea is to adapt different individual VLODMs to different downstream tasks. However, a general agent often needs to simultaneously recognize objects from diverse tasks, each

1-th Task: Rabbits

Zero-shot

General IOD

IVLOD

?

e

1-th Task AP:64.72

1-th TaskAP:72.01

1-th TaskAP:72.55

60

Incrementally

2-nd Task: Aquarium

Zero-shot

General IOD

IVLOD

1-th Task AP:64.72

1-th Task AP:71.98

1-th Task AP:71.72

2-ndTaskAP:20.75

2-nd TaskAP:46.11

2-nd TaskAP:47.73

Zero-shotCOCO

Zero-shot

General IOD

IVLOD

Zero-shot AP:47.37

Zero-shot AP:32.15

Zero-shot AP:46.06

learned at distinct times. Moreover, in a dynamically changing environment, equipping VLODMs with the ability of incremental learning is paramount [20, 31], considering that downstream tasks typically arrive in a streaming manner and incremental learning paradigm better aligns with the learning and cognition processes of a general intelligent agent.

To endow VLODMs with the incremental learning ability, a naive idea is applying existing methods [10, 23] designed for detector on closed set directly. However, from our observations, it will weaken the excellent zero-shot generalization ability of VLODMs to detect unseen categories. To highlight the distinctiveness of incremental learning for VLODMs, we believe it is necessary to introduce a new task, Incremental Vision-Language Object Detection (IVLOD), which stresses the importance of maintaining the zero-shot generalization ability of VLODMs while adapting to various downstream tasks incrementally.

To further demonstrate the uniqueness of the IVLOD task, we have compared three different variants of adapting VLODMs to multiple downstream tasks: zero-shot learning ( i.e ., no adaptation), conventional incremental learning (applying CL-DETR [23] to VLODM), and zero-shot generalizable incremental learning ( i.e ., the IVLOD task) in Fig. 1. Unlike general incremental object detection (general IOD, ref. second column), IVLOD is characterized by preserving the original zero-shot performance of VLODM while also performing incremental learning. From this comparison, IVLOD faces two major challenges. One is the issue known as catastrophic forgetting [29], where the performance on previously learned tasks may sharply decline when new tasks are introduced. The other challenge is to maintain the zero-shot generalizability of VLODMs while learning new downstream tasks.

From a unified perspective, both challenges can be attributed to the forgetting problem. Technically, two kinds of methods can tackle this issue: selecting exemplar samples from past datasets for replaying [31, 23] or maintaining a duplicate model to facilitate knowledge distillation [20, 23]. In order to prevent forgetting, the replaying-based approach necessitates preserving enough examples to ensure the representativeness of the samples for replaying, but it is hard to do this with a limited-size memory, specifically when the model is pre-trained on a large-scale vision-language dataset. At the same time, current knowledge distillation techniques do not adequately focus on preventing the forgetting of pre-training knowledge, rendering them suboptimal for IVLOD tasks. Furthermore, they often need to store an entire model copy, which requires a large memory budget. Such storage demands are often untenable, particularly in scenarios that use resource-constrained edge devices.

To effectively address IVLOD's challenges while improving memory efficiency, we introduce a novel approach named Zero-interference Reparameterizable Adaptation (ZiRa). ZiRa retains the parameters of the original model and introduces a parallel dual branch structure named Reparameterizable Dual Branch (RDB) designed for efficient tuning on downstream tasks. The RDB structure is

reparameterizable, allowing the model to adapt to downstream tasks without increasing memory usage. Furthermore, the RDB structure serves a dual purpose-it achieves the branch labor division for downstream continual learning, protecting the learned knowledge from being excessively overwritten, and more crucially, it lays the structural foundation for the key novel element of ZiRa named Zero-interference Loss (ZiL).

ZiL simultaneously penalizes the output norms of both the entire RDB and a high-learning-rate branch within the RDB, guiding the RDB to learn in a direction that protects both knowledge learned from pre-training and downstream tasks. The idea of ZiL comes from the following insight: During pre-training, the model is trained with visual-language inputs that contain significant noise, and the model has learned how to handle this noise. As a result, the pre-trained VLODMs have a certain degree of robustness to the input, and they can handle a certain range of interference without their performance being affected. ZiL ensures that the fine-tuned residual input of RDB is of small norm, thereby guaranteeing that the model's original performance. At the same time, for adaptation, only a small adjustment to the input is needed to learn new concepts. Although ZiL constrains the input norm of the RDB, it still allows the RDB to acquire sufficient downstream knowledge. As a result, ZiL effectively addresses the two challenges of IVLOD at the same time. Crucially, ZiRa accomplishes this without necessitating the duplication of the entire model or any exemplars, thereby ensuring efficient utilization of memory.

Compared to existing methods [34, 23], ZiRa leverages the advantages offered by VLODMs more fully through a principled design ( i.e ., the ZiL based on RDB), offering a superior and more effective solution without the need for excessive additional memory. The contributions of this paper can be summarized as follows:

We introduce Incremental Vision-Language Object Detection (IVLOD), a novel learning task that incrementally adapts VLODMs to multiple specialized domains while preserving zero-shot generalization ability.

We present a novel approach called ZiRa to tackle IVLOD challenges. Based on the reparameterizable parallel tuning structure of RDB, ZiRa introduces ZiL to simultaneously minimize interference with zero-shot performance and prevent forgetting on downstream tasks. Notably, ZiRa makes these achievements in a memory-efficient manner.

We conduct comprehensive experiments on COCO and ODinW-13 datasets, showcasing the effectiveness of our method. For instance, ZiRa outperforms CL-DETR [23] and iDETR [10] by a significant margin, making a 13.91 and 8.74 improvement in zero-shot AP on COCO, respectively, while offering competitive results on downstream tasks incremental learning.

2 Related Work

Vision-Language (or Open Vocabulary) Object Detection. Vision-Language (or Open Vocabulary) Object Detection Models combine natural language descriptions with object detection in images. Early research in this area primarily focused on grounding textual phrases to image regions, such as grounding referring expressions [28, 39, 4] and visual question answering [1, 25, 42]. More recent studies have developed unified and scalable approaches for simultaneous object detection and grounding tasks. For instance, GLIP [19] unifies phrase grounding and object detection tasks by treating object detection as context-free phrase grounding and treating phrase grounding as a contextualized object detection task. This approach enables the integration of multi-layered textual information into the detection model. Grounding DINO [22], based on the DINO [44], employs a DETR-like [2] architecture that uses a Transformer as the detector, which allows for unified processing of image and language data and offers the outstanding capacity to utilize large-scale datasets. Based on Grounding DINO, our work further addresses the newly introduced IVLOD tasks.

Incremental Learning and Incremental Object Detection. Incremental learning, also known as continual learning or lifelong learning, tackles the challenge of learning new tasks without forgetting previous ones. Incremental learning has given rise to two primary strategies, memory replay, and knowledge distillation, to tackle its inherent challenges [20, 31, 3, 5, 38, 37, 40]. For example, Learning without Forgetting (LwF) [20] employs knowledge distillation to maintain performance on old tasks while learning new ones. Another notable approach, iCaRL [31], combines exemplar memory

and a nearest-mean-of-exemplars classification method for incremental learning. Incremental object detection, a specialized subfield within incremental learning, focuses on improving the adaptability of object detection models to new object categories while maintaining performance on previously learned categories [34, 17, 30, 41, 11, 10, 23]. ILOD [34] was the first to introduce incremental object detection. More recently, CL-DETR [23] effectively combined knowledge distillation and exemplar memory techniques to enhance incremental object detection using Transformer-based models. Our work distinguishes itself by performing incremental learning on VLODMs, which are more favorable for open-world problems. Besides the need to continually adapt across multiple specialized tasks, it is also important to preserve their zero-shot generalization ability. To the best of our knowledge, our work is the first to pose and tackle IVLOD tasks.

Open-World Object Detection. Open-World Object Detection (OWOD) [16, 13, 26] also targets the detection of both seen and unseen objects, but it involves simultaneous learning of new objects and detecting unknown objects during the incremental learning process. This means that as new tasks are introduced, the model is trained to recognize new objects while also developing its ability to detect unknown objects. In contrast, Incremental Vision-Language Object Detection (IVLOD) pre-trains models to detect unknown objects before the incremental learning process begins. The focus of IVLOD during incremental learning is on preserving the model's ability to detect these unknown objects while adapting to new tasks. Given the superior zero-shot generalization capabilities of current vision-language models, we believe that starting with a pre-trained vision-language model for IVLOD is a more effective approach compared to the traditional OWOD paradigm.

Reparameterization. Reparameterization techniques optimize model inference speed by transforming the training-time architecture into an equivalent, simplified form for inference [8, 9, 7]. RepVGG [9] exemplified this approach by employing 3 × 3 and 1 × 1 convolutional branches for training, which are streamlined to 3 × 3 convolutional branches and ReLU activations during inference. RepMLPNet [7] introduced 'Locality Injection" to merge trained parameters of parallel convolutional kernels into fully connected layers, enhancing MLP-based model performance. Our work builds upon the Reparameterizable Dual Branch (RDB) concept initially proposed by [43]. The key innovations introduced in our approach include the use of differentiated learning rates for the High Learning Rate Branch and Low Learning Rate Branch, and the implementation of Zero-interference Loss (ZiL). These enhancements allow for a more effective balance between learning new tasks and preserving previously acquired knowledge, distinguishing our method from prior work.

3 Methodology

In this section, we systematically present our approach. We begin with an overview (Sec. 3.1), and then introduce Zero-interference Reparameterizable Adaptation (ZiRa), which includes the Reparameterizable Dual Branch (RDB) (Sec. 3.2) and the Zero-interference Loss (ZiL) (Sec. 3.3).

3.1 Overview

The framework of our model for addressing the IVLOD task is illustrated in Fig. 2, with the selected VLODM being Grounding DINO [22]. Other VLODMs with a similar structure are also suitable. Specifically, the VLODM takes image features and category text prompts as inputs and produces class-agnostic proposals whose features are closely aligned with the text prompts. These text prompts are also refined to obtain text classification features by fusing the image features. Subsequently, the model classifies the class-agnostic proposals based on the cosine similarity between the visual features of proposals and the text classification features. These VLODMs are pre-trained on extensive grounding datasets annotated with detailed human language and object boxes, which equips them with great zero-shot recognition capabilities. To address the challenges of IVLOD, we propose ZiRa which is based on the structure of RDB and constrains the output of RDB by ZiL.

3.2 Reparameterizable Dual Branch

Adaption on Both Language and Vision Sides. Fine-tuning the entire model is a common method for adapting the original model to downstream tasks. However, it has been demonstrated that tuning only a few newly introduced parameters in a pre-trained model is more effective [14, 15], while full fine-tuning often leads to significant forgetting. In view of this, we introduce additional parallel

Lzil

Tunable

LinearRDB

Cat.

Frozen

Text

Pre-trained

Dog.

Lcls

Backbone

*Linear

Flower.

Vision-

Language

Prompts

Text Classification

Text

Object

Features

Detector

Image

Pre-trained

Lloc

Backbone

Project Conv

ConvRDB

Image

Lzil

Class-agnosticProposals

branches on both the language and vision sides of the VLODM to adapt the model for sequential downstream tasks.

As shown in Fig. 2, these additional branches are termed the Reparameterizable Dual Branch (RDB). On the language side, we integrate an RDB alongside a pre-trained linear layer positioned between the detector and the language backbone network, aiming to learn the new high-level semantic concepts. On the visual side, we also connect a supplementary RDB in parallel with a pre-trained convolutional layer located between the visual backbone and the detector, aiming to learn the visual features of new classes. It's pivotal to adapt on both sides, as these two sides capture distinct structures of vital knowledge and the lack of either will lead to insufficient adaptation to downstream tasks.

Dual Branch Structure within the RDB. Within the RDB, illustrated in Fig. 3, our design features a dualbranch structure, comprising the Low-learning rate Branch (LLRB) and the High-learning rate Branch (HLRB). The HLRB is set to a high learning rate for rapid task adaptation, whereas the LLRB employs a more conservative learning rate. Specifically, the LLRB is set at η ( 0 < η < 1 ) times the learning rate of the HLRB. This differentiation in learning rates brings division of labor between LLRB and HLRB, especially when combined with our next strategy that reparameterizes the HLRB into the LLRB after learning each task. The LLRB's low learning rate can help maintain downstream task knowledge, while the HLRB's high learning rate can help swift adaptation to new tasks. This division protects the knowledge stored in the LLRB from being excessively overwritten when incrementally learning downstream tasks, achieving a better balance between stability the plasticity, while the single branch or naive dual branch structure can not take advantage of this division mechanism.

ppv

S

Scaling

ZiL for

HLRB

S

Frozen

Pre-

trained

LLRB

Branch

HLRB

Reparameterable DualBranch(RDB)

Reparameterization after each task

Reparameterization after all tasks

Specifically, the collective output of the RDB x rdb , is formulated as:

where x is the input and s is a learnable scaling factor.

For the language side, the RDB is structured as two parallel linear layers, and its output x l rdb is expressed as:

where W l hlrb and W l llrb represent the weights of HLRB and LLRB, respectively, while b l hlrb and b l llrb denote their corresponding biases.

For the vision side, the RDB is structured as two parallel convolution layers, and its output x v rdb is illustrated as:

Here, W v hlrb and W v llrb stand for the kernel of HLRB and LLRB, respectively, while b v hlrb and b v llrb represent their corresponding biases. ⊗ denotes the convolution operation.

Reparameterization from the HLRB to the LLRB. With each new task, ZiRa resets the HLRB parameters to zero and, after completing learning on a new task, merges the trained parameters from the HLRB into the LLRB. This approach not only allows the LLRB to incrementally acquire new knowledge but also avoids the linear increase in the number of additional branches as the number of tasks increases, effectively managing memory consumption. Moreover, it offers a structural prerequisite for ZiL to prevent forgetting on downstream tasks by penalizing the output norm of HLRB, which will be detailed in the next section.

On the language side, ZiRa combines the HLRB and LLRB into a consolidated LLRB with new weights W l hlrb · s + W l llrb and new biases b l hlrb · s + b l llrb , given by:

Similarly, on the vision side, the HLRB and LLRB are merged into a singular LLRB, as depicted by:

where W v hlrb · s + W v llrb and b v hlrb · s + b v llrb can be considered as the new kernel and new biases.

Single Branch for Inference. Sub-branches inside the RDB like the LLRB and HLRB can increase computational overhead. We efficiently address this by reparameterizing all sub-branches in RDB and the frozen pre-trained branch into a new one for inference. Since the frozen pre-trained branch is also a linear structure, after learning all the tasks, we can create a new branch that merges the frozen pre-trained branch and the RDB via reparameterization. During inference, we exclusively use this newly consolidated branch, but we still retain the parameters frozen pre-trained branch and the RDB for future learning. This approach significantly reduces the computational cost during inference, as only one branch is required for prediction

3.3 Zero-interference Loss

Protect Zero-shot Generalizability with ZiL. Preventing performance degradation on the original pre-training domain can be viewed as preventing the VLODM from forgetting the original pre-training domain knowledge. Two common methods to avoid forgetting is to reserve some old samples for replaying or distill the current model from the previous ones. However, preserving old samples and storing the previously trained model requires significant additional memory. On the contrary, Zero-interference Loss (ZiL) can prevent forgetting in a memory-efficient manner.

Specifically, ZiL penalizes the norm of the whole RDB's output (ref. Fig. 3), aiming at safeguarding the zero-shot performance of VLODM. The ZiL for the RDB, L rdb , is defined as follows:

Here, L 1 corresponds to the L 1 norm.

Why ZiL works. The effectiveness of ZiL can be explained as follows. First, the VLODM's input (What we fine-tuned in this paper) is robust to small-norm random perturbations, which have little impact on the model's performance. To verify this, we add Gaussian noise to the input of the pretrained VLODM's detector and observe the model's performance. As shown in Fig. 4, the model's performance is not significantly affected by the addition of small-norm random noise to the input.

Second, the ZiL ensures that the fine-tuned input of the VLODM has a small norm additional term. To further investigate the effectiveness of ZiL, we plot curves illustrating the evolution of the RDB's output norm throughout the IVLOD process, as shown in Fig. 5. It is evident that training the RDB with ZiL leads to a substantial reduction in the output norm of the RDB on COCO, compared to training the RDB without ZiL. This preliminary observation strongly suggests that ZiL effectively preserves the zero-shot generalizability of VLODMs. Examining the evolution curves in Fig. 5 more closely, we observe that in the initial stages, ZiL plays a relatively minor role in curbing the growth

2

4

6

8

10

12



0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5









2

4

6

8

10

12



1

0

1

2

3









of the RDB's output norm. However, as the process of accumulating new knowledge progresses, the influence of ZiL becomes more significant, leading to stronger constraint effects. This increase in ZiL's impact leads to a dynamic balance: the interference caused by integrating new knowledge is counteracted by the interference reduction achieved by ZiL, which is reflected in the fact that as the number of tasks increases, the norm of the RDB's output doesn't rise unboundedly but rather stabilizes after learning a certain number of tasks. This dynamic balance is crucial for maintaining the zero-shot generalizability of VLODMs.

Prevent forgetting on downstream tasks with ZiL. The L rdb serves primarily to preserve the zero-shot generalization capability, leaving the challenge of downstream task forgetting unresolved. By exploiting the advantage of RDB, we address this challenge by further applying ZiL on HLRB, which is computed as:

where s is a learnable scaling parameter.

The L hlrb can effectively protect the knowledge that is learned from previous tasks and encompassed within the combined LLRB and the pre-trained branch. Specifically, given that the knowledge learned from the previous task has already been reparameterized into LLRB and LLRB learns at a lower learning rate, the total output of the branches previously trained can be regarded as LLRB ( x ) + PTB ( x ) , where PTB is the pre-trained branch. The total output of the current branches is LLRB ( x ) + HLRB ( x ) · s + PTB ( x ) , which has an additional component HLRB ( x ) · s compared to the output from the branches trained on the previous task. Hence, optimizing L hlrb equates to penalizing the discrepancy in the outputs between the models trained on the previous and the current task, effectively guiding the HLRB to learn new tasks without much downstream task forgetting.

Total Optimization Objective. For a general VLODM, we continue to employ its original training loss. To prevent forgetting, we incorporate an additional loss term, L zil . Taking Grounding DINO as an example, the total loss function becomes:

where L zil = L rdb + L hlrb is the proposed ZiL, λ is its loss weight, and L cls and L loc are the contrastive classification loss and localization loss of Grounding DINO. Throughout the training process, the entire original VLODM is frozen, except for the RDB.

4 Experiments

4.1 Setup

Datasets. We conduct our experiments on the COCO [21] datasets and the 'Object Detection in the Wild (ODinW)' [18] benchmark. ODinW is a more challenging benchmark designed to test model performance under real-world scenarios. It comprises numerous sub-datasets from various domains for evaluation, such as Thermal (to detect objects in heat map images) and Aquarium (to detect different marine animals). Following GLIP [19], we use ODinW-13 datasets, they are labeled as Ae

(Aerial Maritime Drone), Aq (Aquarium), Co (Cottontail Rabbits), Eg (Egohands), Mu (Mushrooms), Pa (Packages), Pv (Pascal VOC), Pi (Pistols), Po (Pothole), Ra (Raccoon), Sh (Shellfish), Th (Thermal Dogs and People), Ve (Vehicles). The 13 sub-datasets of ODinW-13 are trained sequentially, one by one, and are tested after all sub-datasets have been trained.

Metrics. As COCO is a dataset within the realm of everyday life, we use zero-shot COCO performance as a metric of the model's zero-shot generalization ability across general domains. Additionally, we provide the average performance on the downstream ODinW-13 benchmark, with performance calculations made only after the model has been sequentially trained on all sub-datasets. We denote the zero-shot mAP on COCO as 'ZCOCO' and the average mAP on ODinW-13 downstream tasks as 'Avg' .

Implementation Details. In this paper, unless otherwise specified, we use the Grounding DINO [22] with the Swin-Tiny [24] backbone as the default original VLODM in our experiments. This Grounding DINO is pre-trained on Objects365 [33], GoldG [19], and Cap4M [19]. We also use OV-DINO [35] with the Swin-Tiny backbone as an additional validation to demonstrate ZiRa's ability to generalize effectively across different DETR-based architectures. This OV-DINO model is pre-trained on the Objects365 [33], GoldG [19], and Cap1M [35] datasets. The learning order of the ODinW-13 sub-datasets is randomly shuffled. All results are obtained after running experiments three times with different random seeds. The hyperparameter experiments are also conducted on the ODinW-13 dataset and COCO, and the results are shown in the appendix.

Our proposed method is implemented with PyTorch and trained on two Nvidia RTX 3090 GPUs. Each downstream task is trained for a total of two epochs with a batch size of 2. For Grounding DINO, we employ an initial learning rate of 10 -3 , which decays to 0.1 times the original value after the first epoch to ensure effective convergence. For OV-DINO, we employ an initial learning rate of 10 -4 , which also decays to 0.1 times the original value after the first epoch to ensure effective convergence. AdamW is used as the optimizer, and the weight decay is 10 -4 . At the beginning of IVLOD, we initialize the parameters of the total RDB as zero.

4.2 Comparison with Existing Methods

In our evaluation, we compare ZiRa with existing Incremental Object Detection (IOD) methods, with a specific focus on methods based on Detection Transformers (DETR) [2], as our approach falls into this category. The baseline methods we consider include:

TFA [36]: A classical linear-probing-based incremental few-shot object detection baseline.

OW-DETR [13]: An open-world object detection approach based on DETR, primarily addressing forgetting through exemplar replaying, which necessitates additional memory to store exemplars.

CL-DETR [23]: A DETR-based method employing refined knowledge distillation and exemplar replaying. This method requires a complete model copy and exemplars, incurring significant extra memory costs.

iDETR [10]: An incremental few-shot object detection method specifically focusing on tuning the projection layer of DETR-like detectors via knowledge distillation and selfsupervision.

Adapting-tuning (AT) [14]: A parameter-efficient adaptation method that has demonstrated its effectiveness in incremental few-shot object detection [6].

We first compare the above methods implemented on the Grounding DINO model. Experiments are conducted under both few-shot and full-shot settings. Since CL-DETR and OW-DETR are not designed for few-shot IOD, we only compare them under the full-shot setting. The results presented in Tab. 1 demonstrate that ZiRa consistently outperforms existing IOD methods in terms of 'Avg' on downstream tasks. Moreover, ZiRa exhibits remarkable performance in preserving the zero-shot generalization ability of VLODMs under both the few-shot and full-shot settings. In particular, ZiRa surpasses CL-DETR and iDETR by substantial margins, with improvements of 13.91 and 8.74 AP under the full-shot setting in terms of 'ZCOCO', respectively.

We then compare ZiRa with existing methods implemented on the OV-DINO model. The results in Tab. 2 show that ZiRa outperforms existing IOD methods in terms of "Avg" on downstream tasks. ZiRa also demonstrates excellent performance in preserving the zero-shot generalization ability of VLODMs under the full-shot setting. Specifically, ZiRa surpasses CL-DETR and iDETR by significant margins, with improvements of 14.55 and 11.36 AP in the full-shot setting for "ZCOCO," respectively.

These results emphasize ZiRa's effectiveness in maintaining the zero-shot AP and its potential as a superior solution for IOD. Distinctively, ZiRa, unlike CL-DETR and OW-DETR, requires minimal extra memory for a few branches without storing image exemplars and the model copy, making it more memory-efficient than other IOD methods.

4.3 Ablation Study

Main Components of ZiRa. Our approach combines the ZiL and the RDB to counteract forgetting in both the general domain and downstream tasks. In this study, we analyze the impact of these components. We utilize 'Rep+' to denote whether to reparameterize the HLRB into LLRB after each task as well as using differentiated learning rates. Please note that even when we do not use 'Rep+', we still retain the dual-branch structure like RDB. ZiL encompasses the RDB's ZiL ( L rdb ) and HLRB's ZiL ( L hlrb ), here we separate them and study their impact separately. We also evaluate performance with the 'hAP' metric, which is the harmonic mean of 'ZCOCO' and 'Avg'.

The results in Tab. 3 provide valuable insights. First, the comparison between the first and second rows demonstrates that 'Rep+' can mitigate forgetting on both pre-training and downstream tasks. This effect comes from the labor division between branches that are built on differentiated learning rates and reparameterization. Second, the comparison between the first row and third rows highlights that ZiL ( L rdb + L hlrb ) significantly enhances the 'ZCOCO'. However, it cannot address forgetting on downstream tasks without 'Rep+', and it even reduces the 'Avg' since it also limits the plasticity

of the RDB. The best results lie in the fourth row, underscoring that combining 'Rep+' and ZiL can effectively mitigate forgetting on both pre-training and downstream tasks to a greater extent. Conversely, the results in the last row illustrate that using ( L rdb ) alone does not optimally address forgetting on downstream tasks. Comparing the second row and the last row, we can find that L hlrb can improve both 'ZCOCO' and 'Avg', showcasing that L hlrb can mitigate forgetting both on downstream tasks and pre-training, but without L rdb , using L hlrb alone can not achieve the best 'hAP'.

Learning on Different Modalities with ZiRa. We carried out a series of experiments to investigate the effects of learning only on the vision side using parallel convolution layers (indicated as 'V') and only on the language side using parallel linear layers (denoted as 'L'). The results in Tab. 4 demonstrate that ZiRa can function independently on either the language or vision side, and it can also effectively function when simultaneously applied to both modalities, highlighting the generality of ZiRa's impact. Since the language and vision side learning acquires knowledge in distinct structures, neglecting either can hinder optimal learning. Consequently, simultaneously learning on both sides can outperform individual tuning on either the vision or language sides when employing ZiRa to avoid forgetting.

Branch Structure. To study the impact of the additional branch structure introduced in VOLDM, we compared the results of introducing a single branch (denoted as SB), dual branches (denoted as DB), and RDB (DB + Rep+). Besides RDB, both the output norm of SB and DB are also penalized with ZiL. As we can see in Tab. 5, the SB structure, due to its inability to introduce inter-branch division of labor, falls short in its ability to prevent forgetting in downstream tasks compared to the RDB. Moreover, we can see that merely introducing DB with ZiL cannot prevent forgetting on downstream tasks, on the contrary, it brings worse forgetting due to increased plasticity. RDB not only incorporates the mechanism of branch labor division but also fully utilizes l hlrb to learn to minimize the interference of HLRB, thereby more effectively preventing the forgetting on downstream tasks.

5 Conclusion

This paper presents a novel learning task, Incremental Vision-Language Object Detection (IVLOD), which aims to continually adapt Vision-Language Object Detection Models (VLODMs) to multiple specialized domains while preserving VLODMs' zero-shot generalization ability. To solve this new task in a memory-efficient way, we introduce Zero-interference Reparameterizable Adaptation (ZiRa). ZiRa inserts Reparameterizable Dual Branche (RDB) on the both language side and vision side of the VLODM and constrains the RDB by Zero-interference Loss (ZiL) to protect the original generalizability of VLODMs and prevent forgetting on downstream tasks at the same time. Notably, ZiRa eliminates the need for saving the entire model copy for distillation or maintaining exemplars for replaying, which makes it a highly memory-efficient method. Extensive experiments conducted on the COCO and ODinW-13 datasets showcase the superiority of ZiRa for IVLOD.

Limitations. While ZiRa has demonstrated its effectiveness in IVLOD tasks, there are still some limitations. First, the current implementation of ZiRa is based on the DETR architecture, which may not be optimal for all VLODMs. Second, the current ZiRa method is designed for the IVLOD task, and its generalization to other incremental learning tasks remains to be explored.

Acknowledgments

We would like to express our deepest gratitude to the anonymous reviewers and the community for their valuable feedback, which helped us enhance the quality of this work. Additionally, we are grateful to our colleagues for their insightful discussions and support throughout the development of this project.

References

Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang, L.: Bottom-up and top-down attention for image captioning and visual question answering. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018. pp. 6077-6086 (2018)

Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J. (eds.) Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I. Lecture Notes in Computer Science, vol. 12346, pp. 213-229 (2020)

Cermelli, F., Mancini, M., Bulò, S.R., Ricci, E., Caputo, B.: Modeling the background for incremental learning in semantic segmentation. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. pp. 9230-9239 (2020)

Deng, C., Wu, Q., Wu, Q., Hu, F., Lyu, F., Tan, M.: Visual grounding via accumulated attention. IEEE Trans. Pattern Anal. Mach. Intell. 44 (3), 1670-1684 (2022)

Deng, J., Hu, J., Zhang, H., Wang, Y.: Incremental prototype prompt-tuning with pre-trained representation for class incremental learning. arXiv abs/2204.03410 (2022)

Deng, J., Zhang, H., Hu, J., Zhang, X., Wang, Y.: Class incremental robotic pick-and-place via incremental few-shot object detection. IEEE Robotics Autom. Lett. 8 (9), 5974-5981 (2023)

Ding, X., Chen, H., Zhang, X., Han, J., Ding, G.: Repmlpnet: Hierarchical vision MLP with reparameterized locality. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. pp. 568-577 (2022)

Ding, X., Guo, Y., Ding, G., Han, J.: Acnet: Strengthening the kernel skeletons for powerful CNN via asymmetric convolution blocks. In: 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019. pp. 1911-1920 (2019)

Ding, X., Zhang, X., Ma, N., Han, J., Ding, G., Sun, J.: Repvgg: Making vgg-style convnets great again. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021. pp. 13733-13742 (2021)

Dong, N., Zhang, Y., Ding, M., Lee, G.H.: Incremental-detr: Incremental few-shot object detection via self-supervised learning. In: Williams, B., Chen, Y., Neville, J. (eds.) Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023. pp. 543-551 (2023)

Ganea, D.A., Boom, B., Poppe, R.: Incremental few-shot instance segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021. pp. 1185-1194 (2021)

Gu, X., Lin, T., Kuo, W., Cui, Y.: Open-vocabulary object detection via vision and language knowledge distillation. In: The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 (2022)

Gupta, A., Narayan, S., Joseph, K.J., Khan, S., Khan, F.S., Shah, M.: OW-DETR: open-world detection transformer. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. pp. 9225-9234 (2022)

Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., Gelly, S.: Parameter-efficient transfer learning for NLP. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA. Proceedings of Machine Learning Research, vol. 97, pp. 2790-2799 (2019)

Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: LoRA: Low-rank adaptation of large language models. In: The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 (2022)

Joseph, K.J., Khan, S.H., Khan, F.S., Balasubramanian, V.N.: Towards open world object detection. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021. pp. 5830-5840 (2021)

Joseph, K.J., Rajasegaran, J., Khan, S.H., Khan, F.S., Balasubramanian, V.N.: Incremental object detection via meta-learning. IEEE Trans. Pattern Anal. Mach. Intell. 44 (12), 9209-9216 (2022)

Li, C., Liu, H., Li, L.H., Zhang, P., Aneja, J., Yang, J., Jin, P., Lee, Y .J., Hu, H., Liu, Z., Gao, J.: ELEV ATER: A benchmark and toolkit for evaluating language-augmented visual models. arXiv abs/2204.08790 (2022)

Li, L.H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J., Chang, K., Gao, J.: Grounded language-image pre-training. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. pp. 10955-10965 (2022)

Li, Z., Hoiem, D.: Learning without forgetting. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV. Lecture Notes in Computer Science, vol. 9908, pp. 614-629 (2016)

Lin, T., Maire, M., Belongie, S.J., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: common objects in context. In: Fleet, D.J., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V. Lecture Notes in Computer Science, vol. 8693, pp. 740-755 (2014)

Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Li, C., Yang, J., Su, H., Zhu, J., Zhang, L.: Grounding DINO: marrying DINO with grounded pre-training for open-set object detection. arXiv abs/2303.05499 (2023)

Liu, Y., Schiele, B., Vedaldi, A., Rupprecht, C.: Continual detection transformer for incremental object detection. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023. pp. 23799-23808 (2023)

Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin Transformer: Hierarchical vision transformer using shifted windows. In: IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021. pp. 9992-10002 (2021)

Lu, J., Batra, D., Parikh, D., Lee, S.: Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In: Wallach, H.M., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E.B., Garnett, R. (eds.) NeurIPS. pp. 13-23 (2019)

Ma, Z., Yang, Y., Wang, G., Xu, X., Shen, H.T., Zhang, M.: Rethinking open-world object detection in autonomous driving scenarios. In: Magalhães, J., Bimbo, A.D., Satoh, S., Sebe, N., Alameda-Pineda, X., Jin, Q., Oria, V ., Toni, L. (eds.) MM '22: The 30th ACM International Conference on Multimedia, Lisboa, Portugal, October 10 - 14, 2022. pp. 1279-1288 (2022)

Maaz, M., Rasheed, H.A., Khan, S., Khan, F.S., Anwer, R.M., Yang, M.: Class-agnostic object detection with multi-modal transformer. In: Avidan, S., Brostow, G.J., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part X. Lecture Notes in Computer Science, vol. 13670, pp. 512-531 (2022)

Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A.L., Murphy, K.: Generation and comprehension of unambiguous object descriptions. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016. pp. 11-20 (2016)

McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: The sequential learning problem. In: Psychology of Learning and Motivation, vol. 24, pp. 109-165 (1989)

Pérez-Rúa, J., Zhu, X., Hospedales, T.M., Xiang, T.: Incremental few-shot object detection. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. pp. 13843-13852 (2020)

Rebuffi, S., Kolesnikov, A., Sperl, G., Lampert, C.H.: icarl: Incremental classifier and representation learning. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017. pp. 5533-5542 (2017)

Ren, S., He, K., Girshick, R.B., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39 (6), 1137-1149 (2017)

Shao, S., Li, Z., Zhang, T., Peng, C., Yu, G., Zhang, X., Li, J., Sun, J.: Objects365: A large-scale, high-quality dataset for object detection. In: IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019. pp. 8429-8438 (2019)

Shmelkov, K., Schmid, C., Alahari, K.: Incremental learning of object detectors without catastrophic forgetting. In: IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017. pp. 3420-3429 (2017)

Wang, H., Ren, P., Jie, Z., Dong, X., Feng, C., Qian, Y., Ma, L., Jiang, D., Wang, Y., Lan, X., Liang, X.: OV-DINO: unified open-vocabulary detection with language-aware selective fusion abs/2407.07844 (2024)

Wang, X., Huang, T.E., Gonzalez, J., Darrell, T., Yu, F.: Frustratingly simple few-shot object detection. In: Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event. Proceedings of Machine Learning Research, vol. 119, pp. 9919-9928 (2020)

Wang, Y., Huang, Z., Hong, X.: S-prompts learning with pre-trained transformers: An occam's razor for domain incremental learning. arXiv abs/2207.12819 (2022)

Wang, Z., Zhang, Z., Lee, C., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J.G., Pfister, T.: Learning to prompt for continual learning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. pp. 139-149 (2022)

Yang, S., Li, G., Yu, Y .: Relationship-embedded representation learning for grounding referring expressions. IEEE Trans. Pattern Anal. Mach. Intell. 43 (8), 2765-2779 (2021)

Yang, Z., Li, R., Ling, E., Zhang, C., Wang, Y ., Huang, D., Ma, K.T., Hur, M., Lin, G.: Label-guided knowledge distillation for continual semantic segmentation on 2d images and 3d point clouds. In: IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023. pp. 1855518566 (2023)

Yin, L., Perez-Rua, J.M., Liang, K.J.: Sylph: A hypernetwork framework for incremental few-shot object detection. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. pp. 9025-9035 (2022)

Yu, Z., Yu, J., Cui, Y ., Tao, D., Tian, Q.: Deep modular co-attention networks for visual question answering. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. pp. 6281-6290 (2019)

Zhang, C., Xiao, J., Liu, X., Chen, Y., Cheng, M.: Representation compensation networks for continual semantic segmentation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. pp. 7043-7054 (2022)

Zhang, H., Li, F., Liu, S., Zhang, L., Su, H., Zhu, J., Ni, L.M., Shum, H.: DINO: DETR with improved denoising anchor boxes for end-to-end object detection. In: The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 (2023)

Zhang, H., Zhang, P., Hu, X., Chen, Y., Li, L.H., Dai, X., Wang, L., Yuan, L., Hwang, J., Gao, J.: Glipv2: Unifying localization and vision-language understanding. In: NeurIPS. pp. 36067-36080 (2022)

Appendix

Effect of Hyperparameter

Impact of λ . The hyperparameter λ modulates the influence of ZiL within the overall loss function. In this study, we investigate the impact of varying values of λ on model performance, as presented in Tab. 6. The results show that a small λ value (e.g., 0 . 01 ) tends to cause the model to overfit new tasks, resulting in reduced performance in the general domain. Conversely, a large λ value (e.g., 1 . 0 ) overly constrains the model, hindering its adaptation to downstream tasks. The optimal λ value falls between these extremes, with the highest hAP observed when λ = 0 . 05 , which strikes a balance between 'ZCOCO' and 'Avg'.

Differentiated Learning Rate. Weexplore the effects of hyperparameter η . The results are detailed in Tab. 7. On the one hand, setting the LLRB's learning rate as zero equivalently freezes it. Although this results in the highest 'Avg' by preserving downstream task knowledge, the 'ZCOCO" performance suffers considerably. This degradation is because HLRB, which has a limited parameter size, endures all the optimization pressure from L zil and L cls + L loc . Despite HLRB's attempts to find a harmonious optimization path, it tends to emphasize downstream task learning over maintaining initial zero-shot generalizability, especially when the predominant loss L cls + L loc outweighs L zil . On the other hand, equating the LLRB's learning rate to the HLRB's is equivalent to creating two HLRBs inside the RDB. The enhanced plasticity that comes from the two-HLRB structure makes it challenging for ZiL to prevent it from gravitating toward directions detrimental to zero-shot generalizability, leading to diminished 'ZCOCO" scores. Moreover, a significantly high LLRB learning rate can also precipitate a swift decline in downstream task knowledge retention. Optimal η lies between these two cases, like 0.1 or 0.2, striking a balance between efficient new task adaptation and zero-shot generalizability protection.

Impact of Different Initial Values of the Scaling Factor. The scaling factor s is a learnable parameter that affects the IVLOD's performance. Its initial value could shape the model's efficiency and adaptation capabilities. As documented in Tab. 8, contrary to initial expectations, a balanced scaling factor for both language and vision components, set at 1.00, does not lead to the best outcomes. Instead, an asymmetrical approach where the vision component's scaling factor is reduced to 0.10 while maintaining the language component's scaling factor at 1.00, or vice versa, appears to enhance overall performance metrics, including 'ZCOCO", 'Avg", and 'hAP". Remarkably, the configuration where both language and vision scaling factors are set to 0.10 achieves superior results, yielding a slight improvement in 'ZCOCO" and the highest 'hAP" score among the tested scenarios. This suggests that a more conservative initial scaling factor may encourage the model to rely more on the base knowledge from pre-training, thus improving its generalization before adapting further through incremental learning.

Visualization

To more vividly showcase the effectiveness of the proposed ZiRa method, we also performed a visualization experiment. As shown in Fig. 6, using the pre-trained Grounding DINO directly to leverage its zero-shot detection capabilities allows for detecting many unseen objects, but fails to detect some. Upon incrementally fine-tuning with iDETR using training samples (with five classes per phase and one sample per class), the model will be able to recognize objects that are undetected by zero-shot detection. However, some objects that can be initially recognized by zero-shot detection are no longer detected. In contrast, after incremental fine-tuning with ZiRa, the images reveal that both familiar and novel objects are detected. This demonstrates ZiRa's dual ability to prevent forgetting both pre-trained knowledge and information from downstream tasks.

Impact of the Norm Types of ZiL

By default, we utilize the L1 norm to regulate ZiL. However, the choice of norm type for ZiL significantly impacts performance. As demonstrated in Tab. 9, L2 excels in ZCOCO, L1 strikes a balance between zero-shot and downstream performance, and Smooth L1 performs exceptionally well in downstream tasks while maintaining competitive zero-shot performance, even outperforming the L1 norm. Regardless of the norm type used, ZiRa remains effective, showcasing its versatility across different norm types.

SeenCategories

banana

apple

pear

yellow_box

green_box

cup

a pipe

black_sponge

gear

angle_aluminum

Zero-shot

Incremental

ZIRA

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of the work are discussed in the conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper fully discloses all the information needed to reproduce the main experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is released at https://github.com/JarintotionDin/ ZiRaGroundingDINO .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper specifies all the training and test details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The paper does not report error bars or statistical significance.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: Each experiment will take 3 hours on 2 Nvidia RTX 3090 GPUs.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have reviewed the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Incremental Vision-Language Object Detection (IVLOD), a novel learning task designed to incrementally adapt pre-trained Vision-Language Object Detection Models (VLODMs) to various specialized domains, while simultaneously preserving their zero-shot generalization capabilities for the generalized domain. To this end, Zero-interference Reparameterizable Adaptation (ZiRa) is proposed to tackle IVLOD without incurring a significant increase in memory usage. Experiments on COCO and ODinW-13 datasets demonstrate that ZiRa effectively safeguards the zero-shot generalization ability of VLODMs while continuously adapting to new tasks.\n\nSTRENGTHS:\n1. The paper is well-written and easy to follow.\n2. The authors propose RDB and ZiL to generalize the VLODMs to specific domains without losing their general zero-shot capability.\n3. A novel learning task is designed to incrementally adapt pre-trained Vision-Language Object Detection Models (VLODMs) to various specialized domains.\n\nWEAKNESSES:\n1. The key contribution of this paper is RDB and ZiL. However, RDB has been proposed by [1]. Therefore, I would owe the novelty to the differentiated learning rate and ZiL (basically L1 norm). From this point of view, the novelty is somewhat incremental. Importantly, the authors should cite [1] and highlight the distinction.\n\n2. The paper only evaluates the general zero-shot performance after incremental learning. Is zero-shot specific-domain inference possible after learning on few examples in this specific domain?\n\n3. It's suggested to include more comprehensive incremental object detection literature, including but not limited to [2-3].\n\n\n[1] Zhang, Chang-Bin, et al. \"Representation compensation networks for continual semantic segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n[2] Qiao, Limeng, et al. \"Defrcn: Decoupled faster r-cnn for few-shot object detection.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n[3] Yang, Ze, et al. \"Efficient few-shot object detection via knowledge inheritance.\" IEEE Transactions on Image Processing 32 (2022): 321-334.\n\nMinors:\nL178 between stability and plasticity\nL342 typo VOLDM\n\nQUESTIONS:\n1. Is TFA also pre-trained on Objects365 [30], GoldG [17], and Cap4M? I have some doubts about the TFA ZCOCO performance in Table 1 if it's pretrained on COCO. To my best knowledge, TFA only finetunes the last classifier layer for the incremental steps and has demonstrated strong performance on the base classes.\n\n2. How is general IOD (e.g. TFA) tested on Zero-shot COCO? General IOD generally does not support zero-shot inference. Is there any adaptation to achieve it?\n\n3. What learning rate is applied to w/ Rep+ (the first line of Table 2)? Is it the high or low one?\n\nLIMITATIONS:\nNo negative societal impact is expected.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn order to extend the application of VLODM in a broader domain, the authors propose to combine incremental learning and zero-shot generalization to solve the problem. However, the widespread catastrophic forgetting and maintaining zero-shot generalization capability in incremental learning are two important issues to be considered. To address the challenges in incremental learning, the authors propose to construct a Zero-interference Reparameterizable Adaptation using Reparameterizable Dual Branch.\n\nSTRENGTHS:\n1. The authors have innovatively proposed a parameterizable double-branching structure.\n2. The writing logic is clear and the article is well structured.\n\nWEAKNESSES:\n1. The authors do not highlight the need for this work and its similarities and differences with other work in the introduction and related work.\n2. Although the authors have designed an innovative Reparameterizable Dual Branch structure, it is not very interpretable for the problem to be solved.\n3. Although the authors give very rich experimental results in Table 1, the description of the metrics. is not clear enough. What is \" the zero-shot performance on COCO as “ZCOCO\" \"? Is it mAP on the base class or the novel class, or all classes? Is it mAP or AP50, or AP75?\n\nQUESTIONS:\n1. The authors introduce the names VLOD and VLODMs in the introduction and cite the literature [11, 17, 20, 24, 39] as the source for the derivation of these names. In these works, tasks such as open vocabulary/open set object detection, grounding, etc. using Vision-Language Model are included. Is the VLOD proposed by the authors consistent with or similar to the task setting of open vocabulary/open set object detection? If it is consistent, why use the term VLOD? If it is similar, the difference between them should be highlighted in the introduction.\n2. In GLIP, the authors give the statement, i.e., \" without seeing any images in COCO during pre-training\". However, in this work, the authors do not discuss or clarify this. Considering the task setting of zero-shot generalization, how do the authors ensure that the images/objects in the test set has not appeared in the pre-training phase?\n3. Based on question 1, where the author claims \"Our work distinguishes itself by performing incremental learning on VLODMs, which are more favorable for open-world problems. \" in related work, is the author's work incremental or zero-shot learning, or a combination of both? What are the similarities and differences between that work and the open-world object detection?\n4. How should the scaling factor “s” be determined in Equation 1? The Reparameterizable Dual Branch structure process contains both HLRB and LLRB, does the size of the scaling factor imply which weight is dominant? If HLRB is dominant, how to ensure that no knowledge forgetting occurs? If LLRB is dominant, how can generalization ability be ensured? Table 7 does not allow a reasonable interpretation of this question.\n5. The authors give Figure 4 and analyze it to prove the validity of ZiL, but how to define \" small-norm \" and why do the authors claim \" the model’s performance is not significantly affected by the addition of small-norm random noise to the input \" when “The average AP on COCO” drops more than 30%?\n6. The authors use the model structure of the Grounding DINO work, in which text representation is an important part of the work. The text template used by the authors is quite different from the one used in Grounding DINO and the authors have not analyzed or explained it, why?\n\nLIMITATIONS:\nSee Waeknesses and Questions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the new problem of incremental visual-language object detection (IVLOD), which aims to preserve zero-shot generalization performance of VLMs, while also adapting to new concepts over time. Authors address IVLOD by proposing the zero-interference reparameterizable adaptation (ZiRa), a light weight branch added to the text and image encoders of VLMs to address this task. Authors demonstrate that their proposed approach outperforms prior work by a considerable margin.\n\nSTRENGTHS:\nProblem Motivation. The proposed problem is of significant interest for practical applications, particularly because many vision-language models (e.g. CLIP) are pre-trained on private datasets, making incremental learning with VLMs challenging. \n\nInteresting Insight. Authors highlight that VLMs are robust to noise (cf. Fig 4 and 5), and propose an incremental learning strategy that updates VLMs weights by ensuring that the updated parameters have low L1 norm. It would be interesting to also plot the zero-shot accuracy on COCO for both modes trained with and without ZiL in Fig 5 as the number of learned downstream tasks increases. \n\nSimple and Scalable Approach. The proposed approach adds a small number of tunable parameters to GroundingDINO. Notably, the proposed approach maintains a relatively constant runtime and number of parameters even when adapting to an increasing number of tasks.\n\nWEAKNESSES:\nDifferences in Baseline Pre-Training. Although authors benchmark their approach against relevant prior work, many of these methods are not trained on the same scale of pre-training data as GroundingDINO. Given that prior works typically operate in a closed-world, and are pre-trained on much smaller datasets, it is unclear how to make an \"apples-to-apples\" comparison between methods. One strategy might be to reimplement prior works (e.g. TFA and iDETR) using GroundingDINO.\n\nDemonstrate ZiRa with Other Architectures. Although authors claim that their proposed approach can work with any DETR based architecture, they only showcase its performance with GroundingDINO. Given that this work establishes a new problem, it would be useful to extensively evaluate different architectures to provide a \"lay of the land\".\n\nQUESTIONS:\nImpact of Pre-Training Data on IVLOD Performance. The performance of VLMs is significantly impacted by the data used for pre-training. How might the specific pre-training data used impact the effectiveness of incremental learning? \n\nAdapting to a Large Number of Tasks. As shown in Figure 5, the L1 norm steadily increases as the number of downstream tasks increases. Does this suggests that the proposed approach does not scale well to a large number of tasks?\n\nAdapting to More Than One Class at a Time. Although each class is considered a new \"task\", one may want to adapt to multiple classes at once. How would the proposed approach perform when learning more than one class at a time? \n\nRuntime and Memory Usage. A key benefit of the proposed approach is that adapting to a new task only marginally increases runtime and memory usage. It would be useful to quantify this such that future works can benchmark on this axis as well. \n\nWriting Quality. Although the approach is clear, the grammar and writing quality hinders comprehension. For example \"inputted\" is not a word, and \"input end\" is a confusing phrase. I would encourage authors to further polish the manuscript. In addition, authors should include more descriptive captions in their tables. Notably, I don't believe the full class names in Table 1, 2 (e.g. Ae, Aq, Co) are provided.\n\nLIMITATIONS:\nYes, authors highlight that the current implementation is based on the DETR architecture, which may not work for all VLODMs. In addition, authors state that ZiRa is designed for IVLOD, and should be evaluated in the context of other incremental learning tasks.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper deals with the object detection problem in a zero-shot and incremental learning setting. Specifically, Incremental Vision-Language Object Detection (IVLOD) task is proposed to incrementally adapt the pre-trained Vision-Language Object Detection Models (VLODMs) to various specialized domain.   \nThe technical design is Zero-interferences Reparametrizable Adaptation (ZiRA), which includes low learning rate and high learning rate adaptation branches for both vision and language encoder. In addition, an L1-norm is applied on top of the adaptation branch outputs. \n\nExperiments are conducted on the Zero-shot COCO (ZCOCO) and ODinW-13 benchmarks. A series of baselines such as TFA, OW-DETR, CL-DETR, Adapting-tuning (AT), and iDETR were compared. Ablation study on modules, vision or language finetuning were also conducted.\n\nSTRENGTHS:\n### 1. The overall framework for incremental object detection is reasonable and complete. \n- 1.1 This paper designs a Reparameterizable Dual Branch module for feature adaptation, and specifically devises low learning rate and high learning rate branches to address the learning-forgetting balance. An additional mechanism to gradually merge high learning rate branch into low learning rate branch is also proposed. The idea makes sense and is verified to be effective in both zero-shot COCO and average performance.   \n- 1.2 The L1-norm is reasonable in zero-shot generalizability and seems to be effective according to Fig. 4-5. \n\n### 2. Experimental results verify the effectiveness of the model design\n- On both the zero-shot and the Avg setting, the proposed method outperforms baseline methods. \n- Ablation studies were conducted w.r.t module designs, vision/language adaptation, branches, etc.\n\nWEAKNESSES:\n### 1. The new task of IVLOD is not well desribed and compared with existing one \n- Incremental Vision-Language Object Detection (IVLOD) is claimed to be a new task. However, it is unclear what are the differences between this new task and existing works such as CL-DETR, Ground DINO, iDETR etc. \n- How is the language specially used in IVLOD compared with existing method? \n- Claimed as the first contribution, but no separate detailed description/subsection to compare IVLOD with existing task setting, either verbally or visually. Fig. 1 only compares with the final performance, which is less intuitive. \n### 2. The main contribution is the different learning rates \n- For the Reparameterizable Dual Branch (RDB), although the name is special, the module design is no much different with existing approaches such as LoRA, Adapter, [6], etc. \n- The structure design is not quite novel, but leveraging different learning rates can be a novel point. \n- It is encoraged to have a better survey of relevant adaptation-based methods for incremental learning. \n### 3. Experimental section have flaws\n- ODinW was adopted, but only 13 datasets were selected. How are those 13 datasets selected? Other methods used much more datasets for evaluation, e.g. 35. \n- In Table 1, AT was not used for Full evaluation. CL-DETR/OW-DETR were not used for lower shots evaluation. What is the reason? \n- The prompt in Fig. 6 looks strange. How is the prompt leveraged by the proposed model? \n- In Table 3, why the model performance on zero-shot COCO underperforms Vanilla one by 1.31%? \n- There are many hyper-parameters such as $\\lambda$ for loss, $\\eta$ for learning rate, scaling $s$. How are those parameters selected and tuned? It seems in Table 5 and 6 that the hyper-parameters matters. \n- In Table 8, how about no norm is adopted? \n### 4. Other minors\n- 1st not 1-th in Fig. 1\n- Incorrect usage of double quotes in this paper, e.g.  “Locality Injection\" \n- Line 209 lacks a period\n\nQUESTIONS:\nPlease refer to the above Weaknesses for questions.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}