DitHub: A Modular Framework for Incremental Open-Vocabulary Object Detection
Authors: Chiara Cappellino
Gianluca Mancusi
Matteo Mosconi
Angelo Porrello
Simone Calderara
Rita Cucchiara
Venue: NeurIPS 2025 poster ()
Keywords: Open-Vocabulary Object Detection
Modular Deep Learning
Incremental Learning
Primary area: deep_learning

Abstract: Open-Vocabulary object detectors can generalize to an unrestricted set of categories through simple textual prompting. However, adapting these models to rare classes or reinforcing their abilities on multiple specialized domains remains essential. While recent methods rely on monolithic adaptation strategies with a single set of weights, we embrace modular deep learning. We introduce DitHub, a framework designed to build and maintain a library of efficient adaptation modules. Inspired by Version Control Systems, DitHub manages expert modules as branches that can be fetched and merged as needed. This modular approach allows us to conduct an in-depth exploration of the compositional properties of adaptation modules, marking the first such study in Object Detection. Our method achieves state-of-the-art performance on the ODinW-13 benchmark and ODinW-O, a newly introduced benchmark designed to assess class reappearance.

DitHub: A Modular Framework for Incremental Open-Vocabulayy Object Detection
Chiara Cappellino
Gianluca Mancusi
Matteo Mosconi
Angelo Porrello
Simone Calderara
Rita Cucchiara
AlmageLab University of Modena and Reggio Emilia name. surname wwtuumiiiiiiii.
Abstract
Veen-Vocbbulary object detectors can generalize to an unrestricted set of categories through simple textual prompting. However, adapting these models to classes or reinforcing their abilities on multiple specialized domains remains essential. While recent methods rely on monolithic adaptation strategies with single of weights, we embrace modular deep learning. We introduce DitHub, a framework designed build and maintain library of efficient adaptation modules. Inspired by Version Control Systems. DitHub manages expert modules as branches that can be fetched and merged as needed. This modular approach allows to conduct an in-depth exploration of the compositional properties of adaptation modules, marking the first such study in Object Detection. Our method achieves state-of-the art performance on the ODinW-14 13 benchmark and ODinW-O, a newly introduced benchmark designed assess class reappearance.
Introduction
Object detection is cornerstone of computer vision, with recent advancements 448 2 57 32 24
enabling architectures that integrate Vision and Language large-scale pre-training (25). This has given rise to Open- Vocabulary Object Detection, where models can classify and localize objects beyond predefined set categories. The ability to process arbitrary textual queries and detect corresponding objects has profound implications for real-world applications, including medical imaging [18), safety and security [4I), and industrial quality control [38]. For instance, the ability to detect rare or atypical objects from safety camera is crucial for issuing early warnings about uncommon dangers. However, this requires Open-Vocabulle Vocabulary detectors to adapt effectively, enabling them to recognize categories beyond their pre-training knowledge distribution.
Recent approaches, such as those detailed in 44, have extended Vision-Language detectors to incrementally accommodate new categories while preserving robust capabilities [40]. However, these methods employ monolithic adaptation, where all newly acquired knowledge condensed into single set of weights. This approach presents challenges in real-world scenarios that demand updates to specific concepts which may reappear under varied input modalities. For instance, in safety and security applications, the class "person" may need to be detected not only in standard RGB imagery but also in thermal imagery, necessitating continuous model adaptation to ensure consistent performance across different data types. Additionally, for rare complex categories, incremental adaptation is crucial to refine the base model for fine-grained concepts. such scenarios, monolithic deep architecture faces challenges similar managing complex program written
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Tra Training
Inference
Tasks
Modules
OV Detector
Turtle Axoloti
DitHub
Class- spaciflecci
Figure 1: DitHub: a novel system to share and update efficient adaptation modules for Open-
Vocabulary Object Detection
single page of code. Namely, when treating multiple classes within unique set of weights, becomes hard to update specific concepts selectively, without compromising information related to other concepts. Considering less-represented categories, the corresponding concepts can become blurred within the unique set of weights, leading to potential degradation in performance over time.
We advocate for modular approaches, outlined in Modular Deep Learning (36), which reimagine neural networks as adaptable and composable entities rather than rigid, monolithic structures, By maintaining a library of expert modules, neural architectures activating only the relevant module when needed. Moreover, this modular design enables selective updates and seamless composability, allowing class-speciff modules trained in different domains to be efficiently merged. Finally, modularity offers a practical solution to mitigate biases if module exhibits undesirable biases toward a particular category, it can be selectively removed adjusted without disrupting the entire model, ensuring more precise and fair adaptation.
achieve class-specific specialization,
With modularity at its core, introduce DitHub (where D stands for Detection), framework inspired by Version Control Systems that efficiently manages growing library of detection modules. This library expands incrementally as new groups of categories, referred to tasks 47), are intro- duced. Upon encountering a new task, DitHub initializes an efficient module (e.g.. LoRA [13].. which first undergoes broad pre-tuning phase across all classes of the task, referred to the warmup phase. This stage establishes a solid and agnostic foundation, enabling the subsequent special- ization of the module for specific classes. Following the warmup phase, the module branches into specialized experts, each optimized independently. To enable selective updates for classes previously introduced to the model, DitHub first performs fetch operation to access potentially existing class-specific modules. If these exist, instead of starting the training from the warmup, DitHub uses merged initialization combining the retrieved pre-existing class-specific modules with the current warmup module. This merged initialization facilitates smooth domain knowledge transfer and ensures continuity of class-specific learning across tasks.
Moreover, to ensure scalability with respect to the number of classes, DitHub shares a subset of parameters across different adaptation modules, enhancing memory efficiency without compromising individual class adaptability. At inference, enables the selective activation of specialized modules for complex classes, ensuring robust adaptation while maintaining the backbone's 's zero-shot and Open- Vocabulary capabilities. A high-level overview of our approach depicted in Figure
Following [4], evaluate DitHub the ODinW-13 220] benchmark, where it outperforms the state-of-the-art by substantial +4.21 mAP in the Incremental Lisguu-Language Object Detection setting. We also introduce ODinW-O (Overlapped), curated subset of ODinW-35 [19] focusing on classes re-appearing across multiple tasks. Furthermore, we thoroughly investigate and ablate the compositional capabilities of modular approach, marking the first such exploration in Object Detection. In summary, our key contributions are:
We investigate the importance of decoupling a warmup phase from a specialization one in the context of efficient adaptation modules.
We introduce DitHub, framework inspired by Version Control Systems, designed to manage growing library of efficient modules for vecabVocayulary detection.
'The creature in the red box is an axolotl, symbolizing rarity as deviation from pre-training. We use stylized axolotl as the icon for DitHub.

---

We achieve sate-of-the-ar performance on ODinW-13 and the newly introduced ODinW-O.
We conduct an in-depth analysis of the compositional and specialization capabilities of class-specific modules, first in the context of Object Detection.
2
Related Works
Incremental Vision- Object Detection. Vision-Language Object Detection (VLOD), also referred to as Open- Vocabulary detection 44. is the task of detecting and recognizing objects beyond fixed set of predefined categories. Early approaches leverage powerful visual and textual encoders (e.g., CLIP [40] to enable object detectors to recognize virtually any object specified by a textual query 10 15 A fundamental trend in this field has been the reliance large-scale pre-training extensive datasets 44), which has become standard practice for tasks such as classification. In the context of Vision-Language Object Detection, state-of-the-an methods have followed this paradigm, with GLIP 20 being one of the first notable examples. GLIP reframes Object Detection as a grouudin problem, leading pre-trained model capable of generalizing unseen objects by integrating textual and visual semantics from large-scale datasets. Building on GLIP's problem formulation, Grounding DINO [25] extends this paradigm by incorporating merged visual and textual semantics at multiple network stages.
Recently, Incremental Vision-Language Object Detection (IVLOD) has emerged natural extension of the VLOD paradigm. By incorporating Incremental Learning 17 [1], IVLOD addresses the challenge of fine-tuning pre- treitra Vision Languuge model on specific object categories while preserving its zero-shot (35) capabilities. Following [4], we adopt Grounding DINO the backbone and use IVLOD explore the merging of class-specific modules.
Parameter Efficient Fine- Tuning and Model Merging. The rise of Transformer models 48 has made pre-training a standard approach in machine learning. Parameter-Efficieet Tine-Tuning (PEFT) [12. and LoRA 113, adapt pre-trained models to downstream tasks by updating only small fraction of parameters while keeping the rest frozen.
methods, such Prefix Tuning (21), Adapters
In the context of IVLOD. ZiRa introduces a distinct approach that can be viewed as specialized form of parallel adapters (28), incorporating reparameterizable side branches. In contrast, our work adopts LoRA and explores the potential of merging class-specific LoRA modules.
Several studies have investigated LoRA module fusion. An early approach relies on simple arithmetic operations such as addition and subtraction [56]. More advanced techniques focus on learning optimal combination coefficients, determining the contribution of each LoRA module in the merging Recently, researchers have introduced closed-form solutions for LoRA merging 44 and proposed regularization objectives to ensure that the fused modules remain close to the pre-trained weight distribution (37]. In this respect, our approach takes novel direction, being the first to study module merging in the context of Object Detection.
process [4] 53 52
29
54
3 Preliminaries
Problem setting. We consider a sequence of tasks {1....,T), T), where each task corresponds to a dataset Dr. Each dataset consists of samples (ar, y), where x represents the input data, and y contains the associated labels, including ground truth bounding boxes.
Given pre-trained Vision-Language detector f(; +), parameterized by θ, the objective of Incremen- tal Vision- Languuge Object Detection (IVLOD) is to improve performance on newly introduced tasks while preserving original zero-shot capabilities. Formally, for given task we seek an adaptation A that minimizes the following objective:
min ((.,)) {(f(z; θ A), y) + Eeem shot
(1)
where l is the Object Detection loss function. The term θ
+ represents the set of adapted model parameters. Additionally, Ezero-sbot quantifies the performance difference in the zero-shot setting before and after adaptation.
Fach task / introduces set of classes C. Unlike standard Incremental Learning 47 IVLOOD allows class overlap between tasks. In other words, there exist task pairs (1, t', with such that C, n Cr. # 0.
51
50
46
3

---

Task 1
Task t
Abia
Zetch
Dog
Adse
Modules Library
B
Modiules Library
B
Ave
Aspiotti
branch
Asoloci
branch
B
Asssictt
Shared
Augnle
Turtle
Shared
B
B
Warm
Specia lization
Warm
Specialization
Figure 2: Overview of DitHub. Through three primitives i.e., branch, fetch, merge DitHub allows to update a library of class-specific modules for Open- Vocabulary Object Detection.
LoRA. Low-Rank Adaptation 13
is technique to significantly reduce the number of trainable parameters. Given generic layer with weight matrix W Rdxk, LoRA introduces low-rank matrices A E IR* and B E RR - with r < min(d, k) such that the adapted weight W is:
W W AWW where AW BA.
The use of these low-rank matrices significantly reduces memory usage and computational overhead, making LoRA an efficient alternative to fine-tuning the entire model.
4
DitHub
Version Control Systems are ubiquitous tools used to manage and track changes in code repositories. Inspired by this, we propose DitHub framework that operates akin to a Version Control System, enabling the creation and expansion of a library of efficient adaptation modules for Vision-Language object detectors. DitHub develops and maintains class-specific expert modules using a simple yet effective strategy that requires architectural modifications additional loss terms (Section 4.1p.
Following 4.
we consider the pre-trained Grounding DINO the base architecture for Open- Vocabulary detection and use LoRA for fine-tuning. We propose an efficient pipeline that handles LoRA's low-rank matrices with two objectives: matrix encodes class-specific knowledge, while matrix B captures general knowledge and is shared among tasks to maintain overall efficiency (Section 4.2). Finally, when fine-grained adaptation needed (e.g., detecting rare object within scene), DitHub enables the dynamic extraction of the corresponding expert modules, ensuring precise and efficient inference, as outlined Section 4.3 Algorithm
The pseudocode of DitHub is provided in
4.1 Training Class-Specific Modular Detectors
To manage a library of class-specific modules, our approach is to train a tailored A matrix for each class to enable specialization. In contrast, the B matrix shared across classes and continuously fine-tuned as tasks progress. Thus, at task f,, we train total of (C|++1 matrices per adaptable layer one for each class and one shared B matrix.
However, in Object Detection, creating class-specific experts is non-trivial, as images often contain multiple objects from different categories, making it difficult isolate single class. To tackle this, we adopt a stochastic training strategy for specializing the A matrices: for each image, we randomly select one of the present classes and update its corresponding A matrix. However, preliminary experiments show this approach is ineffective for rare categories, as random selection limits their exposure to sufficient training data.
As better highlighted in Algorithm we overcome this issue by introducing an earlier warmup phase at the beginning of each task. This phase is meant to provide a robust, class-agnostic initializa- tion, ensuring that the subsequent specialization stage is more effective.

---

Algorithm 1: DitHub Training
GD Encader
(A, 8)
shared
Input: T tasks each containing [Ce] classes; li-
brary A <- {) of class experts; global B.
Memory MemoryUsage Usage (MB)
150
for task t [I]... do
(Awu, B f warmup (training)
100
for class E C₁ do
Branch
A. E A then
Fetch
A +
use Equation B
Merge
50
else
A Awu (Ae, B + specialization (training)
Number Classes
+ A A.
Figure 3: Memory requirements as the num- ber of classes grows.
B +- use Equation
Warmup. At the onset of the new task, shared warmup matrix Awu is instantiated and optimized (see Figure yellow arrow) to capture class-agnostic yet task-specific knowledge. This phase, which runs for a fixed number of epochs, provides common initialization for the A modules before their specialization. By doing so, the specialized modules start from a common base point, factor that has been shown greatly enhance model compositionality 33 391
Often associated with improved linear mode connectivity between models [9], these properties are leveraged by our approach to combine class-specific modules based on input textual queries, as detailed in Section 4.3 Moreover, considering rare and underrepresented classes with few training examples, the warmup phase seeks establish an already robust and effective initialization.
Specialization. After the warmup phase, we proceed to the core DitHub: the specialization phase. To train class-specific modules, DitHub generates |Ce| distinct branches (branch in Figure 2 each dedicated to single class. These branches undergo independent optimization, represented by a blue arrow in Figure capture class-specific capabilities. In doing so, for each training image, employ the stochastic strategy introduced in Section 4.1 to select single expert for training.
Notably, the modular design of our approach effectively handles potential class overlaps across consccutive tasks. In real-world scenarios, classes may reappear in new domains with altered distributions semantics. For instance, consider the class "dog". originally learned from real- world camera images, later appearing knowledge and enable "dog" expert to recognize the entity across domains, despite semantic variation. In this case, the modularity of DitHub enables the selective adaptation of the "dog" module, while minimizing interference with other classes, such the "axolot!" module.
task involving thermal imagery. The goal is to retain prior
To favor selective updates, DitHub performs a fetch operation (Figure 2), which simply retrieves the class-specific module from previous tasks if the class has already been seen followed by the merge (Figure 2) For instance, considering the class "dog", the previously stored expert module dog is fetched and merged with the current task's 's warmup matrix A following weighted strategy:
A dog
= (1 A + > Awu
Here, AA is set to low value, prioritizing consolidated knowledge over new insights from the warmup phase. Details on the impact Xa are in Section 5.2 Crucially, our merging process creates an on-the-fly fly initialization that blends historical class-specific knowledge with new domain information, resulting A which serves as the starting point for the specialization phase.
4.2 Bridging Modularity with Memory Efficiency
To keep library memory-efficient we maintain single shared B matrix instead of distinct B for each class. As shown Figure using class-specifie (A, B) pairs results in a linear increase in memory consumption as the number of classes grows. Sharing the B matrix reduces this requirement by half Additionally, considering DitHub "B shared" (line -0-), the memory footprint of our
PI Section A.2
present DitHub's results when B is shared.
5

---

approach is consistently lower than the storage needed for the Grounding DINO encoder alone (line --). This makes DitHub's memory demands manageable for practical applications. Moreover. memory efficiency is optimized by fine-tuning a targeted portion of the base architecture (Section [A.]p.
Due to the shared design, the matrix B is more susceptible interference and catastrophic forget- ting [31]. Consequently, we implement mechanism that ensures smooth updates as tasks progress. Specifically, the end of each task t, we update the matrix B by merging the previous checkpoint, BB-1, with the weights learned after the current task, BoP!. This process is formulated as follows:
B₁ = (1 B,) AB
Unlike Equation 3 the merging coefficient Xn is assigned a higher value, giving greater weight the latest state. This approach results in a dual strategy for Xa and Xn, further detailed in Section 5.2
4.3 Inference with DitHub
With our library of efficient, class-specific modules in place for adapting the Vision-Language detector, DitHub enables two complementary inference strategies. For requests targeting individual classes where fine-grained adaptation is needed, the corresponding expert module can be directly selected and used to detect that class. Conversely, when evaluating on a textual prompt that includes multiple classes, we retrieve the corresponding class-speciffe modules from the library and merge them by computing an average of their weights to create a single composite adaptation module. This unified module is then used in a single forward pass to perform detection over all target classes. In our experimental studies, we present results and discussions that address both scenarios.
Experimental Studies
5.1 Incremental Vision-Language Object Detection
Evaluation settings. Following 4 we employ ODinW-13 20 to assess the model ability to learn new tasks while mitigating forgetting. ODinW-13 includes 13 sub-datased spanning from the traditional Pascal VOC [8] to more challenging domains with significant distribution shifts, such as thermal imagery. Also, the IVLOD setting requires the Open- Vocc-locabulary detector to retain strong zero-shot capabilities after fine-tuning, which are evaluated on MS COCÓ (23]
One issue of ODinW-13 13 is that most classes are confined to a single task among the 13. However, DitHub is conceived handle recurring classes across domains, necessitating updates to their respective modules. To better assess incremental detection and module reuse, we introduce ODin W- O (Overlapped), subset of ODinW-35 that is designed to reflect realistic scenarios where classes can reoccur across tasks over time. Further details are provided in Section C
For comparisons, employ mean Average Precision (mAP) and compare DitHub against six state- of-the-art methods. CL-DETR (26) and OW-DETR [IT] are explicitly designed for Incremental Learning, while TFA
[9] specializes in few-shot adaptation. AT [12] has been extended both incremental and few-shot learning, whereas iDETR (6) natively integrates both aspects. Finally, ZiRa 4 our primary competitor, specifically tailored for IVLOD.
Implementation details. We focus on Grounding DINO and adapt only its encoder network, as further validated in Section The merging coefficient X for the A matrix is set to 0.3 for WDin 13 and 0.1 for ODinW-O: for the matrix B, we set X 0.7. The choice of and its underlying rationale are further discussed in Section 5.2 During training, we allocate an equal number of epochs to the warmup and the subsequent specialization phases see Section F for further details.
Performance on W-13. Table I presents the IVLOD results, including the average across the 13 tasks (Avg) and zero-shot preservation (ZCOCO). We evaluate performance in both the full-data setting and few-shot scenarios with 1, 5, and 10 shots per class. All baseline results, except those of Grounding DINO, ZiRa, and our method, are taken directly from 44. Each number represents the average of three runs with different random seeds, with standard deviations provided in Section D.I
In the full-shot setting, DitHub outperforms ZiRa by a substantial +4.21 mAP and achieves the best results in nine out of thirteen tasks. Notably, on ZCOCO, it surpasses ZiRa by +0.75 mAP, setting new state-of-the-ane art in both incremental and zero-shot retention. While not explicitly designed

---

Table 1: Comparison of mAP values across ODinW-13 tasks. Avg represents the average across the 13 tasks, while zero-shot performance is reported in the ZCOCO column. Bold indicates best results.
ZCOCO
Avg
Aq
Eg
Mu
G-Dino
47.41
46.81I
19:11
20.82
64.75
63.38
25.34
56:27
54.80
65.99
22.13
02.01
32.85
70.35
07.07
TEA
18.54
20. 51
16,
16.81
63.30
G1179
25.47
50.57
20.45
50.16
21.30
50.67
13.86
60.85
43.97
IDFTR
22.81
73.24
61.45
13
58.64
46.22
21.65
23.02
06.00
08.96
27.58
54.68
622.47
36.94
53:17
20.37
70.31
ZiRa
06.44
45. 56
20:34
19.64
60.47
60.00
30.00
56. 27
58.12
65.13
25.85
55.69
30.50
72.50
$8.55
Diiffils
46.40
49.
21 33
24.34
69.25
61.09
42. 10
57.
58.85
55.24
24.62
57.58
35.
70.90
02:50
TFA
30.63
1DETR
43.51
53, 65
25,
25.55
70.42
62.95
69.98
50.54
54.85
64.50
35.24
57.64
42.95
76.51
56.92
5
AT
63.67
47. 16
14.65
24.97
66.55
66.19
35.5 35.85
62.03
55.00
65.20
27. 08.88
52.65
32.21
71.23
57.54
ZiRa
63.41
58.77
24.22
71.34
63.96
49.54
80.51
64.39
56,31
54.30
DiHHub
45.56
322.5 86
26.84
28.91
08.45
600.25
51.93
55.10
50.43
50.44
38.18
68.12
38.05
76.04
61.33
TEA
$4.35
46.61
21.17
22.16
66.82
60..33
32.5
50.16
55.64
61.98
27.50
57.40
28.14
066882
52.31
IDETX
63.54
58.29
25.29
27.70
65.02
67.58
47.99
180.20
56.32
63.15
35.18
50.37
74.70
10.12
10
47.34
48
55-37
46.14
53.200
25,
26.96
71,72
60.22
45:22
132.5
61.24
65.47
35.08
56.73
70.97
T6.14
Difffbb
66.54
54.
26. 19
30.28
68.11
65.75
52.82
56.55 82
63.36
61.85
36.68
60.82
41.46
76,00
61.48
TFA
30:97
47. .3.93
23.
30.65
67.21
61.77
30.52
50:23
47.73
60.91
20.25
61.72
31.42
06.23
01.61
37.332
32.64
55.33
71,01
50-39
39.19
17:12
AT
$2:30
51.14
23.62
39.00
72.32
65.51
31:47
50.48
00.51
66.07
20.00
$3.50
34.04
68.07
90.23
Full
OW-DEER
31:222
55.58
28,
13.78
70.54
61.75
03.84
56.75
63.13
00.51
45.16
58.99
36.99
74.42
153.20
CL DETR
32.15
57. 26
29.35
15.15
71,94
69.00
45.21
58.33
63.10
71,68
06.58
60.83
38.14
76.74
85.18
ZIRA
31.76
47.35
71.17
64.74
46:53
71.00
48.48
63.01
41.44
76.13
Diitub
47.01
62.19
34.62
50.65
70.
08.56
49.25
65.57
69.58
71.10
56.65
70.88
52.82
79.30
68.18
ZCOCO (ODinW-13)
ODinW-13
ZCOCO (ODinW-O)
ODinW-O
Norallize melp
1.00
0.98
30000
mAP
1.00 1
0.98
Normelized mvp
0.90
Normalized
0.95
1.00
0.96
0.94
0.94
0.96
0.90
0.92
0.80
0.90
0.90
0.88
0.92
0.85
0.80
0.86
Task ID
Task ID
Task ID
Task
- ZiRa
Dithfub
ZiRa
DitHub
Figure 4: Normalized Average mAP as tasks progress. To assess performance degradation, we normalize the mAP for each task by dividing the current value by the original mAP recorded at the end of that task, i.e., the curve R 0.90, the model retains 90% of its original performance.
for data-scarce conditions, DitHub also excels in the few-shot setting. In particular, in the five-shot and ten-shot settings, DitHub emerges the best-performing approach, with the ten-shot scenario showing a slightly larger improvement over ZiRa (+1.23 mAP). Our method also performs well in the one-shot setting, where iDETR leads, followed closely by DitHub and ZiRa.
These findings - optimal performance in Incremental Learning, effective retention of zero-shot capabilities, and robustness in the low-data regime- clearly highlight the advantages of our modular approach.
Performance on ODinW-O. The ODin W-O scenario emphasizes that frequently encountered classes may reappear in subsequent tasks or at later times, such the class "person" in common industrial and smart-city scenarios. As detailed in Section C.2 ODinW-O guarantees that each class is revisited least once in subsequent tasks. In this benchmark, we primarily compare our approach with ZiRa, as it is the most competitive method in the standard IVLOD setting, excluding ours.
The results in Table inddicate that our approach secures substantial improvement of +4.75 in mAP and +2.08 mAP gain on ZCOCO over ZiRa. We attribute the success of our method to its class-oriented modular design, which enables selective updates to recurring concepts. This de- sign prevents unintentional overwriting knowl- edge across classes, tasks, and domains.
Table 2: mAP values on ODinW-O. Best bold.
Method ZCOCO
Asg
Ae
Vo
G-Dino
17.11
53.15
45.12
67.51
58.11
25.81
70.40
51.87
ZiRa DiHHub
44.43
57.63
39.02
68.00
64.00
46.26
77.28
49.47
62.38
53.35
80.21
66.90
7

---

Table 3: Experts-not- (EnE). Training by selecting a random expert.
Method
ZCOCO
Avg
Ae
Eg
Th
G Dino
47.41
64.75
59.95
56.27
65.94
22.13
32.85
70.38
EnE
46.86
60.96
40.51
69.67
67.54
63.42
53.76
78.36
66.95
DitHub
47.01
62.19
34.62
50.65
70.46
68.56
49.28
65.57
69.58
71.10
56.65
70.88
52.82
79.30
68.18
70
Et-ANIGO
53
60
50
52
52
O-MVIGO
10
Startish
Stingray
Pathole
Raccoon
Pistal
Package
0.1
0.3
0.5
Aa
0.7
0.9
0.1
0.3
0.7
0.9
Figure 5: Performance comparison of Ground- ing DINO, EnE, and DitHub on rare object cate- gories from the 1-13 dataset.
Figure 6: Sensitivity analysis on da and XB- Performance measured as harmonic mean Avg and ZCOCO mAP.
Performance analysis tasks progress. In Figure 4 we present an incremental analysis comparing the performance of DitHub and its strongest competitor, ZiRa, after each task. The evaluation, conducted on both ODinW-13 (left) and ODinW-O (right), allows to assess the retention of both zero-shot and in-domain capabilities throughout the task sequence, providing a direct measure of their resistance forgetting (further diseussed in Section B). On ZCOCO, DitHub exhibits stable performance across tasks, with a negligible loss in zero-shot capabilities unlike ZiRa, which shows more pronounced decline. When evaluating performance across the sequence of downstream tasks, both methods demonstrate a decreasing trend, indicating that forgetting remains a significant challenge in Incremental Object Detection. Nevertheless, as more tasks are introduced, the performance gap widens, with DitHub consistently outperforming its competitor. This highlights the effectiveness of our modular design, which enables scalable knowledge organization and adaptation as the number of tasks grows. Moreover, the gap is still present in the ODin setting introduced in this work, which models the realistic scenario where subsequent tasks may share classes. This setting, being inherently more susceptible to forgetting, further underscores the advantages of our approach.
5.2 Model Analysis and Ablation Studies
We examine the properties of our specialization framework and the benefits of composing class- specific experts. To this aim, we introduce an ablative approach, Experts--oot-Exprrt (EnE), which removes class specialization. EnE is identical DitHub, except during training, where it omits the stochastic strategy from Section 4.1 Instead, given an image from the current task, EnE performs the training step on one module chosen at random among all task classes, As result, modules no longer specialize in class but instead learn from the whole task data, including images without their designated class. Consequently, they become generalist learners rather than class- slasssceccific experts.
In Table 3 we compare DitHub with its non-specialized variant, ie.., EnE. Notice that, at inference, EnE as an ensemble of class-agnostic modules. In contrast, DitHub employs class-specialized counterparts with an expertise on data containing the same class. This targeted specialization yields +1.23 mAP improvement average, with additional gains also the ZCOCO metric. These results affirm that system of modular experts, each with distinct knowledge, offers a promising approach for systems requiring continuous and selective adaptation.
Evaluation on rare classes. As discussed in Section 4.3 DitHub offers tne-ggained control at inference, enabling users to tailor the system their use case - innluding the option to use single module specialized for a target class. To assess this, we conduct targeted evaluation from ODin 13, activating only the corresponding target class module. Results in S classes, highlighting the benefits of class specialization.
rare classes
that DitHub achieves the best results

---

47.41
ZZ
7COCO
GD
45.80
Avg
Metbod
Memory (MB) (mAP)
Base
56.00
DitHub
57.01
ZiRa
57.98
+WU
58.30
DitHub
60.26
47.22
DitHub
60.53
-Eq. 4 + wu
58:79
DidHuh
61.93
DitHub 16
147
62.19
DitHuub
62.19
G. Dino (encoder only)
~- 659
46.80
60
65
AP
Figure 7: (left) Ablation study on DitHub components. showing average performance on ODin W3 alongside the ZCOCO metric. (right) Memory usage (MB) versus performance (mAP) for DitHub at varying ranks, compared against ZiRa, with performance averaged over the ODinW-13 tasks.
Sensitivity Analysis of A. Varying the merging coefficients Xa and X offers valuable insights into method. To this end, define aggregate metric the harmonic mean of DitHub's ' full-shot performance, balancing specialization (Avg) and zero-shot generalization (ZCOCO). Figure shows performance trends for ODinW-13 (top) and ODinW-O (bottom) as AA and Ag in 0.2 increments.
Considering AA its role significant while merging the expert matrices (see Equation 3 From Figure 6 lower values of AA lead to better results, indicating that prioritizing past expert knowledge over the warmup module is more effective. Interestingly, while A = 0.3 performs best for ODin W- 13. ODinW-O exhibits monotonic decline as A increases. The contrast reflects dataset structure: W-O includes overlapping classes, emphasizing knowledge retention, whereas ODinW-13 has varied class distributions, demanding stronger integration of domain-specific warmup knowledge.
For the incremental adaptation of the shared B matrix, a different trend emerges. The harmonic mean increases with Åg, peaking at 0.7 for both datasets. This suggests that prioritizing the knowledge in the Bop matrix from the current task, as shown in Equation 4 is crucial. Our intuition, supported by [30, is that the latest gop matrix integrates knowledge from both earlier and recent tasks, making more valuable than B which is trained solely on earlier tasks.
On Warmup and Shared B. To evaluate the impact of DitHub's components, we progressively integrate them and present results in Figure (left). We establish the zero-shot performance of Grounding DINO (GD) as a reference. Our baseline fine-tuning procedure (Base) trains class- speciiic matrices and the shared matrix independently for each new task, omitting both parameter merging and the warmup phase. While this baseline suffers from complete catastrophic forgetting, it still demonstrates improved performance compared to the zero-shot Grounding DINO.
Building on this baseline, we first add the warmup phase (+wu), which yields a notable improvement by stabilizing training and facilitating the learning of rare classes through robust initialization. Subsequently, incorporating Equation 4 into the varmup-augmente improvement- this expected, as the shared B matrix constitutes only a small fraction of the total parameters. Crucially, the most substantial performance gain is achieved upon integrating Equation (DitHub in Figure [ (left)), highlighting the importance of retaining domain- domain-specific knowledge within class-specific modules to support stable and effective Incremental Learning. Collectively, the synergistic combination of these three key components the warmup phase for robust initialization, the merging strategy for A matrices to retain class-specific knowledge across domains, and the shared B matrix for efficiency yields a substantial +6.13 mAP improvement compared to the baseline model.
strategy results in a modest
Finally, we note that results show a progressive decrease in the ZCOCO metric as the average mAP rises- an expected trade-off specialization degrades zero-shot capabilities. Notably, the Base fine-tuning baseline scores higher on ZCOCO than the original Grounding DINO zero-shot model. We attribute this counter-intuitive result to catastrophic forgetting. Our full method, DitHub, is trained to be robust across the varied domains of ODinW. contrast, the Base procedure completely forgets such broad knowledge and overfits to the new tasks. This process of forgetting diverse, out-of- distribution data results in a model that is coincidentally more aligned with the data distribution of the COCO benchmark, leading to an artificially inflated ZCOCO score.

---

DitHub class-uniearning thorse
M DitHub class-unlearning
W
a
Figure 8: Qualitative results of DitHub's 's class unlearning. Each column compares zero-shot Ground- ing DINO predictions (left) with the output after subtracting specific class module (right).
Analysis of the Performance-Memorr TradeofT. To study the trade-off between performance and memory, we examined how the average mAP on ODinW-13 is affected by the LoR rank r, Le., the dimensionality of the low-rank update matrices. The results, presented in Figure 7 (right), demonstrate a graceful degradation in performance even as the rank is substantially reduced. Notably, rank of r 2 achieves performance within 2 mAP points of our best model (r 16) while reducing the per-class parameters and memory usage by a factor of eight. We also evaluated the extreme case of 1, which collapses the adaptation matrix into vector, and found that the performance remains remarkably strong. We attribute this robustness to our framework's 's granularity; since each module only encodes the knowledge of single class, it can operate effectively with fewer parameters than approaches that aggregate information from multiple classes into one module.
In terms of memory efficiency, our method with 2 requires approximately the same memory as ZiRa while performing significantly better (+2.28 mAP). Furthermore, our r = variant delivers highly competitive results, scoring less than 1 mAP point below ZiRa while using only half the memory. A detailed breakdown of the per-task mAP for these varying ranks is provided Section A.3
Specialized modules allow training-free unlearning. The advantages of DitHub go beyond additive Incremental Learning. Expert modules can also be used remove specific knowledge by simply subtracting the corresponding weights from the base model. This operation, akin to zero-training unlearning. has been studied in image and text categorization [5 to the best of our knowledge, we are the first to explore its application within the context of Object Detection. Such mechanism allows the model to unlearn the detection of sensitive object categories such as faces, license plates, medical equipment when required to comply with privacy regulations or data retention policies.
Specifically,, we test whether class knowledge can be unlearned from the zero-shot Grounding DINO baseline by subtracting the corresponding class c module from the pre-trained model: W' ABe. where we set = 0.3. Figure Seguur@@qualttenn @qualitatively demonstrates DitHub's class- unlearnnln capability. Each column illustrates the effect of removing specific class module. For instance, when the horse module is subtracted, the previously detected horses are no longer recognized, resulting in false negatives that indicate successful forgetting. This highlights DitHub's s ability to selectively unlearn target classes while preserving knowledge of unrelated ones. quantitative analysis is provided in Section D.2
6 Conclusions
In this research, we introduced DitHub, a methodology akin to a version control system that maintains expanding library of modules to adapt Openn-ocabulary object detectors. Our approach achieves state-of-the-art performance in the Incremental Vision Lasguu-Language Object Detection scenario. The incremental nature of this domain allowed us explore the compositional capabilities of efficient modules, marking the first such investigation in Object Detection.
In future works, we aim to further leverage modular library management to deepen our understanding of knowledge transfer between efficient modules for Object Detection. For instance, if our library contains "axolotl" module trained real-world images and possess domain knowledge for the cartoon image domain, can we synthesize a module specialized in detecting cartoon "axolotls" through knowledge transfer? DitHub will provide fork of this manuscript explore such questions.
10

---

Acknowledgments and Disclosure of Funding
Chiara Cappellino's position was funded through the project supported by the European Defence Fund (EDF) under grant agreement EDF-2022-101 121405. Angelo Porrello was financially supported by the Italian Ministry for University and Research through the ECOSISTER ECS 00000033 CUP E93C22001 100001 project and the European Commission under the Next Generation EU programme PNRR. Rita Cucchiara was financially supported by the EU Horizon project "ELIAS European Lighthouse of Al for Sustainability" (No. 101120237).
References
[1] Matteo Boschini, Lorenzo Bonicelli, Pietro Buzzega, Angelo Porrello, and Simone Calderara. Class- incremental continual learning into the extended der-verse. IEEE Transactions on Pottern Analysis and Machine Intelligence, 2022.
[2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-ena object detection with transformers. Proceedings of the European Conference on Computer Vision, 2020.
[3] Arslan Chaudhry, Puncet Dokania. Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vision, 2018.
[4] Jieren Deng, Haojian Zhang, Kun Ding, Jianhua Hu, Xingxuan Zhang, and Yunkuan Wang. 7ero-shot generalizable incremental learning for vision- lasion-language object detection. In Advances in Neural Information Processing Systems, 2024.
[5] Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- Fre-training of deep bidirec- tional transformers for language understanding, In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, 2019.
[6] Na Dong, Yongqiang Zhang, Mingli Ding, and Gim Hee Lec. Incremental-dedt Incremental few-shot object detection via self- learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
[7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[8] Mark Everingham. Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 2010.
[9] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, 2020.
[10] Xiuye Gu, Tsung-Y Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In International Conference on Learning Representations, 2022.
[11] Akshita Gupta, Sanath Narayan, KJ Joseph, Salman Khan, Fahad Shahbaz. Khan, and Mubarak Shah. Ow-detr: worn-world detection transformer. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[12] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficien transfer learning for nlp. In International Conference on Machine Learning, 2019.
[13] Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[14] Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint wxXiv:2307.13266 2023.
[15] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi, Editing models with task arithmetic. International Conference on Learning Representations, 2023.
11

---

[16] Aishwarya Kamath, Mannat Singh. Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modullted detection for end-to-end multi-modal understanding. In IEEE International Conference on Computer Vixion, 2021.
[17] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Grahska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 2017.
[18] Jahanzaib Latif, Chuangbai Xiao, Azhar Imran, and Shanshan Tu. Medical imaging using machine learning and deep learning algorithms: a review. In International conference on computing, mathematics and engineering technologies, 2019.
[19] Chunyuan Li. Haotian Liu, Liunian Li, Pengchuan Zhang, Jyoti Aneja, Jianwei Yang. Ping Jin, Houdong Hu, Zicheng Liu, Yong Jac Lec, al. Elevater: A benchmark and toolkit for evaluating longgugg--uggeente visual models. Advances Neural Information Processing Systems, 2022.
[20] Liunian Harold Li, Pengchuan Zhang. Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan. Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-iman -mmgg pre- ave-training. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2022.
[21] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In International Joint Conference on Natural Language Processing, 2021.
[22] Zhong- Yu Li, Xin Jin, Yuan Sun, Chun-Le Guo, and Ming-Minn Cheng. Towards raw object detection in diverse conditions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2025.
[23] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Computer Vision, 2014.
[24] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. Dab-detr: Dynamic anchor boxes are better queries for detr. International Conference on Learning Representations. 2022.
[25] Shilong Liu, Zhanyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chumyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In Proceedings of the European Conference on Computer Vision, 2024.
[26] Yaoyao Liu, Bernt Schiele, Andrea Vedaldi, and Christian Rupprecht. Continual detection transformer for incremental object detection. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2023.
[27] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Reprexentations, 2019.
[28] Haoyu Lu, Yuqi Huo, Guoxing Yang, Zhiwu Lu, Wei Zhan, Masayoshi Tomizuka, and Mingyu Ding. Uniadapter: Unified parameter-efficient transfer learning for cross-modal modeling. In International Conference on Learning Representations, 2024.
[29] Gianluca Mancusi, Mattia Bernardi, Aniello Panariello, Angelo Porrello, Rita Cucchiara, and Simone Calderara. Is multiple object tracking a matter of specialization? Advances in Neural Information Processing Systems, 2024.
[30] Imad Eddine Marouf. Subhankar Roy. Enzo Tartaglione, and Stéphane Lathuilière. Weighted ensemble models are strong continual learners. In Proceedings the European Conference on Computer Vision, 2024.
[31] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, 1989.
[32] Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jingdong Wang. Conditional detr for fast training convergence. In IEFE International Conference on Computer Vision, 2021.
[33] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu, and Hassan Ghasemzadeh. Linear mode connectivity in multitask and continual leaming.. In International Conference on Learning Representations, 2021.
12

---

[34] Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Systems, 2024.
[35] Mark Palatucci, Dean Pomerleau, Geoffrey E Hinton, and Tom M Mitchell. Zero-shot learning with semantic output codes. Advances in Neural Information Processing Systems, 2009.
[36] Jonas Pfeiffer, Sebastian Ruder, Ivan Vulić, and Edoardo Ponti. Modular deep learning. Transactions on Machine Learning Research, 2023.
[37] Angelo Porrello, Lorenzo Bonicelli, Pietro Buzzega. Monica Millunzi, Simone Calderara, and Rita Cue- chiara. A second-order perspective model compositionality and incremental learning. In International Conference on Learning Representations, 2025.
[38] Christian Poss, Olim Ibragimov, Anoshan Indreswaran, Nils Gutsche, Thomas Irrenhauser, Marco Prueglmeier, and Daniel Goehring. Application open source deep neural networks for object detection industrial environments. In IEEE International Conference on Machine Learning and Applications, 2018.
[39] Yujia Qin, Cheng Qian, Jing Yi, Weize Chen, Yankai Lin, Xu Han, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Exploring mode connectivity for pre-trained language models. Conference on Empirical Methods in Natural Language Processing, 2022.
[40] Alee Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, 2021.
[41] Apoorva Raghunandan, Pakala Raghav, HV Ravish Aradhya, et al. Object detection algorithms for video surveillance applications. In International Conference on Communication and Signal Processing, 2018.
[42] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[43] Tal Ridnik, Emanuel Ben- Baruch, Asaf Noy, and Lihi Zelnik Imagenet-21k pretraining for the masses. In Conference on Neural Information Processing Systems Datasets and Benchmarks Truck, 2021.
[44] Riccardo Salami, Pietro Buzzega. Matteo Mosconi. Jacopo Bonato, Luigi Sabetta, and Simone Calderara. Closed-fore merging of parameter-efffcientt modules for federated continual learning. In International Conference on Learning Representations, 2025.
[45] Shuai Shao, Zeming Li. Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: large-scale, high-quality dataset for object detection. In IEEE International Conference on Computer Vision, 2019.
[46] James Scale Smith, Lconid Karlinsky, Vyshnavi Gutta, Paola Cascante- Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention- based prompting for rehearsal-free continual learning. In Pruceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023.
[47] Gido M Van de Ven, Tinne Tuytelaars, and Andreas S Tolias. Three types of incremental learning, Nature Machine Intelligence, 2022.
[48] A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
[49] Xin Wang, Thomas E Huang, Trevor Darrell, Joseph Gonzalez, and Fisher Yu. Frustratingly simple few-shot object detection. In International Conference on Machine Learning, 2020.
[50] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Y Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal- free continual learning. In Proceedings of the European Conference on Computer Vision, 2022.
[51] Zifeng Wang, Zizhao Zhang, Chen-Yi Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2022.
[52] Xun Wu, Shaohan Huang, and Furu Wei. Mixture of lora experts. In International Conference on Learning Representations, 2024.
[53] Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal, Ties merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 2024.
13

---

[54] Enneng Yang. Zhenyi Wang. Li Shen, Shiwei Liu, Guibing Guo. Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In International Conference Learning Representations, 2024.
[55] Yuhang Zang. Wei Li, Kaiyang Zhou, Chen Huang. and Chen Change Loy. Open-vocabulary detr with conditional matching. Proceedings of the European Conference on Computer Vision, 2022.
[56] Jinghan Zhang, Junteng Liu, Jumxian He, et al. Composing parameter--fficient modules with arithmetic operation. Advances Neural Information Processing Systems, 2023.
[57] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li. Xiaogang Wang. and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection, International Conference on Learning Representations. 2021.
14

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We substantiate the investigation of a warmup phase and the subsequent introduction of DitHub in Section 4 Section details the establishment of state-of- ootte-------trttt performance and the analysis of the compositional capabilities of class-specific modules, with complementary information provided in the Appendix.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
A specific section is presented in the Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
Justification:
This paper does not present theoretical results requiring formal proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We document the implementation details in Section 5.1 A supplementary section providing additional information, necessitated by space constraints, is included in the Appendix. Together, these sections and the available code furnish all the necessary particulars enable complete reproducibility of the paper's results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
16

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: The code is provided in the supplementary material, accompanied by README md file that details the functioning of the codebase and enables the reproduction of the experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.cc/ htaplss.cc/pbbiicguuu.odo
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 5.1 provides all particulars regarding the training and testing setup. A further integration of related information is reported in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: A dedicated section in the Appendix provides the standard deviations required to support the statistical significance of the experimental results detailed in the main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: A dedicated section in the Appendix details the computational resources utilized obtain the experimental results.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https:
Answer: [Yes]
Justification:
The work presented herein is compliance with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
A specific section is presented in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This work poses no elevated risks, as it avoids utilizing Large Language Models any tools with high propensity for misuse. Only pretrained and broadly adopted Lingua-Language detector, Grounding DINO [25], employed.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We acknowledge all authors the tested models and employed datasets within the main paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, papprsaipphaeeeooccaaappp thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
No new asset is provided in this contribution. The benchmark introduced herein, ODinW-O, leverages already existing resources. Comprehensive documentation for this benchmark is presented in the Appendix.
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/code/mmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Large Language Models were utilized solely for polishing the textual presenta- tion of this manuscript.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurd) httt::/:/earrrppccco/offeeereeee /c/////oeeeeeeee//222////.. for what should or should not be described.
21

---

Ethics Statement
This work focuses on improving the efficiency and robustness of object detection models. All experiments were conducted on publicly available datasets that do not contain personally identifiable information. The proposed methods could, principle, be applied to surveillance monitoring systems; however, our goal is to advance fundamental research in detection architectures rather than enabling privacy-invasive applications. encourage the responsible and transparent use these techniques in compliance with data protection regulations.
Appendix Overview
This appendix provides supplementary information to the main manuscript, organized as follows.
Section
Further Insights on Efficiency. Analyzes the performance optimization of DitHub when adapting Grounding DINO.
Blockwise Adaptation Analysis: Justifies adapting only the encoder for DitHub based on empirical metrics.
Section A7
Section Employing Full LoRA for Class-Specificity: Explores class-specific
LoRA, dltmllyyyy favoring parameter-efficient shared B matrix.
Section
Performance-Menene Tradeo-ff Details: Details the per-task mAP for the experiments introduced in the right panel of Figure
Section B
Further Insights on Forgetting. Evaluates the anti-forgetting capabilities DitHub.
Section Forgetting on ODinW-13: Presents anti-forgetting results for the ODinW- 13 benchmark.
Forgetting ODinW-O: Demonstrates robust anti-forgetting perfor- mance on the ODinW-O benchmark.
Section
B.2
Section Additional Benchmark Details. Provides comprehensive information about the
datasets used.
Section C.7 ODin W-13 and MS COCO: Describes the ODinW-13 dataset for spe- cialization and MS COCO (ZCOCO) for zero-shot evaluation.
Section C22 ODinW W-O: Introduces the ODinW-O benchmark for Incremental Learn
ing and domain shift analysis.
Section D
Additional Results. Presents complementary experimental results.
Section D.
Standard Deviations: Reports standard deviations for the reproduced and Table
methods
Section Further Unlearning Results: Illustrates the modular capability of DitHub for selective knowledge removal, as shown in Figure C
Section D
Performance with Swin-B Backbone: Shows the scaling capabilities of
DitHuu when employing a larger backbone.
Section D.4
Beyond ODinW-13: Presents results for the AODRaw benchmark.
Section Implementation Details. Outlines the experimental setup, including hardware, model configurations, and optimization specifics.
Section
Limitations and Broader Impact. Discusses the current limitations of the proposed
method and its broader societal implications.
Section
Qualitative Results. Provides visual comparisons in Figure D demonstrate the improved detection accuracy and robustness of DitHub.
22

---

Table A: Performance comparison across datasets when adapting only the decoder (Only Dec), both encoder and decoder (Dec/Enc), and encoder only (DitHub).
ZCoCO
Avg
Ac
Ag
Co
Eg
Mu
Ra
Th
DirHuh
47.01
62.19
34.02
50.65
70.46
08.56
49.28
65.57
69.58
71.10
06.65
70.88
52.82
79.30
08.18
Only Dee
46.11
54.35
28.00
35.22
65,
65.11
37.62
39.71
68. 08.11
66.27
35,22
67.31
39.46
73.87
65.30
Dec/Eine
46.55
89.92
32.84
43.16
71.47
67.44
49.78
02.84
70.32
70.53
49.24
70,80
46.12
77.05
67.48
Grounding-Dino
Grounding-Dino
Grounding-Dino
Image Backionn
Imaga Backbona
Image Backbonn
Encoder
Decoder
Encoder
Decoder
Encoder
Decoder
Text
Toxt
Text Backbons
Backbonn
Backbone
(a) Encoder
(b) Decoder
(c) Decoder/Encoder
Figure A: Blockwise analysis of adaptation strategies: (a) encoder-only adaptation (our selected
method), (b) decoder-only adaptation, and (c) joint encoder-decoder adaptation.
A Further Insights on Efficiency
A.1 Blockwise Adaptation Analysis
While Grounding DINO consists four main components an image backbone for visual encoding, textual backbone for language understanding, and an encoder-decoder pair for detection we focus our adaptation on the latter two. This choice is motivated by the need for a more detection-centric adaptation and efficiency considerations, accounts for more than half of Grounding DINO 's parameters. In Table A we present results for adapting only the encoder (Figure Aaj, only the decoder (Figure Ab),, and both encoder and decoder (Figure Ach. DitHub employs the first approach adapting only the encoder - as it delivers the best performance on both ZCOCO and Avg metrics.
the textual backbone, being a BERT Base model (5),
A.2 Employing Full LoRA for Class-Specificity
While our methodology leverages a shared and unique matrix for all tasks, Table B presents results obtained using class-specific pairs. In this variant of DitHub, both the A and B low- rank matrices of LoRA undergo the same spe- cialized optimization process: initial warmup phase followed by dedicated specialization. In other words, the training pipeline for matrix B mirrors that of matrix A described in Section[41 Applying our full performance across the tasks of O-1nW-13, albeit at the cost of doubling the number of trainable parameters (Figure 3). Prioritizing efficiency, we opt for a lightweight implementation that trades marginal performance gain for significantly reduced memory requirements. Notably, ZCOCO results favor our efficient approach, as single, unspecialized matrix remains more aligned with the pretraining distribution, thereby mitigating degradation in zero-shot capabilities.
Table B: LoRA full results on ODinW-13, with DitHub results taken from Table for reference.
Method
ZCOCO
Avg
DitHub
47.01
62.19
DitHub (LoRA full)
46.66
63.
strategies to both matrices yields a +1.03 mAP improvement in average
A.3 Seftormmane--Mmmor Trade-off Details
Table details the per-task mAP for the experiments introduced in Figure (right).
B Further Insights on Forgetting
This section presents an additional analysis of the anti-forgetting properties of DitHuu, complementing the study Section 5.2 by offering an alternative perspective. We report the forgetting metric for both ODinW-13 and ODin W-O, computed for DitHub and its primary competitor, ZiRa [4]. This
23

---

Table C: Performance of DitHub at different ranks the ODinW-13 benchmark.
Mothod
ZCOCO
Avg
Ac
Aq
Co
Mu
Pn
Pv
Pi
Po
Ra
Sh
Th
Ve
DutHub r
47.14
57.04
31.89
37.01
67.93
72.58
43.63
53376
69.80
67.56
92
72. 49
40.31
76.11
65. 27
DitHub y
46.66
60.25
35.86
19.59
11.93
74.27
51.37
63.37
67
70.50
39. 65
61.05
49. 19.39
76.32
06.11 11
DitHub
46.73
36.54
48. 09
71.13
74.45
50.46
68.20
69:97
64.97
78.13
DitHub 46.81
61.93
37.38
50.19
71 02
71.87
53.31
66. 06.09
68 08.83
70.16
19.33 31
66.31
58.55
78.61
65.88 85.85
DitHub
62.19
34.62
50.65
70:46
56
65.57
09.58
56.65
70.88
52.5
79.30
Table D: Forgetting metrics in terms of mAP for the ODinW-13 dataset. The numbers above each task indicate the order in which they were processed. Bold indicates best results.
10
13
1
6
Method
Avg
Ae
Aq
Co
Eg
Mu
Ps
Pv
Pi
P6
Ra
Sh
Th
Ve
ZiRa
-6.66
-4.31
-4.82
-1.80
0.00
-2.09
-8.71
-5.80
-7.76
-12.63
199.19
-1882
-9.85
-7.77
DitHub
-1.73
-1.56
-1.09
-2.34
0.00
-1.61
3.62
-1.91
1.54
-6.94
-7.27
-2.11
0.10
-2.31
metric calculated using the canonical definition 1 the difference in performance on a specific
task after learning the entire sequence of tasks compared the original performance that task immediately after its initial learning.
B.1 Forgetting on ODinW-13
Table D presents the forgetting in terms of mAP points for the 13 tasks of ODinW-13, with the average across all tasks reported as Avg. DitHub demonstrates strong performance both average and individual tasks, consistently outperforming its competitor by notable margin. Given the limited class overlap within ODinW-13, the primary component susceptible to catastrophic forgetting in this analysis is the shared matrix B.
Notably, for some tasks, this metric is even positive for our approach. We attribute this behavior to the initialization of the component. Specifically, the metric higher in the first learned task (Pa) While this might seem counterintuitive, the Pa dataset comprises very few instances, which means the matrix cannot undergo robust initialization. Consequently, subsequent tasks introduce greater data diversity, leading to improved performance compared to the initial one limited by scarce training data.
B.2 Forgetting on O-inW-O
Table reports the forgetting metric results for the ODinW-O benchmark. Unlike the previous analysis, this assessment allows us evaluate the forgetting of DitHub's 's class-specific A com- ponents for all classes within the dataset. Indeed, ODinW-O is specifically designed to test Incre- mental Learning capabilities, featuring complete class overlap across tasks. This overlap leads to catastrophic forgetting of the A, expert compo- nent for the same class across different domains. It's worth noting that even in this setting, which emphasizes the component, the forgetting of remains a concern. Even in this challenging context, DitHub surpasses its competitor by an appreciable margin, demonstrating difficulty only with the Ae task and highlighting the non-trivial nature of merging and preserving knowledge from the aerial domain.
Table Forgetting metrics in terms of mAP for the ODinW-O dataset. Bold indicates best results.
Method
Avg
Ae
Hw
Pv
Sd
Th
Ve
ZiRa DitHuh
-3.84
-0.08
0.00
-9.67
10066
-0.98
-1.02
-2.68
-4.75
0.00
-2.74
-9.51
1.74
-0.79
C Additional Benchmark Details
This section provides additional details regarding the datasets utilized in our evaluation framework. We first outline the key characteristics of ODinW-13 and MS COCO, as these datasets constitute the foundation for evaluating the specialization and zero-shot capabilities of DitHub. Subsequently,
The order in which tasks presented to the model is random and dependent the selected seed.

---

Person
Car
Truck
Dog
Boat
Figure B: Visualization of domain shifts for shared categories across different tasks within the WDinW-O benchmark.
we introduce and discuss the ODinW-O benchmark in greater depth, which we explicitly design to evaluate Incremental Learning performance under scenarios involving recurring categories across multiple tasks, accompanied by domain shifts. By leveraging these datasets, we establish a rigorous evaluation protocol to comprehensively assess model robustness in challenging real-world scenarios.
C.1 ODin W-13 and MS COCO
we assess our model specialization using the Object Detection in the Wild (ODinW- 13) (20] benchmark and the MS COCO (23] dataset to verify whether the zero-shot capabilities preserved after the sequential fine-tuning on ODinW-13.
Following 4
Unlike structured and standardized datasets such as MS COCO, ODinW-13 aggregates data from thirteen sources: Aerial Maritime Drone (Ae), Aquarium (Aq), Cottontail Rabbil (Co), EgoHands (Eg), Mushrooms (Mu). Packages (Pa), Pascal VOC (Pv), Pistols (Pi), Pothole (Po), Raccoon (Ra), Shellfish (Sh), Thermal Dogs and People (Th), Vehicles (Vc). This comprehensive collection covers rare object categories, diverse contextual conditions, and challenging domains. Consequently, the ODinW-13 benchmark offers an ideal scenario to evaluate the capability of DitHub to sequentially adapt heterogeneous task distributions.
Following the convention established by 4. we use the term ZCOCO to denote the zero-shot evaluation conducted on the MS COCO validation set comprising 5000 samples and consisting of annotations for 80 common object categories. The ZCOCO evaluation specifically measures the robustness and generalization capabilities of DitHub in zero-shot scenarios, after training on each successive task from O-1n
C.2 ODinW-O
We introduce a new benchmark, ODinW-O (Overlapped), designed to further evaluate the class- incremental and specialization capabilities of DitHub. Derived from ODinW-35 [1), ODinW-O retains only classes shared across at least one task (i.e., overlapping classes), while removing tasks with no shared classes. This filtering is crucial because retaining classes that do not reappear in subsequent tasks would cause the benchmark corresponding non-recurring class modules would remain frozen, requiring no further adaptation. In contrast, ODinW-O eliminates this scenario by ensuring all retained classes appear in multiple tasks. By enforcing strictly overlapping classes, ODinW-O provides a challenging benchmark for evaluating Incremental Learning. Additionally, to rigorously evaluate the model robustness to domain shifts, we designed O-in by explicitly selecting datasets that feature markedly different contexts (Figure B). enabling clear insights into how effectively the model adapts variations in visual appearance, environmental conditions, and imaging domains. We select the following datasets, already mentioned in Section C.1 and extracted only the overlapping classes:
resemble the original one. Specifically, the
Aerial Maritime Drone: boat and car;
*
Pascal VOC: boat, car, dog, and person;
25

---

Table F: Standard deviations of the reproduced methods in Table 1
Shots
Method
ZCOCO
Avg
Ae
Aq
Co
Eg
Mu
Pa
Pv
Pi
Po
Ra
Sh
Th
Ve
ZiRa
+0.90
=2.0
+1.49
+0.17
+2.16
=0.08
#0. 73
=1.0 133
+1.33
+7.31
#3.29
=12.17
#4446
=3.07
=3.25
Diilluh
10.60
20.81
10.05
1:0.7
12.66
1:158
13.72
-0.00
1:155
11.10
12.00
17.00
1:368
1:4.6
22.66
=0.43
#1.08
#1.54
#2200
DiHHuh
4:.17
$2.77
20.02
20.57
#1.04
#1.21
23.71
Ditllub
10.20
20.54
12.26
10.72
1.420
23.50
$1.35
23.19
4,0.30
4.2.74
10.34
42299
4:101
1:1.15
21.27
Full
ZIRA
Didlub
+0.44
10.10
20.44
-0.68
+2.13
12.39
11.00
+1.17
12.83
+1.39
+:4.77
20.08
+7.06
11.34
20.00
-2 -2.19
+1133
10.38
+-0.3
11.88
10.76
+4.14
+-355
11.57
+2.44
12.87
+2.88
10.00
21.01
-3.13
Thermal Dogs and People: dog and person;
Vehicles: car and truck.
Additionally, we introduce two further datasets from ODinW-35 to emphasize domain variation:
Hard Hat Workers: person;
Self Driving Cars: car and truck.
Despite ODinW-O enforcing class overlap and introducing domain shifts, our model consistently achieves state-of-the-an performance by effectively refining class knowledge while adapting across diverse contexts.
D Additional Results
D.1 Standard Deviations
We report the standard deviations of the reproduced methods from Table and Table ii Table SableFlan and respectively, computed over three runs with different random seeds.
bicycle
32.4
56.7
34.
84.0
79.0
77.2
bird
Removed class madule
boat
37.3
55.7
29.9
36.9
847
41.2
58.1
77.7
9.5
D.2 Further Unlearning Results
car
57.9
35.0
83.8
77.4
As previously discussed in Section 5.2 modu- lar framework facilitates the precise removal of spe- cific knowledge from Grounding DINO through the
horse
37.7
$7.5
33.5
35.1
$3.9
37.6
58.6
77.9
12.3
subtraction of corresponding experts. Figure C vi- sually represents this process: each column depicts the mean Average Precision (mAP) of target class after unlearning different modules (reported on the y-axis), with colors normalized column-wise to em- phasize relative performance decrements. The promi- nent darker diagonal pattern signifies that removing a class-specific module results in the most substan- tial performance reduction for its associated class. This observation substantiates that our modules effec- tively encapsulate specialized knowledge, as their un- learning selectively impacts their respective classes. Furthermore, this matrix representation elucidates correlations between certain classes during the unlcarning procedure. For instance, the darker shading in the "bicycle" column indicates that the "person" class exhibits the second most significant mAP drop. This correlation arises from the frequent co-occurrence of "person" instances riding bicycles, naturally linking these classes within the comprehensive unlearning pipeline.
person
75.0
train
54.3
32.1
83.0
38.4
71.0
tymonitar 37.7
57.7
35.
39.
83.9
40.7
58.9
75.0
bicycle-
bird
boatt
cat-
harse
person
train
tomonitor
Figure C: Per-class mAP degradation after removing class modules. Darker cells indicate greater performance drops.
D.3 Performance with Swin-B Backbone
To further assess the scalability of our approach, we conducted new experiments utilizing the larger, publicly available pre-trained Grounding DINO model with a Swin-B backbone. Table H details the
26

---

Table G: Standard deviations of the reproduced methods in Table
Method
ZCOCO
ZiRa
+0.25
+0.27
#1.7
#L14
+1.35
20.32
#1 75
#1.76
DitHlub
10.24
10.66
22.39
41.88
10.76
11.31
40.46
10.84
Table H: Performance Evaluation with the Swin-B backbone on ODinW-13.
Method
ZCOCO
Aug
Ae
Aq
Co
Eg
Mu
Pa
Pv
Th
Ve
GD ZS w/Swin-B
56.47
59.82
33.32
03.26
72.98
74.90
37.
39.14
70.30
73.56
48.42
67.30
62.61
70.69
64.18
w/Swin-B
73.32
73.97
47.25
72.39
Ditflub w/Swin-B
$3.82
67.03
42.41
53.74
74.53
75.18
53.28
68.05
74.04
74.22
39.25
76.51
67.63
82.85
71.67
comparative performance of our method, DitHub, against the primary competitor, ZiRa. The findings confirm that DitHub scales effectively, maintaining its robust performance when integrated with more powerful backbone model.
D.4 Beyond ODin W-13
To rigorously evaluate the generalizability of our approach, we have extended our experiments to more diverse benchmark. Our initial experimental setup necessitated benchmark with class re- occurrence across semantically distinct domains characteristic not commonly found standard long-tailed datasets.
For this purpose, we selected AODRaw dataset is ideally suited for our evaluation, as it comprises 9 distinct domains with substantial class overlap, encompassing a wide range of metcorological conditions as well both indoor and outdoor scenes. The performance analysis is presented in Table
22 a challenging and recently introduced benchmark. This
E
Implementation Details
All experiments for DitHub were conducted on single RTX A5000 GPU with 24 GB of VRAM. Each training run, with a batch size of 2, required approximately 8 hours. Our PyTorc implementation efficiently utilized around 10 GB of VRAM during training. We employed Grounding DINO Tiny as an Open-Vocabulan Vocabulary object detector, which was pre-trained on the O365 [45), GoldG [6], and Cap4M 20 datasets.
For LoRA fine-tuning, we applied low-rank updates the encoder layers with a rank of 16. The optimization was performed using AdamW (27). with a learning rate of x 10-3 and a weight decay of x 10-2. We exclusively relied on Grounding DINO's contrastive classification loss and localization loss, without introducing any additional loss functions. The final model comprised approximately 173 million parameters, with only 19.6 million LoRA parameters actively updated during training on ODinW-13. 131 After fine-tuning on ODinW-13, 13, only 75.65 MB of LoRA parameters were retained storage.
F
Limitations and Broader Impact
limitation the current work that the merging strategies employed by DitHub for both class- specific experts (A matrices) and the shared B matrix could be further optimized. While the current solutions demonstrated effectiveness, their reliance on empirically derived coefficients highlights a potential for refinement towards strategics grounded in more robust theoretical principles. Morcover, when merging modules pertaining to the same class but originating from different tasks, incorporating explicit domain knowledge becomes crucial. We hypothesize that a mechanism explicitly accounting for domain information (e.g., distinguishing between RGB and thermal images) could not only enhance DitHub's 's performance but also facilitate the development of novel expert module merging techniques. Despite these acknowledged limitations, DitHub achieves state-of-the-an performance, underscoring the efficacy of its expert module management approach.
propose that our method, and more broadly Modular Deep Learning, can serve as a cornerstone for fostering more accessible Artificial Intelligence. The development of methods for training
27

---

Table I: Domain wise performance comparison ODinW-13
Method
ZOOCO
Avg
Log. Jow light
log. rain
log
indour Jonw light
indoor mommal light
Jow. light_rinn
low Right
nermal, light
rain
GD zs
47.40
15.65
13.52
10.56
10.42
15.84
26.77
11:74
16.25
5.81
14.11
ZIRA
44.39
27.00
36.34
28.25
31.73
30.49
38.47
and managing libraries of expert modules could enable the granular selection and deployment of knowledge onto large, otherwise computationally intractable, pre-trained architectures. Specifically, object detectors to
DitHub facilitates the management of modules that enable large recognize specific and potentially rare classes. This specificity also contributes to enhanced model controllability, potentially enabling targeted unlearning capabilities for bias mitigation and thereby advancing towards fairer deep learning systems.
G Qualitative Results
In Figure compare zero-shot Grounding DINO (left) with ZiRA (center) and DitHub (right). Our method excels at detecting challenging objects, including small ones that other models miss. Additionally, DitHub avoids false positives (ZiRA) and false negatives (Grounding DINO), demon- strating more reliable predictions. It also correctly identifies both cars (last row) despite occlusions and distractors. This improved stability stems from per-class specialization, which mitigates halluci- nations and ensures smoother adaptation to domain shifts.
28

---

Grounding DINO
ZiRa
DitHub
Figure D: Qualitative comparison of Grounding DINO (left), ZiRa (center), and DitHub (right).
29


Reviews:
[{"review_id": "zQUZskJAvq", "reviewer": "Reviewer_voji", "summary": "DitHub is a modular framework for Incremental Open-Vocabulary Object Detection, which proposes (1) a two-stage LoRA fine-tuning recipe for efficient adaptation, (2) to manage class-specific modules in a Version Control Systems manner using `branch`, `fetch`, and `merge` operations, and (3) sharing a part of weights for MemoryEfficiency. The method is evaluated on ODinW-13 and ODinW-O with Grounding DINO and demonstrates superior performance to the baseline.", "questions": "- As the number of categories grows, an important question arises: how can the $A$ matrices be efficiently managed, particularly in scenarios where classes exhibit clear hierarchical or semantic dependencies?\n\n- More experiments are needed.", "limitations": "See Weaknesses and Questions.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 3, "strengths_and_weaknesses": "- Strengths \n\n  - In general this paper is well-written and easy to follow. \n  - Achieving new state-of-the-art results in open-vocabulary settings is a substantial contribution.\n  - Ablation results are detailed and show reasonable analysis of different model components.\n\n- Weaknesses\n\n  - In some scenarios, maintaining a separate matrix for each category can be expensive. For instance, in large-scale detection datasets like V3Det [1], which contains 13,029 categories, training and managing such a vast number of weights leads significant computational and storage costs.\n  - In Fig 8, paper only reports the average performance across the tasks of ODinW-13. The ZCOCO metric is also essential, as it provides insight into how different design choices affect the forgetting after fine-tuning.\n  - The paper evaluates the proposed method in only a single setting (Grounding DINO), without validation across different models or scales. The authors are encouraged to conduct experiments with diverse open-vocabulary detectors and model sizes to demonstrate the broader applicability of the approach.\n  - While the paper proposes a new benchmark (ODinW-O) to test class recurrence, most experiments are still focused on ODinW-13. Additional experiments on other open-vocabulary or long-tailed benchmarks ( e.g., LVIS [2], V3Det [1], or Objects365 [3] ) would strengthen generalization claims.\n\n[1] Wang, Jiaqi, et al. \"V3det: Vast vocabulary visual detection dataset.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[2] Gupta, Agrim, Piotr Dollar, and Ross Girshick. \"Lvis: A dataset for large vocabulary instance segmentation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.\n\n[3] Shao, Shuai, et al. \"Objects365: A large-scale, high-quality dataset for object detection.\" Proceedings of the IEEE/CVF international conference on computer vision. 2019.", "was_revised": true, "final_justification": "With the additional results and analysis, the authors have addressed most of my concerns, making the work overall novel and solid.", "rebuttal": "We sincerely thank Reviewer **voji** for their valuable feedback, for finding our manuscript to be well-written, and for recognizing our method's state-of-the-art performance as a substantial contribution. We also appreciate their acknowledgment of our detailed and analytically reasonable ablation studies. We will now address the reviewer's concerns in detail and incorporate all resulting adjustments into the final manuscript.\n\n**On the memory cost**\n\n> In some scenarios, maintaining a separate matrix for each category can be expensive. [...]\n\nWhile the memory cost can be a concern in some scenarios, our work adopts the exact experimental setting from [1] to ensure fair comparison and build upon established benchmarks. This framework focuses on enhancing a large, pre-trained model with a smaller set of specialized or rare categories (e.g., \"cottontail-rabbit\" or \"pothole\" from the ODinW-13 benchmark) or handling significant domain shifts (e.g., RGB to thermal), rather than on adding thousands of new classes.\n\nWithin this established setting, our use of class-specific modules is a deliberate design choice, as it is fundamental to our core contributions: exploring the composability of learned concepts and fine-grained unlearning. An architecture without this modularity would undermine our ability to investigate these properties.\n\nTo ensure this modularity remains efficient, we implement each module with only a single low-rank projection matrix ($A$, with dimensions $r \\times d$, where the rank $r \\ll d$). To further address the reviewer's valid concern, we have conducted new experiments specifically optimizing the memory consumption of our approach. These are detailed below and in our answer to Reviewer **Xs7M** (in the section \"On the memory cost\").\n\n> [...] how can the $A$ matrices be efficiently managed, particularly in scenarios where classes exhibit clear hierarchical or semantic dependencies?\n\nTo address the reviewer's insightful question, we designed a new experiment to evaluate the management of $A$ matrices based on **semantic hierarchies**. Following the DitHub pipeline, instead of training a specific $A$ component for each class, we define three macro-categories: *animals*, *vehicles*, and *objects*. When a class from one of these groups is encountered during training, the corresponding group matrix (e.g., $A_{animals}$) is updated.\n\nThis methodology allows us to study the semantic relationships within our learned modules and to analyze how individual class concepts can coexist within a single, hierarchically superior representation. The class-to-group mappings are as follows:\n\ngroup|classes\n-|-\nanimals|c-rabbit, raccoon, dog, cat, cow, sheep, horse, bird\nvehicles|car, bus, motorcycle, airplane, boat, train, truck, ambulance, bicycle\nobjects|pistol, package, chair, dining table, tv, couch, potted plant\n\nThe results, comparing the standard DitHub (per-class $A$) with the semantic aggregation approach, are reported below.\n\n&nbsp;|Avg|c-rabbit|raccoon|dog|cat|cow|sheep|horse|bird\n-|-|:-:|:-:|-|-|-|-|-|-\nDitHub|**70.51**|71.26|69.04|84.31|84.13|67.72|62.21|66.11|59.29\n$A_{animals}$|**68.08**|68.18|66.42|83.93|82.92|65.04|60.95|63.30|51.91\n\n&nbsp;|Avg|car|bus|motorcycle|airplane|boat|train|truck|ambulance|bicycle\n-|-|-|-|:-:|:-:|-|-|-|:-:|-\nDitHub|**62.79**|55.74|76.58|58.92|77.90|45.67|78.17|38.42|85.55|48.16\n$A_{vehicles}$|**58.47**|54.04|74.47|52.96|74.19|35.55|74.33|35.90|80.12|44.63\n\n&nbsp;|Avg|pistol|package|chair|dining table|tv|couch|potted plant\n-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:\nDitHub|**49.27**|71.23|65.08|38.93|21.24|62.90|52.14|33.38\n$A_{objects}$|**44.55**|62.90|58.99|34.89|15.92|60.32|49.10|29.75\n\nThe results are highly encouraging. The experiment confirms that a single group matrix can learn a robust, shared representation for an entire semantic category. We found that groups with high semantic relation (e.g., *animals*) are more composable than more disparate ones (e.g., *objects*), which we attribute to the semantic similarity between their constituent classes (e.g., *cat* and *dog* vs. *pistol* and *couch*).\n\nCrucially, this approach demonstrates a significant gain in efficiency. By reducing the number of trainable $A$ matrices from 24 to just 3, we observe only a minor impact on accuracy. This confirms that our modules can effectively leverage semantic dependencies, as the reviewer correctly hypothesized.\n\n**Additional metrics**\n\n> In Fig 8, paper only reports the average performance across the tasks of ODinW-13. The ZCOCO metric is also essential [...]\n\nWe thank the reviewer for this suggestion. As requested, we now report the **Zero-Shot COCO** (ZCOCO) metric along with the Avg AP scores across tasks. \n\n&nbsp;|ZCOCO|Avg\n-|-|-\nGD ZS|47.41|46.80\n`Base`|47.84|56.00\n`w/ WU`|47.38|58.30 \n`w/ Eq.3+WU`|47.22|58.79\nDitHub|47.01|62.19\n\nThe results show a progressive decrease in the ZCOCO metric as the average AP across tasks rises, which is an expected trade-off as specialization degrades zero-shot capabilities.\n\nNotably, the `Base` fine-tuning baseline scores higher on ZCOCO than the original Grounding DINO zero-shot model. We attribute this counter-intuitive result to catastrophic forgetting. Our full method, DitHub, is trained to be robust across the varied domains of ODinW. In contrast, the `Base` procedure completely forgets such broad knowledge and overfits to the new tasks. This process of forgetting diverse, out-of-distribution data results in a model that is more aligned with the data distribution of the COCO benchmark, leading to an artificially inflated ZCOCO score.\n\n**Backbone diversity**\n\n> The paper evaluates the proposed method in only a single setting (Grounding DINO), without validation across different models or scales. [...]\n\nTo address the concern regarding model scales, we have conducted new experiments using the larger, publicly available pre-trained version of Grounding DINO with a **Swin-B** backbone. The table below presents these new results for our method, DitHub, and the primary competitor, ZiRa.\n\n&nbsp;|ZCOCO|Avg|Ae|Aq|Co|Eg|Mu|Pa|Pv|Pi|Po|Ra|Sh|Th|Ve\n-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-\nGD ZS w/Swin-B|56.47|59.82|33.32|43.26|72.98|74.90|37.67|59.14|70.30|73.56|48.42|67.30|62.61|70.69|64.48\nZiRa w/Swin-B|53.22|65.09|40.71|54.16|73.32|73.97|47.25|67.03|72.39|73.29|59.83|73.05|61.98|82.59|66.63\nDitHub w/Swin-B|53.82|67.03|42.41|53.74|74.53|75.18|53.28|66.05|74.04|74.22|59.25|76.51|67.63|82.85|71.67\n\nThese results confirm that our method, DitHub, scales effectively and maintains its strong performance when integrated with a larger and more powerful backbone model.\n\nRegarding model diversity, our decision to focus on Grounding DINO was twofold. Firstly, while other Detector with large-scale pretraining on both Detection and Grounding data, like GLIP [2] exist, Grounding DINO [3] is its state-of-the-art successor, and a full adaptation of our method to GLIP would be infeasible within the rebuttal period. Secondly, and crucially, Grounding DINO is the foundational model used in the work that established our experimental setting [1], making it the most appropriate choice for a direct and fair comparison. For these reasons, we believe our in-depth evaluation on this single, highly challenging model provides a strong and valid test of our method's contributions.\n\n**Additional datasets**\n\n> [...] Additional experiments on other open-vocabulary or long-tailed benchmarks [...]\n\nTo address the reviewer's concern and better demonstrate the generalizability of our approach, we have conducted new experiments on two additional, diverse benchmarks. We excluded datasets like Object365, as it was part of the pre-training corpus for the Grounding DINO backbone and would not be a valid test set. Furthermore, our original benchmark choice was guided by our experimental setting, which specifically requires **class re-occurrence** across semantically distinct domains. This property is not typically found in standard long-tailed benchmarks, as they are generally not structured with overlapping classes across divergent domains.\n\nFirst, we present results on the large-vocabulary dataset **LVIS**. To make this evaluation computationally feasible while retaining the complete set of 1203 categories, we adopt an established procedure from the literature [4] that uses a 10% subset of the annotations.\n\n&nbsp;|ZCOCO|LVIS (AP)\n-|-|-\nGD ZS|47.41|15.20\nZiRA|46.67| 22.31\nDithub|46.91| 25.51\n\nAdditionally, we evaluated our method on **AODRaw** [5], a challenging benchmark recently introduced at CVPR 2025. This dataset is particularly well-suited to our setting as it features 9 distinct domains with significant class overlap, spanning varied meteorological conditions as well as indoor and outdoor scenes. The results are reported in the following table.\n\n&nbsp;|ZCOCO|Avg|fog_low_light|fog_rain|fog|indoor_low_light|indoor_normal_light|low_light_rain|low_light|normal_light|rain\n:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:\nGD ZS |47.40|15.65|13.32|19.56|19.42|15.84|24.77|11.74|16.28|5.81|14.11\nZiRA|44.39|32.34|27.60|36.34|29.28|28.28|43.18|30.67|31.73|30.49|33.47\nDithub|45.23|34.03|30.30|35.48|31.40|31.02|47.77|31.36|32.88|30.90|35.12\n\nThese additional results on both LVIS and the challenging AODRaw benchmark provide substantial new evidence for the robustness and generalizability of our proposed method beyond the primary ODinW-13 setting.\n\n[1] Deng, Jieren, et al. \"Zero-shot generalizable incremental learning for vision-language object detection.\" NeurIPS 2024.\n\n[2] Li, Liunian Harold, et al. \"Grounded language-image pre-training.\" CVPR 2022.\n\n[3] Liu, Shilong, et al. \"Grounding dino: Marrying dino with grounded pre-training for open-set object detection.\" ECCV 2024.\n\n[4] Kamath, Aishwarya, et al. \"Mdetr-modulated detection for end-to-end multi-modal understanding.\" ICCV 2021.\n\n[5] Li, Zhong-Yu, et al. \"Towards raw object detection in diverse conditions.\" CVPR 2025."}, {"review_id": "DF1wOm7VCz", "reviewer": "Reviewer_i82i", "summary": "This paper introduces DitHub, a modular system for incremental open-vocabulary object detection. Inspired by version control systems, DitHub manages a growing library of lightweight adaptation modules, allowing objects to be added or re-adapted efficiently as new tasks or domains are encountered. A key innovation is the decoupling of the adaptation process into a warmup phase (providing class-agnostic initialization) and a subsequent class-specific specialization phase. The framework is instantiated using LoRA modules and demonstrates state-of-the-art performance on both the ODinW-13 benchmark and the new ODinW-O (which tests recurring class adaptation). The authors systematically analyze memory, performance degradation, rare category adaptation, and unlearning capabilities, and include ablation and sensitivity studies.", "questions": "Please refer to weakness.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- DitHub is a thoughtfully constructed modular adaptation framework, directly addressing the need for selective, composable, and revisable adaptation in open-vocabulary object detection\n- Figure 3 directly compares the memory requirements of DitHub's modular scheme against naive class-specific adaptation, and clearly demonstrates the practical benefit of parameter sharing.\n\n\n### Weaknesses\n- In Table 2, comparison on ODinW-O is limited to ZiRa and G-Dino only. It is acknowledged that ODinW-O is new, but running additional strong baselines (such as dDETR or AT) would better contextualize claims of superiority and strengthen the evidence for state-of-the-art claims on this new task.\n- Although the shared $B$ matrix strategy (Figure 3) is an improvement, as the number of classes scales up (e.g., hundreds or thousands), even linear scaling could raise practical concerns, especially on limited-resource deployments. A discussion is warranted already in the main text, indicating under what conditions DitHub's memory profile could become a bottleneck and whether further compression/merging strategies are feasible.\n- The mechanism for merging expert modules when encountering recurring classes (Sec. 4.1, Eq. 2) is motivated and evaluated, but deeper analysis of potential information loss/conflict when merging across heavily divergent domains is lacking. It will strengthen the work to directly visualize, for example, failure cases or class confusion arising from merging experts trained on very distinct domains.", "was_revised": true, "final_justification": "I would like to keep my original rating.", "rebuttal": "We sincerely thank Reviewer **i82i** for their constructive feedback and for acknowledging the strengths of our work. We are particularly grateful for the reviewer's recognition that our methodology successfully addresses the critical need for selective, composable, and revisable adaptation—our primary objective—and for their acknowledgment of our model's memory efficiency over naive approaches.\n\nWe will now address each of the reviewer's concerns in detail, and all corresponding revisions will be integrated into the final manuscript.\n\n**Additional baselines on ODinW-O**\n> In Table 2, comparison on ODinW-O is limited to ZiRa and G-Dino only. [...]\n\nAs requested, we have expanded our comparison on the ODinW-O benchmark to include the results for **iDETR** and **AT**. For completeness and ease of comparison, the updated table also retains G-Dino and ZiRa and includes our method, DitHub. This revised table will replace the original Table 2 in the final manuscript.\n\n||ZCOCO|Avg|Ae|Hw|Pv|Sd|Th|Ve\n|---|---|---|---|---|---|---|---|---\n|GDino|47.41|53.15|45.12|67.54|58.11|25.84|70.40|51.87\n|iDETR|37.42|59.02|49.69|67.76|67.72|35.95|78.44|54.58\n|AT|44.94|56.89|46.89|65.79|67.28|30.56|75.98|54.83\n|ZiRa|44.43|57.64|39.92|68.00|64.90|46.26|77.26|49.47\n|DitHub|46.51|62.38|53.35|71.07|71.01|41.75|80.21|56.90\n\nThe expanded results in the table are consistent with our findings in Table 1. While iDETR outperforms ZiRa on the average metric (Avg), it demonstrates weaker performance in the zero-shot scenario (ZCOCO). Conversely, AT achieves a lower average score but is more effective at preserving zero-shot capabilities, yielding results comparable to ZiRa. Our method, DitHub, confirms its state-of-the-art performance by a consistent margin.\n\n**On the memory cost**\n> [...] DitHub's memory profile could become a bottleneck [...]\n\nWe thank the reviewer for acknowledging both the existing discussion on potential memory bottlenecks within the original manuscript and our efforts to enhance memory efficiency over naive class-specificity.\n\nFor a more comprehensive analysis, including additional experimental studies on this topic, we kindly refer the reviewer to our detailed responses to Reviewer **voji** and Reviewer **Xs7M** (\"On the memory cost\" sections).\n\n**Analysis on divergent domains**\n> [...] deeper analysis of potential information loss/conflict when merging across heavily divergent domains [...]\n\nWe agree that investigating the effects of merging modules from heavily divergent or semantically conflicting domains is a valuable research direction, which we also highlighted in the conclusions of our original manuscript. Prompted by the reviewer's feedback, we designed a new experiment to shed further light on this matter.\n\nThe experiment focuses on the class \"Person\" and compares two merging scenarios. In the first scenario (\"Similar Domains\"), we merge two class-specific modules ($A_{class}$) that were both trained on standard RGB image datasets (e.g., Pascal VOC, Pv, and Hardhat Workers, Hw). In the second scenario (\"Divergent Domains\"), we merge one module trained on a standard RGB dataset (Pv) with another trained on a heavily divergent dataset—the Thermal (Th) dataset. We then evaluate the zero-shot performance of the merged modules from both scenarios on the COCO validation set. The results are presented below.\n\n|Merged Domain|ZCOCO|Pv|Th|Hw\n|:---:|:---:|:---:|:---:|:---:\n|Similar|**59.89**|67.00|-|68.24 \n|Divergent|**57.64**|67.00|72.92|-\n\nWe repeat the same experiment for the class \"Car,\" using the Aerial (Ae) dataset as the divergent domain and the Vehicles (Ve) dataset as the similar one. The results are presented in the following table.\n\n|Merged Domain|ZCOCO|Pv|Ae|Ve\n|:---:|:---:|:---:|:---:|:---:\n|Similar|**48.21**|66.51|-|61.01\n|Divergent|**47.66**|66.51|72.77|-\n\nThe results show that merging modules from divergent domains is, in general, more challenging than merging modules from similar ones. As the results for the \"Person\" class demonstrate, merging modules from **similar domains** yields a higher ZCOCO score (59.89) compared to merging from **divergent domains** (57.64). This modest drop suggests that a minor degree of negative interference may occur when merging modules from significantly different data distributions, as the reviewer hypothesized. For the \"Car\" class, the drop in performance is less pronounced, suggesting that the Aerial domain conflicts less with standard RGB images than the Thermal domain does."}, {"review_id": "MXrq3yPYXK", "reviewer": "Reviewer_Xs7M", "summary": "DitHub is a framework for open-vocabulary object detection inspired by the version control system, which manages a growing library of detection modules. When encountering a new task, a class-agnostic warmup phase is adopted to provide initialization for the later specialization stage. The specialization stage will decide if a new task has common classes with previous learned tasks (the \"fetch\" operation) and initialize the training from there. Otherwise, it will train from the warm-up results. The authors provide detailed experiments and analysis to show the effectiveness of this method.", "questions": "1. If B is not shared (class-specific), what is the difference between A and B?\n2. Can this method apply to other open-vocabulary tasks, like classification?\n3. Following weakness 3, what is DitHub's memory usage compared to baselines?", "limitations": "Overall, I think the idea of training LoRA matrices to fine-tune Grounding-DINO for each class is a bit too simple and inefficient. Combining general and specialized information is also not original for open-vocabulary incremental learning. For other limitations, please see the weakness and question section. No potential negative societal impact.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The basic idea of this paper is to combine \"general\" information with \"specialized\" information. From a training aspect, it divides the training stage into \"warmup\" and \"specialization\". From a modular aspect, it uses a shared B matrix and a class-specialized A matrix. This idea is straightforward and has proven to be useful in ablation studies.\n2. The introduction of the dataset ODinW-O presents realistic scenarios where classes can reoccur.\n3. The experiment section is detailed and thoroughly.\n4. Codes are attached. \n\nWeaknesses:\n1. I am not a big fan of the terms used in this paper. I think the idea of using terms like \"merged\", \"fetch\", and \"branches\" is for readers to associate the concepts of this paper with GitHub, but somehow it makes this paper more difficult to understand, as these terms do not have the same meanings as the ones used in GitHub. But it's a personal preference. Some reviewers may love it. You don't actually need to response to it in the rebuttal.\n2. I think the method section should be better arranged. 4.1 focuses on two-stage training: warmup and specialization, and 4.2 focuses on memory efficiency, which proposes to share the matrix B instead of a distinct B for each class. However, B is shared has already been mentioned in 4.1. Meanwhile, Figure 3 the legend (A, B) cannot present that (A, B) is distinct for each class. Overall, I think this section is not easy to read and needs to be further polished.\n3. I think training a LoRA matrix for each class is not a significant contribution and is still far from efficient and scalable. \n4. The method is only tested on one benchmark ODinW-13. Seems a bit limited.", "was_revised": true, "final_justification": "The authors provide additional experiment results on semantic hierarchies and an additional dataset. Although I don't think the idea is intriguing, due to its SOTA performance and comprehensive evaluation, I think this paper can be accepted. Hence I am willing to raise my score to Borderline Accept.", "rebuttal": "We sincerely thank Reviewer **Xs7M** for their constructive feedback and positive assessment of our work. We particularly appreciate their acknowledgment of our straightforward and effective warm-up strategy, the thoroughness of our experimental section, the realism of our novel ODinW-O benchmark, and the value of our provided codebase—an acknowledgment we deeply value. We will now address each of their concerns in detail and incorporate all resulting adjustments into the final manuscript.\n\n**Rearrangement of the method section**\n> I think the method section should be better arranged. 4.1 focuses on two-stage training [...]\n\nWe thank the reviewer for this constructive feedback. Accordingly, we will adopt the recommended rearrangement of Sections 4.1 and 4.2 and adjust the positioning of Figure 3 in the final version of the paper.\n\n\n**On the memory cost**\n> I think training a LoRA matrix for each class is not a significant contribution and is still far from efficient and scalable.\n\nWe respectfully clarify that the use of per-class LoRA modules is not presented as our primary contribution, but rather as a necessary architectural choice to enable our main research goals: investigating the fine-grained compositional properties of fine-tuning modules—a direction previously unexplored in the context of open-vocabulary object detection. These properties include not only module addition, which is instrumental for incremental learning, but also module subtraction, enabling targeted unlearning. The warm-up phase, which the reviewer kindly acknowledged, is crucial to this design. It serves not only to extract general knowledge but also to initialize the class-specific modules in a shared representation space. Indeed, as suggested by prior work [1, 2], a common starting point for adaptation greatly enhances model compositionality, which is a central theme of our investigation.\n\nTo address the valid concerns about efficiency and scalability, we refer the reviewer to our response to Reviewer **voji** (section \"On the memory cost\"), where we introduce a more **parameter-efficient variant**. This variant trains semantic-specific modules rather than class-specific ones, a strategy that significantly reduces memory overhead by grouping related classes together. Furthermore, we invite the reviewer to proceed to the following section, where we have devised additional experiments and considerations to specifically address this point.\n\n> Following weakness 3, what is DitHub's memory usage compared to baselines?\n\nPrompted by the reviewer's question, we conducted additional experiments to analyze the trade-off between performance and memory usage in our class-specific modules. Specifically, we evaluated how the average AP on ODinW-13 varies with the rank $r$ of our matrices. The results are presented in the table below.\n\n|Method|ZCOCO|Avg|Ae|Aq|Co|Eg|Mu|Pa|Pv|Pi|Po|Ra|Sh|Th|Ve\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:\n|DitHub $r=1$|47.14|57.04|31.89|37.01|67.93|72.58|43.63|53.76|69.80|67.86|42.92|72.49|40.31|76.11|65.27\n|DitHub $r=2$|46.66|60.25|35.86|49.59|71.93|74.27|54.37|63.37|67.61|70.50|39.65|64.05|49.39|76.32|66.41\n|DitHub $r=4$|46.73|60.53|38.54|48.59|71.13|74.45|50.46|61.49|68.20|69.97|41.48|64.97|55.64|78.13|63.84\n|DitHub $r=8$|46.81|61.93|37.38|50.19|71.02|74.37|53.34|66.09|68.83|70.16|49.31|66.31|53.55|78.64|65.85\n|DitHub $r=16$|47.01|62.19|34.62|50.65|70.46|68.56|49.28|65.57|69.58|71.10|56.65|70.88|52.82|79.30|68.18\n\nAs shown, performance degrades gracefully even as the rank substantially decreases. Notably, a rank of $r=2$ achieves a result less than 2 AP points below our best-performing model ($r=16$), while representing an 8-fold reduction in per-class parameters and memory usage. We also test the extreme case where $r=1$, which collapses the adaptation matrix into a vector, and we find the performance remains remarkably strong. We attribute this robustness to the granularity of our framework; since each module only needs to encode the knowledge of a single class, it can do so effectively with fewer parameters compared to approaches that require information aggregation from multiple classes into a single module.\n\nBelow, we provide the direct memory usage comparison (in MB) against baselines, as requested by the reviewer.\n\n|Method|Memory (MB)|ODinW-13 (AP)\n|:---|:---:|:---:\n|DitHub $r=1$|~9|57.04\n|ZiRA|~18|57.98\n|DitHub $r=2$|~18|60.26\n|DitHub $r=4$|~37|60.53\n|DitHub $r=8$|~74|61.93\n|DitHub $r=16$|~147|62.19\n|GDino *(encoder only)*|~659|46.80\n\nThe results show that our method with $r=2$ employs approximately the same memory as ZiRA while performing significantly better (+2.28 AP). Furthermore, our $r=1$ variant delivers surprisingly competitive results, scoring less than 1 AP point below ZiRa while using only half the memory.\n\n**Additional datasets**\n> The method is only tested on one benchmark ODinW-13. [...]\n\nOur focus on ODinW-13 was initially motivated by its suitability for testing incremental learning across domain gaps and because it is the primary benchmark used by ZiRa, the method that introduced our experimental setting.\n\nHowever, we agree with the reviewer's point and have now performed additional experiments on two diverse datasets. We evaluated our method on the large-vocabulary LVIS benchmark, as well as on AODRaw, a recent and challenging benchmark encompassing different meteorological and environmental conditions. We kindly refer the reviewer to our detailed discussion and results in the response to Reviewer **voji** (section Additional datasets).\n\n**Other questions**\n> If $B$ is not shared (class-specific), what is the difference between $A$ and $B$?\n\nIndeed, as the reviewer suggested, when matrix $B$ is not shared, it is also class-specific. In this configuration, we do not enforce any distinct roles upon $A$ and $B$. Their purpose becomes a collective one: to jointly learn a low-rank matrix ($\\Delta W=BA$) that optimally adapts the model's behavior for that class. The empirical results for this configuration are reported in Appendix A.2.\n\n> Can this method apply to other open-vocabulary tasks, like classification?\n\nYes, we believe the core principles of our method are indeed applicable to other open-vocabulary tasks, such as classification. For instance, one could maintain a library of modules to incrementally adapt the vision or text encoder of a model like CLIP [3] for specialized zero-shot classification tasks (e.g., adapting it to a fine-grained domain like \"species of birds\" or \"models of cars\").\n\nHowever, we chose to focus on Object Detection as we believe it is a more challenging task and, therefore, better suited to studying the compositional properties of specialized modules (e.g., composing modules to detect multiple, distinct classes in a single image).\n\n[1] Mirzadeh, Seyed Iman, et al. \"Linear Mode Connectivity in Multitask and Continual Learning.\" ICLR 2021.\n\n[2] Qin, Yujia, et al. \"Exploring Mode Connectivity for Pre-trained Language Models.\" Conference on Empirical Methods in Natural Language Processing 2022.\n\n[3] Radford, Alec, et al. \"Learning transferable visual models from natural language supervision.\" ICML 2021."}]

Author rebuttal:

Meta-review: The paper received all positive reviews, leading to a final acceptance recommendation.
Decision: Accept (poster)