Title: DitHub: A Modular Framework for Incremental Open-Vocabulary Object Detection
Venue: NeurIPS 2025
Accepted: yes
Keywords: Open-Vocabulary Object Detection

Modular Deep Learning

Incremental Learning
Abstract: Open-Vocabulary object detectors can generalize to an unrestricted set of categories through simple textual prompting. However, adapting these models to rare classes or reinforcing their abilities on multiple specialized domains remains essential. While recent methods rely on monolithic adaptation strategies with a single set of weights, we embrace modular deep learning. We introduce DitHub, a framework designed to build and maintain a library of efficient adaptation modules. Inspired by Version Control Systems, DitHub manages expert modules as branches that can be fetched and merged as needed. This modular approach allows us to conduct an in-depth exploration of the compositional properties of adaptation modules, marking the first such study in Object Detection. Our method achieves state-of-the-art performance on the ODinW-13 benchmark and ODinW-O, a newly introduced benchmark designed to assess class reappearance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDitHub is a framework for open-vocabulary object detection inspired by the version control system, which manages a growing library of detection modules. When encountering a new task, a class-agnostic warmup phase is adopted to provide initialization for the later specialization stage. The specialization stage will decide if a new task has common classes with previous learned tasks (the \"fetch\" operation) and initialize the training from there. Otherwise, it will train from the warm-up results. The authors provide detailed experiments and analysis to show the effectiveness of this method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The basic idea of this paper is to combine \"general\" information with \"specialized\" information. From a training aspect, it divides the training stage into \"warmup\" and \"specialization\". From a modular aspect, it uses a shared B matrix and a class-specialized A matrix. This idea is straightforward and has proven to be useful in ablation studies.\n2. The introduction of the dataset ODinW-O presents realistic scenarios where classes can reoccur.\n3. The experiment section is detailed and thoroughly.\n4. Codes are attached. \n\nWeaknesses:\n1. I am not a big fan of the terms used in this paper. I think the idea of using terms like \"merged\", \"fetch\", and \"branches\" is for readers to associate the concepts of this paper with GitHub, but somehow it makes this paper more difficult to understand, as these terms do not have the same meanings as the ones used in GitHub. But it's a personal preference. Some reviewers may love it. You don't actually need to response to it in the rebuttal.\n2. I think the method section should be better arranged. 4.1 focuses on two-stage training: warmup and specialization, and 4.2 focuses on memory efficiency, which proposes to share the matrix B instead of a distinct B for each class. However, B is shared has already been mentioned in 4.1. Meanwhile, Figure 3 the legend (A, B) cannot present that (A, B) is distinct for each class. Overall, I think this section is not easy to read and needs to be further polished.\n3. I think training a LoRA matrix for each class is not a significant contribution and is still far from efficient and scalable. \n4. The method is only tested on one benchmark ODinW-13. Seems a bit limited.\n\nQUESTIONS:\n1. If B is not shared (class-specific), what is the difference between A and B?\n2. Can this method apply to other open-vocabulary tasks, like classification?\n3. Following weakness 3, what is DitHub's memory usage compared to baselines?\n\nLIMITATIONS:\nOverall, I think the idea of training LoRA matrices to fine-tune Grounding-DINO for each class is a bit too simple and inefficient. Combining general and specialized information is also not original for open-vocabulary incremental learning. For other limitations, please see the weakness and question section. No potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DitHub, a modular system for incremental open-vocabulary object detection. Inspired by version control systems, DitHub manages a growing library of lightweight adaptation modules, allowing objects to be added or re-adapted efficiently as new tasks or domains are encountered. A key innovation is the decoupling of the adaptation process into a warmup phase (providing class-agnostic initialization) and a subsequent class-specific specialization phase. The framework is instantiated using LoRA modules and demonstrates state-of-the-art performance on both the ODinW-13 benchmark and the new ODinW-O (which tests recurring class adaptation). The authors systematically analyze memory, performance degradation, rare category adaptation, and unlearning capabilities, and include ablation and sensitivity studies.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- DitHub is a thoughtfully constructed modular adaptation framework, directly addressing the need for selective, composable, and revisable adaptation in open-vocabulary object detection\n- Figure 3 directly compares the memory requirements of DitHub's modular scheme against naive class-specific adaptation, and clearly demonstrates the practical benefit of parameter sharing.\n\n\n### Weaknesses\n- In Table 2, comparison on ODinW-O is limited to ZiRa and G-Dino only. It is acknowledged that ODinW-O is new, but running additional strong baselines (such as dDETR or AT) would better contextualize claims of superiority and strengthen the evidence for state-of-the-art claims on this new task.\n- Although the shared $B$ matrix strategy (Figure 3) is an improvement, as the number of classes scales up (e.g., hundreds or thousands), even linear scaling could raise practical concerns, especially on limited-resource deployments. A discussion is warranted already in the main text, indicating under what conditions DitHub's memory profile could become a bottleneck and whether further compression/merging strategies are feasible.\n- The mechanism for merging expert modules when encountering recurring classes (Sec. 4.1, Eq. 2) is motivated and evaluated, but deeper analysis of potential information loss/conflict when merging across heavily divergent domains is lacking. It will strengthen the work to directly visualize, for example, failure cases or class confusion arising from merging experts trained on very distinct domains.\n\nQUESTIONS:\nPlease refer to weakness.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDitHub is a modular framework for Incremental Open-Vocabulary Object Detection, which proposes (1) a two-stage LoRA fine-tuning recipe for efficient adaptation, (2) to manage class-specific modules in a Version Control Systems manner using `branch`, `fetch`, and `merge` operations, and (3) sharing a part of weights for MemoryEfficiency. The method is evaluated on ODinW-13 and ODinW-O with Grounding DINO and demonstrates superior performance to the baseline.\n\nSTRENGTHS AND WEAKNESSES:\n- Strengths \n\n  - In general this paper is well-written and easy to follow. \n  - Achieving new state-of-the-art results in open-vocabulary settings is a substantial contribution.\n  - Ablation results are detailed and show reasonable analysis of different model components.\n\n- Weaknesses\n\n  - In some scenarios, maintaining a separate matrix for each category can be expensive. For instance, in large-scale detection datasets like V3Det [1], which contains 13,029 categories, training and managing such a vast number of weights leads significant computational and storage costs.\n  - In Fig 8, paper only reports the average performance across the tasks of ODinW-13. The ZCOCO metric is also essential, as it provides insight into how different design choices affect the forgetting after fine-tuning.\n  - The paper evaluates the proposed method in only a single setting (Grounding DINO), without validation across different models or scales. The authors are encouraged to conduct experiments with diverse open-vocabulary detectors and model sizes to demonstrate the broader applicability of the approach.\n  - While the paper proposes a new benchmark (ODinW-O) to test class recurrence, most experiments are still focused on ODinW-13. Additional experiments on other open-vocabulary or long-tailed benchmarks ( e.g., LVIS [2], V3Det [1], or Objects365 [3] ) would strengthen generalization claims.\n\n[1] Wang, Jiaqi, et al. \"V3det: Vast vocabulary visual detection dataset.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[2] Gupta, Agrim, Piotr Dollar, and Ross Girshick. \"Lvis: A dataset for large vocabulary instance segmentation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.\n\n[3] Shao, Shuai, et al. \"Objects365: A large-scale, high-quality dataset for object detection.\" Proceedings of the IEEE/CVF international conference on computer vision. 2019.\n\nQUESTIONS:\n- As the number of categories grows, an important question arises: how can the $A$ matrices be efficiently managed, particularly in scenarios where classes exhibit clear hierarchical or semantic dependencies?\n\n- More experiments are needed.\n\nLIMITATIONS:\nSee Weaknesses and Questions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 3, "DATE": "2025-07-12"}