iDPA: Instance Decoupled Prompt Attention for Incremental Medical Object Detection
Authors: Huahui Yi
Wei Xu
Ziyuan Qin
Xi Chen
Xiaohu Wu
Kang Li
Qicheng Lao
Venue: ICML 2025 poster (2025)
Keywords: Continual Learning.+Medical Object Detection
Primary area: general_machine_learning->transfer_multitask_and_metalearning
TL;DR: iDPA improves incremental medical object detection by introducing instance-level prompt generation and decoupled prompt attention, achieving superior performance on ODinM-13 across various data settings.

Abstract: Existing prompt-based approaches have demonstrated impressive performance in continual learning, leveraging pre-trained large-scale models for classification tasks; however, the tight coupling between foreground-background information and the coupled attention between prompts and image-text tokens present significant challenges in incremental medical object detection tasks, due to the conceptual gap between medical and natural domains. To overcome these challenges, we introduce the iDPA framework, which comprises two main components: 1) Instance-level Prompt Generation (IPG), which decouples fine-grained instance-level knowledge from images and generates prompts that focus on dense predictions, and 2) Decoupled Prompt Attention (DPA), which decouples the original prompt attention, enabling a more direct and efficient transfer of prompt information while reducing memory usage and mitigating catastrophic forgetting. We collect 13 clinical, cross-modal, multi-organ, and multi-category datasets, referred to as ODinM-13, and experiments demonstrate that iDPA outperforms existing SOTA methods, with FAP improvements of f 5.44%, 4.83%, 12.88%, and 4.59% in full data, 1-shot, 10-shot, and 50-shot settings, respectively.

iDPA: Instance Decoupled Prompt Attention for Incremental Medical Object Detection
Huahui Yi
Wei Xu Ziyuan Qin
Xi Chen
Xiaohu Wu Kang Li 17t
Qicheng Lao 6
Abstract
1. Introduction
Existing prompt-based approaches have demon- strated impressive performance in continual learn- ing, leveraging pre-trained large-scale models for classification tasks; however, the tight cou- pling between burgrrouddbacckrronn informa- tion and the coupled attention between prompts and image-text tokens present significant chal- lenges in incremental medical object detection tasks, due to the conceptual gap between medical and natural domains. To overcome these chal- lenges, we introduce the iDPA framework, which comprises two main components: 1) Instance- level Prompt Generation (IPG), which decouples fine-grained instance-level knowledge from im- ages and generates prompts that focus on dense predictions, and 2) Decoupled Prompt Attention (DPA), which decouples the original prompt atten- tion, enabling a more direct and efficient transfer of prompt information while reducing memory usage and mitigating catastrophic forgetting. We collect 13 clinical, cross- multi-organ, and multi-category datasets, referred to as ODinM- 13, and experiments demonstrate that iDPA out- performs existing SOTA methods, with FAP im- provements of 5.44%, 4.83%, 12.88% and 4.59% in full data, 1-shot, 10-shot, and 50-shot set- tings, respectively, Code is available at https: / //github.... com wtt.eeeeeiiii/iiii.... git.
Attention
Promots
Pe-Ri
Pg
Prior Work
Image-led
Po-+
CLS
nodule
Tokens
Instances
Decoupled Prompt Attention
Generate
P--V
Our
Decoupling
Work
V
Image
Tokere
Pr P
Figure 1: Comparison of our method (iDPA) with prior methods, highlighting improved object localization and recognition through instance-level prompt generation and decoupled prompt attention for medical detection tasks.
Vision Language Object Detection (VLOD) (Li et al., 2022; Liu et al., 2023a; Cheng et al., 2024), a paradigm that enables recognition of novel categories and scenes, has advanced object detection (OD) by integrating a lan- guage branch and leveraging large-scale image-text datasets. While these models demonstrate strong zero-shot capabili- ties in general domains, they struggle in the medical domain due to the degradation of medical object localization and recognition. However, neither fine-tuning separate OD mod- els for each task nor jointly training a single model for all tasks is practical, as maintaining multiple models is ineffi- cient and predefining all medical concepts is infeasible. In- stead, Continual Learning (CL) (Li & Hoiem, 2017; Rebufii et al., 2017; Wang al., 2022c) is essential for adapting emerging medical concepts while retaining prior knowledge. It must balance stability and plasticity to enable continuous learning and improve healthcare outcomes.
West China Biomedical Big Data Center. West China Hospital, Sichuan University "School Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technol- ogy of China CCase Western Reserve University "Sports Medicine Center, Department of Orthopedics and Orthopedic Research In- stitute, West China Hospital, West China School of Medicine, Sichuan University 'Department of Orthopedics and Orthope- die Research Institute, West China Hospital, Sichuan University 'Beijing University of Posts and Telecommunications 'Sichuan University Pittsburgh Institute. Correspondence to: Kang Li d wchscu.cn>, Qicheng Lao <qicheng www.nng...l@ubu....oo bupt.edu.cn.
Recently, prompt-based CL approaches (Wang et al., 2022c;b;a Smith al., 2023) have gained popularity for encoding knowledge into prompt sets, enabling frozen pre-trained model to handle sequential tasks. Compared to previous methods, these approaches not only achieve remarkable performance but also offer key advantages for continual learning. By keeping the base model unchanged and tuning only the prompt vectors, training efficiency is enhanced while eliminating the need for exemplar storage,
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

iDPA: Instance Decoupled Prompt Attention for IMOD
reducing both memory overhead and computational costs. However, these methods are primarily designed for classifi- cation tasks and are not well-suited for OD. Unlike classifi- cation, which relies on global information, object detection demands aner-grained instance information. In previous prompt-based CL methods, as shown at the top of Fig. 1, global prompt learning incorporates both foreground and background information, which can interfere with detection tasks. Furthermore, excessive background information can confuse category recognition, especially when task modali- ties are similar, leading to misclassification. Additionally, prepending prompts directly to the image and text tokens
by preserving the original category distribution. Finally, it strategically employs the fusion encoder as the optimal knowledge injection position, which is critical for learning new medical concepts and further enhances efficiency.
In this paper, our contributions are summarized in threefold:
We propose a novel prompt-based framework iDPA to effectively address incremental medical object detec- tion (IMOD), It decouples instance-level knowledge and efficiently injects it into VLOD models through DPA.
To evaluate the effectiveness of our method, we compile medical data from 13 datasets covering multiple modali- ties and organs, forming ODinM-13 for the IMOD task.
dilutes the prompt information because the length of image- text tokens far exceeds those of prompts, coupling the two
Extensive experiments demonstrate the effectiveness of our proposed approach. achieving state-of-the of--oh--tthh---- art perfor- mance with only 1.4% of the trained parameters in both full-data and few-shot settings.
and hindering task-specific learning.
Given the complex attention interactions between vision and language in VLODD, the prepending approach introduces fur- ther interference between vision and text prompts, Lastly, inserting prompts into pre-trained model, i.e., at the back- bone level, limits the effectiveness of tuning, as fine-grained reasoning for detection occurs post-backbone, i.e., at the fusion level.
2. Related Work
Vision Language Object Detection. Vision- Language Models (VLMs) enhance generalization by aligning visual and textual features through large-scale image-text learning. CLIP (Radford et al., 2021) and ALIGN (Li et al., 2021) leverage contrastive learning to associate images with text, inspiring GLIP (Li al., 2022) to unify phrase grounding and object detection. Building on GLIP, MQ-Det (Xu et al., 2024) integrates multimodal query encoder, while Ground- ing DINO (Liu et al., 2023a) employs a DETR-like (Carion et al., 2020) head for improved scalability. For MOD tasks, existing methods adapt pre-trained natural-domain VLOD models to the medical domain, such as MIU- VL (Qin et al.) with prompt engineering, Guo al. (Guo et al., 2023) with prompt fusion. However, they struggle with generalization across tasks and domains.
In response to these challenges, we propose framework called instance Decoupled Prompt Attention (iDPA) for incremental medical object detection. iDPA integrates Instance-level Prompt Generation (IPG) for generating fine- grained object knowledge and Decoupled Prompt Attention (DPA) to enhance prompt decoupling and precise knowledge injection during the multimodal fusion process, as shown at the bottom of Fig. 1. In the IPG module, instance features are decoupled from images, and using cross-attention to query the concept buried in the instance can separate and integrate knowledge across tasks. The DPA module decou- ples the originally coupled attention between prompts and tokens in previous methods, retaining three key components: vision-language mutual enhancement (V ++ T),, prompt-to- vision (P: - V), and prompt-to-texx (P. + T) knowledge injection. Additionally, instead of injecting knowledge only at the backbone level, we also innovatively apply knowledge injection during the multimodal fusion encoder.
Continual Learning. Continual Learning (CL) mitigates catastrophic forgetting when learning new tasks through three primary approaches. Regularization-based meth- ods (Li & Hoiem, 2017; Kirkpatrick et al., 2017: Aljundi et al., 2018; 8;; Ding et al., 2022; Lao et al., 2021b) constrain loss functions to retain prior knowledge while adapting to new data. Architecture-
Consequently, iDPA incorporates three key strategies. First, by decoupling instance features from background infor- mation and incorporating fine-grained visual details into the prompt vectors, iDPA enhances object localization and recognition precision compared to randomly initialized prompts. It also effectively mitigates category confusion by focusing on the target entities and reducing spurious correlations with the background. Second, the decoupled prompt attention approach, which separates prompt vectors from token representations, accelerates knowledge injection, making it more effective for localizing and recognizing med- ical concepts. This also mitigates catastrophic forgetting
methods (Douillard et al., 2022; Li et al., 2019; Yoon et al., 2017: Mallya & Lazeb- nik, 2018) assign dedicated parameters for each task to pre- vent interference. Rehearsal- based methods (Rolnick et al., 2019; Lopez-Paz & Ranzato, 2017; Shin et al., 2017: Re- buffi et al., 2017: Lao et al., 2021a) replay stored exemplars or generate pseudo-samples to mitigate forgetting. With the rise of large-scale pre-trained models, prompt-based con- tinual learning, an architecture-bar prominence. L2P (Wang et al., 2022c) first introduced a prompt pool for continual learning, with DualPrompt (Wang et al., 2022b) further partitioning knowledge into general
approach, has gained
2

---

iDPA: Instance Decoupled Prompt Attention for IMOD
and expert components. S-Prompt (Wang et al., 2022a) enables domain-adaptive prompt learning, while CODA- Prompt (Smith et al., 2023) improves prompt selection via attention mechanisms. DIKI (Tang et al., 2025) reduces task interference with residual tuning, and NoRGa (Le et al., 2024) models prompts as a Mixture-of-Expers with adap- tive weighting. Eclipse (Kim et al., 2024) enables efficient continual panoptic segmentation via visual prompt tuning, avoiding retraining and reducing forgetting. Recently, con- tinual learning in medical has gained increasing attention for its flexibility and adaptability to downstream tasks (Yi et al., 2023; Ye et al., 2024), offering more practical fit for clinical use than all-in-one models like Medical SAM (Zhu et al., 2024; Ma et al., 2024).
dation for improving generalization and robustness in data- scarce scenarios, making them highly suitable for practi- cal medical settings. Unlike traditional object detectors, VLOD models replace the classification head with a tex- tual encoder, such BERT (Devlin, 2018), and introduce monslity fusion encoder that enhances the model's ability to detect medical objects across different imaging modalities. VLOD models for object detection consist of four key components: 1) Visual Encoder L., 2) Textual Encoder 9, 3) Cross-Modality Fusion Encoder & Localization Head liee
f. and 4)
fo = (Img), ft = #x(xxt), fer fi = (e(fv fe),
(1)
(2)
Ple = Dco(f)) Pas = (1)).
(3)
Continual Object Detection. Continual Object Detection (COD) extends object detection to new categories while re- taining prior knowledge. ILOD (Shmelkov et al., 2017) first introduced COD, followed by CL-DETR (Liu et al., 2023b), which improved incremental detection with distillation and memory. ZiRa (Deng et al., 2024) was the first to adapt pre-trained VLOD models for COD, mitigating forgetting through regularization and reparameterization. However, most COD research focuses on natural domains, leaving its effectiveness in data-scarce medical applications uncertain, making this an open challenge.
The workflow is as follows: First, visual and textual features are extracted using P. and D (Eq. (1)). These features are then fused through the cross-modality fusion encoder $ (Eq. (2)). Finally, localization Piue and classification Pas predictions are generated (Eqq (3)), where denotes matrix multiplication. The fusion of these features enhances object localization and recognition accuracy.
3.3. Prompt-Based Continual Learning
Prompt- based CL. a type of CL. method based on prompt- ing pre-trained models, incrementally learns and stores lightweight, learnable parameter (known as a prompt) for each task, gradually building a "prompt pool" P
3. Preliminary
3.1. Task Definition
{PI,P2, ... }N where p; E IR Here, N represents the number of tasks, / is the prompt length, and d is the feature embedding dimension. At inference time, a selected prompt from the prompt pool is appended to the frozen pre-trained model to restore learned knowledge. Given the feature embeddings fc E Rexd for a transformer layer, the input is formed by concatenating the selected prompt p. E R with fe as follows:
Incremental Medical Object Detection (IMOD) involves in- crementally detecting and localizing medical objects (e.g., lesions, tumors, organs) in medical imaging data (e.g., CT, MRI, X-ray, PET scans) over time. The task requires sequen- tial learning from multiple tasks [Ti, T2. ......... ,Tl. where each task T; consists of a dataset D. = (x/ v/ J== with r representing images and y including bounding boxes and class labels. Each task also includes a class name set G = [c]] }= linking linking label indices to category names used by the text encoder of VLOD models. The main challenge in IMOD, particularly in class insseinnrremetal learning, is to adapt to new object classes introduced in each task without forgetting previously learned ones, allowing the model to handle an expanding range of medical objects while main- taining detection accuracy across all learned classes. This work is developed based on pre-trained VLOD models (such as GLIP (Li et al., 2022)). When training task t, the task's ' classes encompass the current task's classes along with the previous tasks' classes
Transformer(ps:: fel), wherelpss; R(++))dd (4)
where Pa is the selected prompt embedding. The prompt selection process relies on query-key matching, with fea- ture centroids IK [K learned during training via cosine similarity or clustering. For a test sample x. the most relevant centroid K, is identified by:
K, = arg max (Dv K.). R Kinik
(5)
Currently, prompt-based CL methods focus on global, mixed knowledge, which suffices for classification tasks. However, for IMOD, fine- knowledge is crucial for precise localization and understanding of medical objects. Therefore, adapting prompt- CL for IMOD requires focusing on learning and preserving specific, fine- ane-grained knowledge to address the unique challenges of IMOD.
3.2. Vision Language Object Detection
To better achieve IMOD, this work builds upon pre-trained VLOD models in natural domains, providing a strong foun-
3

---

iDPA: Instance Decoupled Prompt Attention for IMOD
Incremental Learning
IPG (Instance-level Prompt Generation)
DPA (Decoupled Prompt Attention)
Kvasir
Task
Task
Task
Prolection
Activation
KV
Attn,
Decoupling
nodule
nucleus
polyp
Attn,
KV
Initial prompt
IPG
IPG
IPG
Concept Prompt: Pu/Pt
Original Attention
Insert
P7
P+
DPA
DPA
Location
Localization
Head
loss
key-value pair
Query Function
Contrastive
Vision Encoder
loss
Prompt Pool
Fuslon Encoder
Text
Task
nodule
nucleus
polyp
Encoder
Task t-1
nodule
nucleus
Incremental
DPA
DPA
weights
product
Activation
Tunable
Incremental Learning Task
Figure 2: Overview of iDPA. Based on a frozen pre-trained VLOD model with visual-language interaction modules (e.g.. GLIP (Li et al., 2022)), iDPA integrates Instance-level Prompt Generation (IPG) and Decoupled Prompt Attention (DPA) to enhance object localization and recognition, optimizing knowledge transfer for medical detection tasks.
4. Methodology
4.2. Instance-Leve Prompt Generation
4.1. Overview of iDPA
Inspired by prior research (Wu et al., 2023; Xu et al., 2024), we first use pre-trained model to extract image features from the training set, then focus on target regions via bound- ing boxes. To refine these features, apply cross-attention to disentangle and clarify different concepts, demon- strated in (Alayrac et al., 2022; Xu et al., 2024).
To effectively achieve the IMOD goal, our core idea is to decouple fine-gained instance-level knowledge, generate enriched concept prompts, and optimize the Prompt Atten- tion (PA) mechanism by retaining key attention components, enabling efficient knowledge injection into the pre-trained model while mitigating class forgetting issues. This reliable, robust approach helps the model focus essential localiza- tion and recognition for clinical medical object detection. Thus, we propose iDPA, an efficient, scalable IMOD frame- work, as shown in Fig.
Decoupling Instance Features for Prompt Construction. For each task T. prior to model training, we decouple the instance-leved representations I, for each the |C;| cate- gories from the training data as follows:
2 ColPool(@(IIn)) Text), j= 1,2, ,,, (6)
I, = (ve v. = (0 1Cal
(7)
To realize this design, iDPA integrates Instance-level Prompt Generation (IPG, Sec. 4.2) and Decoupled Prompt Atten- tion (DPA, Sec. 4.3) to enhance robust, scalable incremental learning. First, IPG extracts fine-grained, adaptive instance- level features from the training set and generates rich, di- verse, stable instance-specific prompt knowledge. This contextual knowledge is then injected into the frozen, pre- trained model through DPA, enabling the model retain focus on reliable, critical, task-specific details while effec- tively mitigating interference from previous tasks. Through this streamlined process, iDPA facilitates efficient, precise and seamless fine- -raiind knowledge transfer, which is es- sential for accurate IMOD performance.
where each v. E R consists of M instance-level repre- sentations of each category, which encoded by the image encoder 4, the fusion encoder 4y. These representations are extracted before the attention layers, which correspond to the attention layers where prompt learning is applied. Specifically, given an instance from the e-th category with bbox g4 in an image, an Rol pooler (Ren, 2015) is used to extract the corresponding region feature v Rd. The scaling factor 7 = 1.3² increases the region size to capture additional contextual information. During training, for each class, we query the instance representations to decouple the concepts (note that | the prompt length

---

iDPA: Instance Decoupled Prompt Attention for IMOD
defined in Eq. (4)) contained within them, which are then used to form the prompt for the current task. In practice, we set M- = 1000 for full-data settings to ensure diverse concept coverage. For few-shot learning, we set M = m, where m is the number of available shots per class.
The following outlines the derivation from PA to DPA, using the visual-language interaction module X-Attn (Li et al., 2022; Liu et al., 2025) as an example. The input multi- modal tokens f. E LR (for vision) and St E RLexd (for text) are fed into X-Athn, where the mutual enhancement of - multimodal knowledge produces updated fu and ft:
Continual Concept Perception and Knowledge Integra- tion (CCPKI). Given the instance-level representations
(fr. X-Attm(ft,
I (ve Ivel = K) K) [ extracted from the training data of the current task for each category, the CCPKI mod- ule decouples the concepts from these instances using Query-Answer framework. Specifically, the generation of the i-th prompt ii can be expressed as follows:
= =(, (fo fe), fo + (ff {c)) = -ih+ ft, fe).
(9)
In Eq. (9). Attm fi) represents the vision-to-text knowledge transfer, denoted as ft. and vice versa for the text-to-visioe transfer fu- Inspired by (He et al., 2021),
( Vd
p softmax
(W.I.); P. = p + aaaao oTT p.).
Original Prompt Attention
(8)
Py
fv
To generate task-specific prompts from instance features, we adopt the following attention-based formulation. Here, Ps E R represents the initial prompt for the i-th task, consisting of / learnable concept components that serve as queries. The instance-level representations I, are projected into key and value vectors using learnable matrices Wi and W., respectively. A cross-altention mechanism is then applied to extract task-relevant conceptual knowledge from the instances. Since different tasks may involve different concepts, a learnable scaling factor t E R(x1 is used to dynamically modulate the concept weights, followed by a nonlinear activation function (-)) (e.g.. tanh) to filter and enhance the meaningful components. Finally, the activated concepts are scaled by a E IR 1xd and added to the initial prompt via a residual connection.
Pv
Pu
fp
Pt
Pt
ft
ft
Decoupling
DPA (Decoupled Prompt Attention)
Pt
Scale
Scale
Pr
To transfer knowledge effectively, it is common for previous tasks to assist in learning subsequent ones. Inspired by this, the CCPKI for the i-th task is initialized with the parameters from the previous CCPKI, i.e., ΦCCCKT + &CCPKI In this way, our approach retains the generated concept knowledge p. from the decoupled instances in the prompt pool after training, resulting in dynamically evolving prompt pool. This design enhances scalability and flexibility, making it particularly well-suited for IMOD tasks.
Figure 3: Comparison between Decoupled Prompt Attention (DPA) and Original Prompt Attention.
we analyze the role of prompt tuning in multimodal fusion through formal derivation. We derive an equivalent formula- tion to closely examine the prompt's 's role in Attn.-- (For simplicity, we focus on Attn,,-ts which can similarly be extended to Attm)-w),, providing an alternative perspective of multimodal prompt tuning:
[F6: fil = Attn (Po (Poifo] fol, (pe: fel)
4.3. Decoupled Prompt Attention
= [(1 - Apt))Attn, (f P.) + Attt pt);
Instead of training series of prepended prompt vectors for each task, we focus on modifying the attention mechanism to learn multimodal knowledge efficiently. Specifically, we decouple the PA mechanism into two components: the original attention and the attention with prompt knowledge injection. These components are then integrated through residual connection. This decoupling process is referred to as the DPA mechanism. Compared to PA, DPA accelerates instance-leved knowledge learning, reduces task interference, and lowers computational complexity, resulting in reduced memory usage during training.
(1 (1- -(-(()))ttm.
+ Atf)) fi)],
(10)
where Ple.1) E [Rd represent vision and text prompts, respectively. These prompts are prepended to the visual and textual features fo and ft before being fed into Attn,-
By examining Eq. (10), we introduce the DPA mechanism, which decouples the knowledge transfer processes P. - ft and f. + ft in PA. This is achieved through: 1) discarding the learning of [p.] similar to VPT-Deep (Jia et al., 2022), which can be replaced by new trainable prompts in subse- quent A-Attn layers to reduce computational complexity
5

---

iDPA: Instance Decoupled Prompt Attention for IMOD
Table I: Performance of various continual learning methods on the ODinM-13 benchmark under the full data setting, FAP
(%): Final Average AP. CAP (%); Cumulative Average AP. FFP (%): Final Forgetting Percentage of old tasks.
DPUC
BCCD
CPM-17
BreastC
TEX1IK
Kidneyr
ADNI
Meneng
Breast 7
Methods
FAPt CAP
FFP 4
Zero-shot
4.60
12.30
3.20
3.70
0.10
1.00
0.00
1.30
0.00
0.00
4.40
4.50
5.50
3.12
SAM2.1
0.06
1.71
0.00
8.83
18.00
0.15
0.00
0.67
0.23
0.00
12.54
0.08
0.22
3.27
MedSAM-2
4.15
5.34
4.53
13,83
22.44
1.54
0.00
1.56
4.32
0.04
18.56
2.96
3.45
5.02
FT (Oracle)
51.74
76.70
79.78
62.63
37.45
54.08
36.76
66.73
38.52
49.22
75.35
45.39
61.72
56,62
Joint (Upper)
46.82
72.19
78.70
61.74
37.18
53.34
35.35
65.72
34.03
47.97
73.90
43.89
59.90
54.67
Non-Promptthased
CL.
Sequential
0.00
0.00
0.00
0.00
3.18
0.00
0.00
0.00
0.00
32.84
0.00
0.00
21.18
4.40
15.87
57.81
WISE-FT
10.47
45.93
1.21
16.23
7.52
6.59
0.20
10.25
0.13
0.03
13.11
14.65
13.06
10.72
26.18
18.60
ER
34.86
60.04
54.07
55.72
15.49
40.90
17.33
48.23
13.45
43.93
65.49
22.15
47.14
39.91
48.73
19.25
ZiRa
0.14
0.13
0.00
0.71
0.10
0.42
0.00
0.00
0.00
26.85
0.00
0.15
19.02
3.66
16.37
49.67
Prompt-based CL
L2P
42.11
70.16
46.58
57.67
1.62
48.09
26.96
54.60
27.36
38.05
25.58
29.44
50.25
39.88
46.04 04
8.24
DualPrompt
44.26
40.47
17.73
34.03
0.59
37.03
5.91
31.29
23.12
42.30
15.58
33.57
49.63
28.89
42.24
20.57
S-Prompt
43.48
63.02
47.49
35.52
8.37
39.51
27.96
57.79
25.82
40.67
65.02
29.61
48.97
41.02
46.70
8.87
CODA
42.01
70.63
58.78
41.96
4.51
47.62
26.50
58.36
22.48
32.99
65.71
27.04
48.39
42.08
49.78
2.80
DKTI
46.91
75.91
54.14
55.12
0.74
47.57
32.63
62.03
29.21
43.29
16.02
34.35
54.77
42.51
49.12
7.74
NoRGa
44.62
75.87
58.44
57.73
1.07
51.76
29.71
63.48
29.53
44.54
37.56
34.11
$4.46
44.84
49,90
4.92
iDPA(OurS) 47.09
73.76
66.85
60.29
36.54
50.98
32.69
64.98
31.15
44.42
57.20
34.65
53.03
50.28
54.10
2.48
Notes: SAM2.1 denotes auto segmentaiinn using the SAM2.1-L model, while MedSAM-2 incorporates MedSAM-25 memory bank based on SAM2.11
and memory usage; and 2) re-normalizing the attention weights through the weight adjustment scalar X(fe), which aims remove the coupling between the visual features fo and the visual prompt p.:
The weight X(ri) before (Pe St) is replaced with a learnable scaling factor, providing greater flexibility in learn- ing prompt knowledge and enabling the model to capture more precise and richer downstream information. Addition- ally, the decoupling process also reduces the computational complexity and memory usage of attention, as shown in 5.4,
Σ, exp( (.Wolaaaa""" )
x(f()
Σ, exp( (eWalon Wo + E, exp( seWolfowor
)
(11)
5. Experiments
where (( E Rdddd are projection layers. Our proposed DPA mechanism is shown in Fig. 3, and the process of knowledge transfer and fusion for the enhanced text features fe (note that p. is omitted) is then updated to as follows:
5.1. Experimental Setup
Benchmark. To ensure a comprehensive evaluation, we col-
lected 13 MOD tasks (Jha et al., 2020; Boccardi et al., 2015; Cassidy et al., 2021; Liu et al., 2020; Gong et al., 2021; Setio et al., 2017; Vu et al., 2019) from publicly available datasets for IMOD, named ODinM-13. This benchmark evaluates model performance in real medical scenarios, covering 8 imaging modalities across 9 organs. Each task is assessed in both full and few-shot settings (k- = 1, 10, 50), ensuring each class has at least A: objects. To further evaluate generalizabil- ity, we supplement ODinM-13 with a cammain-nccremenall benchmark constructed from four polyp datasets (Ji et al., 2022; Jha et al., 2020; Bernal et al., 2015; Ngoc Lan et al., 2021) across different medical centers. For more details, please see the appendix.
Xft)) f:= Attn (fo, ft) + 1- -(F) Attn App,,
(12)
= AAttn (fo. AAttm
(13)
where À E Ridd is learnable parameter, initialized to O, ensuring that the additional information in Pa -> ft does not affect the original branch fo fo before train- ing on downstream datasets. For X(fa), we argue that in object detection tasks, fo is typically much larger than Pa (e.g.. fo may exceed 10,000 tokens, while Pe is typically set to 10), causing the prompt information overshadowed by the input features, reducing learning efficiency. This imbalance in the original PA hinders the model's effective use of the prompts, impeding knowledge transfer and task performance. Compared to PA, DPA reduces the interfer- ence between prompt information and the pretrained model by lowering their coupling. Specifically, the normalized attention weight 1 - X(fi) before Attm;-o(ffe. fo) is re- moved, preserving the full pre-trained model information.
Evaluation Metric. To evaluate the model's ' continual learn- ing and forgetting mitigation, we use Final Average AP (FAP) for final performance, Cumulative Average AP (CAP) for overall performance, and Final Forgetting Percentage (FFP) to measure resistance to forgetting old tasks. The pre- diction performance for task after learning task is denoted as AP, 1,3. where Average Precision (AP) is the standard
6

---

iDPA: Instance Decoupled Prompt Attention for IMOD
Table 2: Overall performance of various continual learning methods on the ODinM-13 benchmark under few-shot data settings.
Full data
50
60
40
40
I-shot
10-shor
50-shox
Method
F
CAPT
FFP L
FAP 1
CAP1
FFP
FAP 1
CAP
FFP 4
Joint (Upper)
20.03
36.68
46.07
10
5
10
Sequential
1.24
11.49
23.43
1.66
13.67
36.51
3,38
14,95
46.86
Task
Task
WiSE-FT
6.16
14.62
9.25
9.47
21.03
12.98
10.24
23.34
16.03
10 shat
1 Isht
ZiRa
6.98
13.59
11.68
16.19
15.12
3.49
15.78
37.28
40
30"
L2P
3.25
7.18
3.14
5.16
9.63
4.48
27.91
35,20
5.66
20-
20-
DualPrompt
7.36
13.30
7,93
5.95
13.99
11.34
20.44
34.94
16.91
10-
S-Prompe
3.34
8.90
5.66
8.39
12.98
4.36
22.76
30.86
7.92
0
CODA
6.89
13.96
4.57
4.41
14,20
12.60
31.75
40.86
4.37
10
10
Task ID
Task ID
DIKI
5.67
13.09
3.88
6.46
15.34
9.78
34.06
42.27
5.85
Sequential WISE
Dualfrompt
CODA
Ga
NoRGa
6.02
11:09
5.17
5.36
11.90
8.14
32,09
39.12
5.70
L2P
SPromot
DRA (Ours]
Figure 4: Performance variation of different CL methods in full-data and few-shot settings.
IDPA (Ours)
12.19
18.03
3.58
23.78
29.68
4.75
38.65
45.03
3,93
Table 3: Comparison on four polyp datasets under the con-
on Object365 (Shao et al., 2019), GoldG (Liu et al., 2023a), and Cap4M (Liu et al., 2023a), as a robust starting point. All experiments employ Adam W (Loshchilov, 2017) with a multistep learning rate scheduler. The learning rate is set to 0.1, and weight decay is set to 1e-4. The experiments run on
tinual domain setting.
Methods
Sun
Kvasir
BKAI
ClinicDB
FAP
C
FFP
L2P
59.22
70.01
73.16
69.24
67.91
68.69
0.35
DualPrompe
62.64
71.76
75.43
72.63
70.62
69.77
1.53
iDPA (ours)
66.10
74.33
78.77
77.93
70.92
GPUs with a batch size of 1 per GPU for epochs, with a learning rate decay of 0.1 at epoch 3. All results aver- aged over 3 random seeds, with the task order determined by the seed. All comparison methods are re- re-implemented based on their official implementations. For more details, please refer to the appendix.
COCO metric (Lin et al., 2014). The average AP across all tasks after learning task i is given by AP; Σ AP; 1.i- After completing all N tasks, the final AP for each task is denoted AP ___ and the final average performance is com- puted as FAP- = AP,, which serves as the primary metric for CL performance. To assess historical performance, we calculate CAP- = ト/4 Σ AP;, and the Final Forgetting Percentage as FFP- tending the Forgetting Percentage Point (FPP) introduced by CL-DETR (Liu et al., 2023b).
5.2. Main Results
Full Data Setting. Tab. presents the final performance for cach task on ODinM-13 under full data training, along with the FAP, CAP, and FFP scores to evaluate the model's final performance, overall performance, and resistance to forgetting. The "Zero-shot" results represent the starting point, derived by leveraging the original GLIP weights for each task. The "FT" results indicate the model's oracle performance, which is achieved by training on single task and testing on the corresponding task. The "Joint" results represent the model trained on the datasets of all tasks, serving as the upper bound in continual learning.
N E. (AP,, AP,,s). ex-
Comparison Methods. We compare our iDPA with both non-prompt-based and prompt-based CL methods. For non- non-prompt-based CL, we select Sequential, ER (Rol- nick et al., 2019), WiSE-FT (Wortsman et al., 2022), and ZiRa (Deng al., 2024). For prompt-based CL. we include L2P (Wang et al., 2022c), DualPrompt (Wang et al., 2022b), CODA (Smith et al., 2023), S-Prompt (Wang et al., 2022a), DIKI (Tang et al., 2025). and NoRGa (Le et al., 2024), which follow a similar task-specific parameter training ap- proach to our iDPA. Note that L2P, DualPrompt, CODA, and NoRGa are originally designed for vision tasks; we ex- tend them to multimodal tasks for fair comparison. For the all-in-one style foundation model, we compared Zero-shot GLIP, SAM2.1 (Ravi et al., 2024), and MedSAM-2 (Zhu et al., 2024). More details on reproduction can be found in the appendix.
As indicated by the bold values, iDPA achieves the best final performance on 9 out of 13 tasks compared to other methods. It outperforms the previous prompt- based SOTA method, NoRGa (Le et al., 2024), by 5.44% in FAP, 4.20% in CAP, and reduces FFP by 2.44%. It also outperforms the previous non-prompt-based SOTA method, ER (Rolnick et al., 2019), which requires extra data (10-shot per class) for rehearsal and full model tuning, by 10.37% in FAP, 5.37% in CAP, and reduces FFP by 16.77%. Furthermore, iDPA uses only 1 1.4% of the trainable parameters and does not require ad- ditional data for rehearsal. Compared to ZiRa (Deng et al.,
Implementation Details. We use the GLIP (Liu et al., 2023a) model with Swin-T (Liu et al., 2021), pre-trained
7

---

iDPA: Instance Decoupled Prompt Attention for IMOD
Table 4: Ablation study of key components in iDPA.
Table 5: Comparison of iDPA at different positions.
Method
FAP
CAP
FFP
#Params
#Memory.
4v 4
4
FAP
CAP1 1
FFP 4
#Params +
#Memory
#Time
Naive
44.99
49.86
4.31
1.24M
7200M
x
47.44
51.78
2.91
4.36M
7622M
7h40m
IPG (w/ T)
48.65
52.69
3.71
3.21M
7285M
/
38.44
44.89
7.53
6.89M
7177M
7h36m
DPA
47.10
51.04
3.73
1.37M
6496M
/
V
x
48.41
53.56
4.38
11.26M
8295M
9h10m
IPG (w/o T)+ DPA
50.17
53.89
2.54
3.22M
6590M
X
50.28
54.10
2.48
3.22M
6590M
6h28m
IPG (w/ T) DPA
50.28
54.10
2.48
3.22M
6590M
/
/
51.56
56.25
3.17
14.48M
8945M
9h32m
Notes: FAP (%). Final Average AP. CAP (%): Cumulative Average AP. FFP (56): Final Forgetting Percentage of old tasks. #Params: The number trainable parameters used during training. #Memory: Memory usuge during truining with 1024 1024 input and batch size #Time: Training time on ODinM- 13sing GPU, butch size for 5 epochs. w/ T: Indicates the presence of weight transfer between fasks. w/o T:: Indicates the absence of weight transfer hetween tacks.
50-
40-
(36)
48-
40-
4.-
48
48
35-
FAP
30-
20-
44-
10-
Images
45-
Dim
(a) Different context knowledge
(b) Different scale types
(c) Different learning rate
(d) Prompt length
Figure 5: Impact of context knowledge, scale types, learning rate, and prompt length on model performance.
Table 6: Comparison of iDPA with varying layers in 4r.
Table 7: Impact of scaling factor in Eq. 7 on performance.
Layers
FAP
FAP
FAP
#Params &
#Memory |
?
FAP
APt
FFP |
45.53
75.07
46.78
0.54M
6054M
1.00
50.07
54.01
2.57
2
48.16
77.72
50.89
1.07M
6162M
1.30
50.28
54.10
2.48
49.94
79.12
53.34
2.15M
6371M
1.50
49.99
53.40
2.77
6 (all)
50.28
79.47
54.29
3.22M
6590M
Table 8: Comparison between the Naïve baseline and iDPA on (D,, + +.).
2024), which is designed for incremental VLOD learning, iDPA surpasses by 46.62% in FAP, 37.73% in CAP, and reduces FFP by 47.1%% This is due to the substantial gap between the medical and natural domains, along with the large differences across modalities, organs, and categories, which makes it difficult for ZiRa to regularize and recon- figure parameters to learn shared space. Additionally, compared to the upper bound, iDPA is only 4.39% lower in FAP. These results demonstrate the effectiveness of using a prompt- based CL method for the comprehensive MOD task. Our method maintains excellent performance while ensuring data independence, parameter efficiency, model resistance to forgetting, scalability, and flexibility.
( + +.))
FAP 1
CAP
FFP 1
Naive
41.72
47.30
8.57
iDPA (ours)
48.41
53.56
4.38
A
6.69
6.26
4.19
transfer capability of our approach, which, by decoupling instance-level knowledge and leveraging DPA, greatly im- proves model performance and efficiency, particularly in data-scarce environments.
Domain Continual Setting. iDPA achieves the best per- formance across all four polyp datasets under the continual domain setting, outperforming L2P and DualPrompt in FAP, CAP, and FFP. These results demonstrate the superior gener- alizability and stability of iDPA in handling domain shifts.
Few-Shot Data Setting, To simulate the challenging real- world scenario of limited data annotation in clinical settings, we also conduct experiments in the few-shot setting on ODinM-13, as shown in Tab. 2. In the 1-shot setting, our iDPA outperforms the best alternative by 4.38% in FAP, and 4.07% in CAP, with only 0.44% increase in FFP. In the 10- shot setting, iDPA outperforms the best result by 12.88% in FAP, 8.65% in CAP, and exhibits a minimal 0.39% increase in FFP. In the 50-shot setting. iDPA outperforms the best result by 4.59% in FAP, 2.76% in CAP, and reduces FFP by 0.44%. These results highlight the strong knowledge
Visualization. Fig. shows the AP variation of different CL methods on ODinM-13, 13. Our method outperforms existing ones throughout the incremental learning process, not just at the end. More qualitative results are provided in Fig. 7 in the appendix showing that iDPA produces more accurate bound- ing boxes with higher confidence for various MOD tasks
8

---

iDPA: Instance Decoupled Prompt Attention for IMOD
than Zero-shot and the L2P method (Wang al,, 2022c), using enhanced knowledge transfer.
full data and few-shot settings (Fig. 5d). A prompt length of 10 offers balanced performance and is chosen the default.
Impact of X-Attn Layer Count. We conduct experiments with different numbers of X- Attn layers (1. 2, 4, and 6) and find that incorporating all layers achieves the best perfor- mance, as shown in Tab. 6.
5.3. Ablation Study
We conduct ablation studies of the proposed modules in iDPA on ODinM-13 under the full data setting, shown in Tab. 4. Compared to the Naive prompt method, adding the IPG module improves FAP by 3.66%, CAP by 2.83%, and reduces FFP by 0.60%. Adding the DPA module increases FAP by 1,1,, CAP by 1.18%, and reduces FFP by 0.58%. Moreover, DPA reduces both gradient backpropagation com- putation and memory usage, while introducing only a mini- mal number of additional parameters. When both modules are combined, FAP increases by 5.29%, CAP by 4.24%, and FFP decreases by 1.83%. These results demonstrate that decoupling instance knowledge from images effectively en- hances object recognition and localization. By decoupling PA, DPA enables more efficient learning and better injection of prompt knowledge. Furthermore, since the two modules are orthogonal, combining them improves the model's abil- ity to complete the IMOD task. Additionally, we investigate knowledge transfer across medical tasks. When IPG is not used for weight transfer, performance slightly decreases. However, despite the substantial differences between medi- cal tasks, knowledge sharing still occurs. This especially evident when reducing the number of training epochs, where weight transfer significantly boosts learning efficiency. For further details, please refer to the appendix.
Impact of scaling factor y in Eq. 7. As shown in Tab. 7, the model achieves the best performance when the scaling factor Y is set to 1.30, yielding the highest FAP and AP, and the lowest FFP. This suggests that moderately enlarging the Rol region helps capture more useful contextual information.
Fairness of Naïve Baseline Comparison. As shown in Tab. 8, iDPA outperforms the Naïve baseline when prompts are injected into both 4, and o, achieving gains of 6.69 FAP and 6.26 CAP, and reducing FFP by 4.19, demonstrat- ing the effectiveness of our design.
6. Conclusion
This study presents iDPA for IMOD learning without catas- trophic forgetting. iDPA efficiently generates and injects instance-level knowledge, reducing computational complex- ity and memory usage. It decouples target instance features and employs a continual concept perception mechanism to create and integrate concept prompts into the multimodal fusion process. Additionally, iDPA refines prompt atten- tion into three key interaction steps, focusing on continual learning for efficient knowledge injection while preserving the original knowledge. For the evaluation, we introduce a new IMOD benchmark, ODinM-13, with 13 MOD datasets. Experiments show that iDPA outperforms previous SOTA methods in both full-data and few-shot settings. Our analy- sis also demonstrates that our method can be more efficient and memory-friende compared to previous CL methods.
5.4. Empirical Analysis
Impact of Knowledge Injection Position. As shown in Tab. 5, we compare different positions for prompt knowl- edge injection in VLOD models. The Fusion Encoder achieves the best balance between performance and cost.
Impact of Context Knowledge. In Fig. 5a, we compare four types of context knowledge: Gaussian noise, image knowledge, instance-level knowledge from different lay- ers (denoted as "Instances**), and instance-level knowledge from the corresponding layer (denoted as 'Instances"). )) Our experiments demonstrate that context-aware knowledge en- hances the IMOD task, with the best performance achieved by instance knowledge from the corresponding layer.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
A Acknowledgments
Impact of Scale in DPA. We test four types of scale X in DPA: constant (1.0), gate mechanism, task- (A E
The authors would like thank the GLIP team and the developers of other continual learning methods for making their code and models publicly available, which greatly facilitated this work. We also acknowledge the contributions of individuals and organizations who have shared open- source medical imaging datasets. We are grateful to the anonymous reviewers and program chairs for their valuable and constructive feedback.
RIXI), and dim-level (A E IR.xd). As shown in Fig. 5b, dim-level À yields the best performance.
Impact of Learning Rate. A grid search over the range [le-5, 0.1] reveals that a learning rate of 0.01 provides the best performance. Results for iDPA at different learning rates are shown in Fig. 5c.
Impact of Prompt Length. We compare prompt lengths in

---

iDPA: Instance Decoupled Prompt Attention for IMOD
References
expansion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9285- 9295, 2022.
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I,, Hasson, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M.. et al. Flamingo: a visual language model for few- shot learning. Advances in neural information processing systems, 15:23716-23733 2022.
Gong, H., Chen, G., Wang, R., Xie, X., Mao, M., Yu, Y., Chen, F., and Li, G. Multi-task learning for thyroid nodule segmentation with thyroid region prior. In 2021 IEFE 18th International Symposium on Biomedical Imaging (ISBI), pp. 257-261. IEEE, 2021.
Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., and Tuytelaars, T. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), pp. 139-154, 2018.
Guo, M., Yi, H., Qin, Z., Wang, H., Men, A., and Lao, Q. Multiple prompt fusion for zero-shot lesion detection us- ing vision-language models. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 283-292. Springer, 2023.
Bernal, J., Sánchez, F. J., Srnnandez-Esparraane G., Gil, D., Rodríguez, C., and Vilariño, F. Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Computerized medical imaging and graphics, 43:99-111 111, 2015.
He, J., Zhou, C., Ma, X., Bong-Kirkpatrick, T., and Neubig, G. Towards unified view of parameter-efficient transfer learning, arXiv preprint arXiv:2110.04366, 2021.
Boccardi, M., Bocchetta, M., Morency, F. C., Collins, D. L., Nishikawa, M., Ganzola, R., Grothe, M. J., Wolf, D., Redolfi, A., Pievani, M.. et al. Training labels for hip- pocampal segmentation based on the eadc-adni harmo- nized hippocampal protocol. Alzheimer's ' & Dementia, 11(2):175-13,, 2015.
Jha, D., Smedsrud, P. H., Riegler, M. A., Halvorsen, P., Lange, T. d., Johansen, D.,, and Johansen, H. D. Kvasir- seg: A segmented polyp dataset. In International Con- ference Multimedia Modeling, pp. 451 Springer, 2020.
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. End-to-end object detection with transformers. European conference on computer vision, pp. 213-229. Springer, 2020.
Ji, G.-P., Xiao, G., ChOu, Y-C.,, Fan, D.-P., Zhao, K., Chen, G., and Van Gool, L. Video polyp segmentation: deep learning perspective. Machine Intelligence Research, 19 (6):531-549, 2022.
Cassidy, B., Reeves, N. D., Pappachan, J. M., Gillespie, D., O'Shea, C., Rajbhandari, S., Maiya, A. G., Frank, E.,
Jia, M., Tang, L.,, Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
Boulton, A. J., Armstrong, D. G., et al. The dfuc 2020 Analysis towards diabetic foot ulcer detection. touchREVIEWS in Endocrinology, 17(1):5, 2021.
dataset:
Kim, B., Yu, and Hwang, S. Eclipse: Efficient continual learning in panoptic segmentation with visual prompt tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3346- 3356, 2024.
Cheng, T. Song, L.,, Ge, Y., Liu, W., Wang, X., and Shan, Y. Yolo-world: Real-time open-vocabuse poca-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 16901-16911, 2024.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska- Barwinska, A., al. Overcoming catastrophic Grabska-Barwinska.
Deng, J., Zhang, H., Ding, K., Hu, J., Zhang, X., and Wang, Y. Zero-shot generalizable incremental learn- ing for vision- vinion-language object detection. arXiv preprint arXiv:2403.01682 2024.
forgetting in neural networks. Proceedings of the national academy of sciences, 1
2017.
Devlin, J. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint arXiv:1810.04800 2018
Lao, Q., Jiang, X., Havaci, M., and Bengio, Y. A two- stream continual learning system with variational domain- agnostic feature replay. IEEE Transactions on Neural Net- works and Learning Systems, (3(9):4466-4477) 202
Ding, Y., Liu, L., Tian, C., Yang, and Ding, H. Don't stop learning: Towards continual learning for the clip model. arXiv preprint ar arXiv:2207.09244 2022.
Lao, Q. Mortazavi, M., Tahaci, M., Dutil, F., Fevens, T. and Havaei, M. Foel: Feature-oriented continual learning for generative models. Pattern Recognition, 120:108127, 2021b.
Douillard, A., Ramé, A., Couairon, G., and Cord, M. Dytox: Transformers for continual learning with dynamic token
10

---

iDPA: Instance Decoupled Prompt Attention for IMOD
Le, M., Nguyen, A., Nguyen, H., Nguyen, T., Pham, T., Van Ngo, L., and Ho, N. Mixture of experts meets prompt-based continual learning. arXiv preprint arXiv:2405.14124 2024.
Liu, Z., Lin, Y., Cao. Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Li, J., Selvaraju, R., Gotmare, A., Joty, S., Xiong, C., and Hoi, S. C. H. Align before fuse: Vision and language representation learning with momentum distillation. Ad- vances in neural information processing systems, 34: 9694-9705, 2021.
Lopez-Paz, D. and Ranzato, M. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
Li, I.. H., Zhang, P., Zhang. H., Yang, J., Li, C.,, Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J.-N., et al. Grounded language-image pre-training, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10965-10975, 2022.
Ma, J., He, Y., Li, F., Han, L., You, C., and Wang, B. Seg- ment anything in medical images. Nature Communica- tions, 15(1):654, 2024.
Mallya, A. and Lazebnik, S. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 7765-7773, 2018.
Li, X., Zhou, Y., Wu, T., Socher, R., and Xiong, C. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In International con- ference on machine learning, pp. 3925-3934. PMLR, 2019.
Ngoe Lan, P., An, N. S., Hang, D. V., Long, D. V.. Trung, T. Q.. Thuy, N. T., and Sang, D. V. Neounet: Towards accurate colon polyp segmentation and neoplasm detec- tion. In Advances in visual computing: 16th international symposium, ISVC 2021, virtual event, October 4-6, 2021, proceedings, part II, pp. 15-28. Springer, 2021.
Li, Z. and Hoiem, D. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40( 40(12):2935-2947, 2017.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D.,, Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Qin, Z., Yi, H., Lao, Q. and Li, K. Medical image un- derstanding with pretrained vision language models: comprehensive study. In The Eleventh International Con- ference on Learning Representations.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-888 8768. PMLR, 2021.
Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Li, C., Yang, J., Su, H., Zhu, J., et al. Grounding dino: Marry- ing dino with grounded pre-training for open-set object detection, arXiv preprint arXiv:2303.05499 2023a.
Ravi, N., Gabeur, V., Hu, Y.-T., Hu, R., Ryali, C., Ma, T.,
Liu, S., Zeng. Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q. Li, C., Yang, J., Su, H., et al. Grounding dino: Marry- ing dino with grounded pre-training for open-set object detection, In European Conference on Computer Vision, pp. 55.55..00 Springer, 2025.
Khedr, H., Rädle, R., Rolland, C., Gustafson, L.,, al.
Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.007 2024.
Rebuffi, S.-A., Kolesnikov, A., Sperl, G., and Lampert, C. H. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Liu, Y., WI, Y.-H., Ban, Y., Wang. H., and Cheng, M.-M. Rethinking computer-aided tuberculosis diagnosis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2646-2655, 2020.
Ren, S. Faster r-cnn: Towards real-time object detec- tion with region proposal networks. arXiv preprint arXiv: Ttw::550.04440 2015.
L Liu, Y., Schiele B., Vedaldi, A., and Rupprecht, C. Contin- ual detection transformer for incremental object detection. In Proceedings of the IEFE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 23799-23803 2023b.
Rolnick, D.,, Ahuja, A., Schwarz, J., Lillicrap, T., and Wayne, G. Experience replay for continual learning. Advances in neural information processing systems, 32, 2019.

---

iDPA: Instance Decoupled Prompt Attention for IMOD
Setio, A. A. A., Traverso, A., De Bel, T., Berens, M. S., Van Den Bogaard, C., Cerello, P., Chen, H., Dou, Q- Fantacci, M. E., Geurts, B., et al. Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the lunal6 challenge. Medical image analysis, 42:1-13, 2017.
IEEE/CVF conference on computer vision and pattern recognition, pp- 139-149, 2022c.
Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Korbllith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pp. 7959-7971, 2022.
Shao, S., Li, Z., Zhang, T. Peng, C., Yu, G., Zhang, X., Li, J., and Sun, J. Objects365: A large-scale, high- quality dataset for object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8430-8439, 2019.
Wu, S., Zhang, W. Jin, S., Liu, W., and Loy, C. C. Aligning bag of regions for open-vocabulary object detection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 15254 15264, 2023,
Shin, H., Lee, J. K., Kim, J., and Kim, J. Continual learning with deep generative replay. Advances in neural informa- tion processing systems, 30, 2017.
Xu, Y., Zhang, M., Fu, C., Chen, P., Yang, X., Li, K., and Xu, C. Multi-modal queried object detection in the wild. Advances in Neural Information Processing Systems, 36, 2024.
Shmelkov, K., Schmid, C., and Alahari, K. Incremental learning of object detectors without catastrophic forget- ting. In Proceedings of the IEFE international conference computer vision, pp. 3400-3409, 2017.
Ye, Y., Xie, Y., Zhang, J., Chen, Z., Wu, Q.. and Xia, Y. Continual self-supervised learning: Towards universal multi-modal medical data representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11114-11124, 2024.
Smith, J. S., Karlinsky, L., Gutta, V.,
P., Kim, D., Arbelle, A., Panda, R., Feris, R., and Kira, Z. Coda- doda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In Pro- ceedings of the IEEE/CVF Conference Computer Vi- sion and Pattern Recognition, pp. 11909-11919, 2023.
Yi, H., Qin, Z., Lao, Q., Xu, W., Jiang, Z.. Wang, D., Zhang, S., and Li, K. Towards general purpose medical ai: Con- tinual learning medical foundation model. arXiv preprint w0X::2200306680, 2023.
Tang, L., Tian, Z., Li, K., He, C., Zhou, H., Zhao, H., Li, X., and Jia, Mind the interference: Retaining pre-trate pre-trained knowledge in parameter efficient continual learning of vision- lision-language models. In European Conference Computer Vision, pp. 346-365. Springer, 2025.
Yoon, J., Yang. E., Lee, J., and Hwang, S. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv: 708.0154777 2017.
Zhu, J., Hamdi, A., Qi, Y., Jin, Y., and Wu, Medical sam Segment medical images video via segment anything model 2. arXiv preprint arXiv:2408 00874, 2024.
Vu. Q. D., Graham, S., Kure, T., To, M. N. N., Shaban, M., Qaiser, T., Koohbanani, N. A., Khurram, S. A., Kalpathy- Cramer, J., Zhao, T., et al. Methods for segmentation and classification of digital microscopy tissue images. Frontiers in bioengineering and biotechnology, pp. 53, 2019.
Wang, Y., Huang, Z., and Hong, X. S-prompts learning with pre-trained transformers: An occam's razor for domain incremental learning. Advances in Neural Information Processing Systems, 35:5682-5695, 2022a.
Wang, Z., Zhang, Z., Ebrahimi, S., Sun, R., Zhang, H., Lee, C.-Y,, Ren, X., Su, G., Perot, V., Dy, J., al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision, pp. 631-648. Springer, 2022b.
Wang, Z., Zhang, Z., Lee, C.-Y., Zhang, H.,, Sun, R., Ren, X., Su, G., Perot, V., Dy, J., and Pfister, T. Learning to prompt for continual learning. In Proceedings of the
12

---

iDPA: Instance Decoupled Prompt Attention for IMOD
A. Theoretical Analysis
This section presents a theoretical analysis demonstrating the superior efficiency of DPA over traditional prompt learning.
A.1. Overall Analysis
f1 Concati(1 Att:)Atttm--<(e.... + + Att
(1 ((m (Pf))Atto
(14)
Concat[A, where Po,t E Biid represent the vision and text prompts, and fv: fa are the visual and textual features before being fed into At
f2 -tm + - fi) -B
(15)
If Attn E R Rixdd then the two terms A(pi) and A(fi) each belong to Rixd and fi E R Lx2dd In contrast,
fa E BRLxd
f2 Attn + ((S)
(16)
where we have defined A Alln,- ft) (So,
0f2
a Attm, X(fe) 04
(17)
00
00
This indicates that /z has lower-dimensional structure, residual components, and a more direct gradient flow.
A.2. Computation Cost
Lemma A.1. f2 is computational light than f1
Proof. The overall fi is Si = Concat(A; BJ. Howeveer, f2 uses only the B branch.f2 = B. The computation cost of branch is roughly O = O(f..p)) + (......) = O(A B) for O2 = O(B). O, > O
A.3. convergence benefit analysis
Lemma A.2. Let fi and f2 be our models. Assume both models have converged to any local minima and there be an optimal representation f* = B. Suppose further output of f1 is locally linear around the optimum, then fo achieves the same performance as S1 at the local convergence.
Proof. The loss function L(for), For f, the output y1 hhConnatt[A. [A; Bl). and 8/2 h(B). At convergence: VC(fi) = and VL(f2) = There exists an optimal representation /* such that it is sufficient to have f B.
Assume is locally linear at the optima, then there exists matrix M such that
(Concat(A: BI) = =M ( B
MI MrB.
(18)
Under convergence, M,A = thus
h (Concat(A:B)) = M2B = RB(B).
(19)
81 h (Conncat (A: [A: hh' =k(B) y2-
=1)=-'
(20)
Thus fa performs as well as fi after convergence to local minima.
13

---

iDPA: Instance Decoupled Prompt Attention for IMOD
B. More Implementation Details
Benchmark. We collected 13 public datasets from the internet: DFUC-2020 (DFUC) (Cassidy et al., 2021), Kvasir (Jha et al., 2020), OpticNerv (OpticN) ', BCCD 2 CPM-17 (Vu al., 2019), Breast Cancer (BreastC) 3, TBXI1K (Liu et al., 2020), Kidney Tumor (KidneyT) Lunal6 (Setio et al., 2017), ADNI (Boccardi et al., 2015), Meningioma (Meneng) 5, Breast Tumor (BreastT)* 6, and TN3K (Gong et al., 2021). Among them, OpticN, BCCD, BreastC, KidneyT, Meneng, and BreastT are from the Roboflow website. These datasets include eight different modalities: Photography, Endoscopy. Light Microscopy, Histopathology. X-ray, CT, MRI, and Ultrasound, covering nine different organs: Foot, Colorectal, Nerve, Blood/Cell, Lung, Brain, Breast, Kidney, and Thyroid. The random seed used for few-shot data generation is kept consistent with the one used during training. k-shot means ensuring that each class in the current dataset contains at least & instances. Three different orders were used during training. The dataset order and corresponding random seed are shown in the Tab. 9.
Table 9: Task order under different random seeds. The table shows the dataset sequences used during training for three different random seeds (0, 5, and 10).
Order
2
3
4
5
6
7
8
10
11
12
13
DFUC
Kvasir
OptioN
BCCD
CPM-17
BreastC
TBX11K
KidneyT
Luma16
ADNI
Meneng
BreastT
TN3K
Seed 0
OpticN
BCCD
BreastT
Meneng
TN3K
Kvasir
TBXIIK
KidneyT T
DFUC
Luna16
BreastC
CPM-17
ADNI
Seed
CPM-17
BreastC
Lunal6
Kvasir
OpticN
Meneng
TN3K
BCCD
BreastT
KidneyT
TBXIIK
DFUC
ADNI
Sced 10
Implementation. The proposed method is implemented in Python using the PyTorch library and runs on a PC. The code is based on the official GLIP (Li et al., 2022) implementation 8, and its environment requirements remain unchanged. For full-data training, we use four NVIDIA 3090 GPUs with a batch size of 4, while for few-shot training, we use a single NVIDIA 3090 GPU with a batch size of 1. Unless otherwise specified, all experiments
trained for epochs, with the learning rate reduced by factor of 0.1 after 3 epochs. For all prompt-based CL methods (Wang et al., 2022c;b; Smith et al., 2023; Wang et al., 2022a; Tang et al., 2025; Le et al., 2024), the initial learning rate is set to le-2, whereas ZiRa (Deng et al., 2024) uses an initial learning rate of le-3. Standard fine-tuning (FT), training, sequential training, WiSE-FT (Wortsman et al., 2022), and experience replay (ER) (Rolnick et al., 2019) use an initial learning rate of 1e-4. The learning rates are determined via grid search within the range of [le-5, 0. To ensure reproducibility, all experiments are conducted with three different random seeds (0, 5, 10), and the dataset order is adjusted accordingly. The final results are reported as the average over three runs.
We reproduce other prompt-based based methods on GLIP by prompting all layers of both the vision and text encoders, whereas the original papers typically use only the embedding layer or few initial layers (e.g., the first five layers). This discrepancy may lead to suboptimal performance on the IMOD task. The vision backbone is used as the query function, and the mean fcature representation from its last layer is utilized to identify the task ID. For L2P (Wang et al., 2022c), we set the prompt length to 5. During inference, the top-5 prompts are selected from the prompt pool, following the official implementation. In the original L2P paper, updated prompts are selected via key- fey-matching mechanism during training, with diversity maintained using a frequency-based weighting technique. However, in the official code repository, specific prompts masked for different tasks. We follow the implementation provided in the official code. For DualPrompt (Wang et al., 2022b). we set the prompt length to 10 for both key and value prompts. Two layers are designated as G(eneral)- Prompt, while the remaining 10 layers serve as Mxxpert)-Pompl., For CODA (Smith et al., 2023), we set the prompt length to 8 and
Reproduction.
in-nniik
"httpe://roboflew.ccmm http:////gtttaaccccoo//oooooo///
14

---

iDPA: Instance Decoupled Prompt Attention for IMOD
additionally add an extra key learn the task identity, following the official implementation. For S-Prompt (Wang et al., 2022a), DIKI (Tang al., 2025), and NoRGa (Le al., 2024), set the prompt length 10. S-Prompt employs K-Means to generate 5 prototypes for task identification. All reproductions adhere to the implementation in the official code.
C. Additional Results
Tab. compares the impact of weight transfer between tasks when training for epochs. The results show that enabling weight transfer improves continual learning performance under limited training time. Tab. 11 presents detailed comparison of iDPA with different knowledge injection positions. The best performance is observed when knowledge is injected simultaneously into the vision, text, and fusion encoders. However, this setting leads to higher forgetting rate compared to injecting knowledge only in the fusion encoder. Fig. 6 visualizes the performance dynamics across different knowledge injection positions. Tab. 12, Tab. 13, and Tab. 14 report the performance of iDPA compared with other continual learning methods under 1-shot, 1 10-shot, and 50-shot settings, respectively. Tab. 15 shows that iDPA achieves the lowest parameter count (3.34M), reduced FLOPs, and significantly lower memory consumption and training time compared to all baselines. While maintaining competitive inference speed (5.93 FPS), offers the best overall efficiency among the evaluated continual learning methods. Tab. 16 demonstrates that iDPA consistently improves performance across 13 datasets under 1-shot, 5-shot, and 10-shot settings, with low variance indicating strong stability and generalization. Fig. 7 provides qualitative comparisons between iDPA, Ground Truth, Zero-shot, and I.2P (Wang et al., 2022c), the pioneering prompt-based continual learning method. iDPA shows superior localization accuracy, more precise classification, and higher confidence scores.
Table 10: Performance of using weight transfer over 3 epochs. EAP (%): Final Average AP. CAP (%): Cumulative Average AP.
Kassa
OpticN
BCCD
CPM-17
BreasTC
TBXIIK
KidneyT
Lunu16
ADNI
Meneng
BreastT
TN3K
Transfer
FAP
CAP
FFP J
W/O
46.86
73.69
63.61
60.25
29.88
50.68
30.59
57.88
30.58
42.80
50.84
19.23
51.28
46.78
51.87
3.98
W
46.34
71.05
66.86
60.33
30.65
49.16
29.39
63.78
28.16
39.99
56.97
27.10
52.36
47.86
52.06
2.47
Table 11: Comparison of performance with different knowledge injection positions. FAP (%): Final Average AP. CAP (%)
Cumulative Average AP. FFP (%); Final Forgetting Percentage of old tasks.
DFUC
Kvara
OptioN
BCCD
CPM-17
BreastC
TBXI
KidneyT
Luna16
ADNI
Meneng
Breast
4 4
11
FAP
CAP
V X x
42.28
72.53
66.30
58.94
27.08
49.73
30.25
62.38
27.51
38.23
56.00
32.98
52.46
47.44
51.78
2.91
X /
40.92
70.24
29.61
47.24
26.08
42.09
20.45
55.47
23.58
22.92
56.93
26.13
38.10
38.44
44.89
7.53
V
46.57
73.05
59.51
59.82
34.62
47.35
32.15
62.24
29.69
41.98
58.83
29.51
54.01
48.41
53:56
4.38
X X
47.09
73.76
66.85
60.29
36.54
50.98
32.69
64.98
31.15
44.42
57.20
34.65
53.03
50.28
54.10
2.48
/
49.40
76.04
66.40
60.91
33.15
53.12
36.43
65.73
32.19
45.94
58.15
35.67
57.19
51.56
56.25
3.17
15

---

iDPA: Instance Decoupled Prompt Attention for IMOD
60
Ov
Ov
Oe
Or
55
Øv Oe Or
AP (%)
50
45
40
2
4
10
12
Task ID
Figure 6: Performance variation of iDPA in different location.
Table Performance of various continual learning methods on the ODinM-13 benchmark under the 1-shot setting. FAP
(%): Final Average AP. CAP (%): Cumulative Average AP. FFP (%): Final Forgetting Percentage of old tasks.
PyRs K
BCCD
CPM-17
BreastC
TBXI1K
KidneyTi
Lumat
ADNI
Heneng
Breash
IN3K
Methods
HAP
CAP
HHP
Joint (Upper)
7.99
51.35
24.88
33.54
36.28
9.18
1.94
22.43
4.20
1.54
46.18
12.35
8.48
20.03
Non-Prompt-basee CL
Sequential
0.00 0.00
0.00
0.00
8.70
0.00
0.00
0.00
0.06
0.00
1.87
0.00
5.54
1.24
11.49
23.43
WiSE-FT
4.63
26.75
0.32
9.04
9.75
0.32
0.08
6.33
0.03
0.00
11.37
7.89
3.52
6.16
14.62
9.25
ZiRa
6.17
26.87
2.40
3.62
12.34
7.66
0.56
4.57
0.00
0.01
19.42
4.85
2.23
6.98
13.59
11.68
Prompt-based CL
L2P
1.56
2.16
0.00
10.46
12.45
1.14
0.21
3.69
0.25
0.00
7.91
0.43
1.99
3.25
7.18
3.14
DualPrompt
6,20
32.05
1.01
9.64
11.00
0.48
0.00
3.61
0.23
0.10
19.63
6.20
5.49
7.36
13.30
7.93
S-Prompt
2,24
7.94
3.55
8.72
2.35
0.42
5.69
1.00
1.53
3.34
8.90
5.66
CODA
0.76
2.69
7.19
24.02
20.42
0.34
0.92
0.69
1.03
0.13
14.95
5.73
10.68
6.89
13.96
4.57
DIKI
0.54
11.07
3.44
16.95
16.22
2.41
0.62
0.01
6.89
3.54
3.30
5.67
13.09
NoRGa
1.00
5.73
13.86
17.62
11.24
0.64
0.56
7.31
0.09
0.01
8.50
8.13
3.57
6.02
11.09
5.17
iDPA(Ours)
6.66 43.04
14.62
20.55
31.13
5.33
2.15
7.38
0.30
0.39
17.34
6.17
3.45
12.19
18.03
3.58
16

---

iDPA: Instance Decoupled Prompt Attention for IMOD
Table 13: Performance of various continual learning methods the ODinM-13 benchmark under the 10-shot setting. FAP
(%): Final Average AP. CAP (%); Cumulative Average AP. FFP (%): Final Forgetting Percentage of old tasks.
Opui
BCCD
CPM-17
BreastC
TBX11K
KidneyT
Lana16
ADNI
Meneng
BreastT
IN3K
Methods
FAP CAP FFP
Joint (Upper)
32.60
60.99
56.07
56.14
41.01
33.29
10.49
44.17
13.38
10.55
67.57
20.77
29.76
36.68
Non-Prompt-based
Sequential
3.84
0.00
0.00
0.00
0.36
0.00
0.00
0.00
0.00
8.19
0.00
1.92
7.33
1.66
13.67
36.51
WiSE-FT
7.13
41.13
1.45
12.68
16.91
1.91
0.13
8.98
0.06
0.04
16.31
10.68
5.69
9.47
21.03
12.98
ZiRa
6.07
30.71
1.32
11.14
16.38
9.18
0.53
7.00
0.44
0.57
40.70
8.74
8.96
10.90
16.19
15.12
Prompt-based
CL
L2P
2.92
16.65
17.90
11.04
4.08
0.09
4.34
0.02
2.72
5.16
4.48
DualPrompt
1.95
6.19
0.06
14.48
7.17
0.91
0.06
0.76
0.06
0.06
36.14
6.63
2.91
5.95
13.99
11.34
S S-Prompt
8.20
16.33
9.13
9.12
10.23
5.53
0.43
8.01
0.25
0.13
37.67
1.13
2.95
8.39
12.98
4.36
CODA
5.23
0.23
1.33
25.57
3.34
0.28
0.22
2.10
0.01
0.01
4.49
6.78
4.41
14.20
12.60
DIKI
0.97
19.66
0.25
27.92
5.83
1,27
1.51
9.31
0.08
0.01
6.11
7.20
3.86
6.46
15.34
9.78
NoRGa
2.60
11.19
1.42
0.18
0.69
0.01
0.01
14.49
1.80
5.36
8.14
iDPA(Ours)
21.37
50.20
29.20
39.11
38.33
19.65
6.03
27.06
6.23
3.15
39.42
15.34
14.02
23.78
29.68
4.75
Table 14: Performance of various continual learning methods on the ODinM-13 benchmark under the 50-shot setting. FAP
(%): Final Average AP. CAP (%): Cumulative Average AP. FFP (%): Final Forgetting Percentage of old tasks.
DPUC
M 11
OpticN
BCCD
CPM-17
BreasiC
TEXIIK IK
Lunal6
ADNI
Meneng
BreastT
TN3K
Methods
FAP
CAP
FFP
Joint (Upper)
39.56
66.52
64.41
59.26
41.78
43.82
26.48
52.90
22.61
66.40
72.93
25.61
47.92
48.48
Non-Prompt-baseee
CL
Sequential
0.00
0.00
0.00
0.00
5.51
0.00
0.00
0.00
0.00
22.19
0.00
0.00
16.20
3.38
14.95
46.86
WiSE-FT
8.18
43.77
1.29
13.05
14.99
2.93
0.11
10.59
0.08
0.05
21.55
10.08
6.44
10.24
23.34
16.03
ZiRa
2.07
5.58
0.39
5.45
4.89
3.56
0.23
0.78
0.45
3.03
4.85
5,51
8.58
3.49
15.78
37.28
Prompt-based CL
1.2P
29.13
54.05
34.74
38.00
35.61
38.23
6.41
34.59
13.33
2.74
46.20
10.18
19.67
27.91
35.20
5.66
DualPrompt
24.84
27.49
25.44
47.18
27.36
29.17
3.33
21.87
13.09
6.57
16.32
9.28
13.82
20.44
34.94
16.91
S-Prompt
23.79
39.45
33.67
16.90
8.74
29.77
6.00
28.24
8.38
4.78
61.73
13.08
21.40
22.76
30.86
7.92
CODA
27.42
65.27
31.74
40.66
25.85
36.81
17.28
43.72
16.37
54.25
31.75
40.86
4.37
DIKI
34,26
68.82
34.05
53.26
33.02
38.06
13.93
50.27
20.74
5.42
48.16
8.17
34.70
34.06
42.27
5.85
NoRGa
30.69
58.17
39.23
53.81
28.12
38.82
15.31
46.16
15.21
6.76
44.04
18.96
21.87
32.09
39.12
5.70
IDPA(Ours)
34.79
59.03
52.64
58.12
39.33
37.35
14.78
52.77
22.70
56.32
10.99
39.10
38.65
45.03
3.93
17

---

iDPA: Instance Decoupled Prompt Attention for IMOD
Table 15: Efficiency comparison of different continual learning methods. Metrics include the number of parameters, floating point operations (FIOPs), memory consumption, total training time, and inference speed. Our method achieves the best trade-off with the lowest parameter count and competitive inference performance.
Methods
#Params.
#FLOPs.
#Memory4
#Timel
Inference Speedt
Joint (Upper)
231.76M
488.03
GMac
13129M
9h55min
6.18 FPS
Sequential
231.76M
488.03
GMac
13129M
9h55min
6.18 FPS
WiSE-FT
231.76M
488.03
GMac
13129M
9h55min
6.18 FPS
ER
231.76M
488.03
GMac
13129M
11h15min
6.18 FPS
ZiRa
10.23M
490.15 GMac
8377M
6h25min
6.11 FPS
L2P
6.97M
601.50 GMac
10288M
7h50min
5.08 FPS
DualPrompt
4.83M
583.82 GMac
9417M
7h36min
5.25 FPS
S S-Prompt
2.73M
590.89 GMac
5366M
8h24min
5.13 FPS
CODA-Prompe
10.97M
583.82 GMac
9803M
9h03min
5.26 FPS
DIKI
8.76M
583.82 GMac
9754M
7h49min
5.16 FPS
NoRGa
8.76M
583.82 GMac
9963M
8h07min
5.17 FPS
iDuA(Ours)
3.34M
506.00/501. ,00 GMac
6590M
5h46min
5.93 FPS
Table 16: Mean performance across 13 datasets under 1-shot, 5-shot, and 10-shot settings. a denotes the average improvement of our iDPA method over the baseline. Variance results are included to illustrate the stability across multiple runs.
DFUC
Kvasir
OpticN
BCCD
CPM-17
BreastC
TBX11K
KidneyT
Lunal6 16
ADNI
Meneng
Breast
TN3K
Methods
6.66
43.04
14.62
20.55
31.13
5.33
2.15
7.38
0.30
0.39
17.34
6.17
3.45
0
2.55
0.95
4.89
3.86
1.71
0.93
0.00
1.55
0.14
0.03
2.18
2.46
1.31
5
21.37
50.20
29.20
39.11
38.33
19.65
6.03
27.06
6.23
3.15
39.42
15.34
14.02
1.62
0.86
3.70
3.24
2.63
1.24
0.01
0.02
1.68
1.26
1.34
1.58
3.80
10
34.79
59.03
52.64
58.12
39.33
37.35
14.78
52.77
22.70
24.55
56.32
10.99
39.10
2.92
1.89
3.75
4.16
3.76
2.34
0.00
0.25
0.40
0.05
2.47
0.51
4.07
18

---

iDPA: Instance Decoupled Prompt Attention for IMOD
DFUC
Kvasir
BCCD
Luna16
Menenga
BreastT
GT
Zero-shot
L2P
iDPA (Our)
Text Prompt: "diabetic ulcer. polyp. optic nerve. platelets. blood white blood cell. nuclei. breast tumor. tuberculosis. kidney tumor. lung nodule. hippocampus. menengioma. tumor. thyroid nodule.
Figure 7: Visualization results of iDPA compared with L2P and Zero-shot at the end of training with random seed on ODinM-13.
19


Reviews:
[{"review_id": "4v9xqubwoK", "reviewer": "Reviewer_rTU6", "summary": "This paper proposes iDPA (Instance Decoupled Prompt Attention), a novel framework for Incremental Medical Object Detection (IMOD). The primary motivation is that existing prompt-based continual learning methods, while effective for classification tasks, struggle with object detection due to the need for fine-grained instance-level reasoning.\n\nTo this end, the authors introduce:\n- Instance-level Prompt Generation (IPG): A mechanism to decouple fine-grained instance knowledge from images and generate prompts that better focus on dense medical object detection.\n- Decoupled Prompt Attention (DPA): A modification of standard prompt attention mechanisms, separating prompt-token interactions to enhance knowledge transfer, reduce memory overhead, and mitigate catastrophic forgetting.\n\nThe authors construct ODinM-13, a benchmark of 13 cross-modal, multi-organ, multi-category medical datasets, and demonstrate that iDPA outperforms state-of-the-art (SOTA) methods in full data and few-shot settings (1-shot, 10-shot, 50-shot).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed and constructive feedback! We treasure the opportunity to address your concerns and improve our work.\n\n# 1. Mathematical Justification for DPA superiority\nWe appreciate the reviewer’s feedback. DPA enhances prompt learning by separating the attention mechanism into distinct prompt-token interactions, minimizing interference from token embeddings. Its key advantage is the re-normalization of attention weights via the scaling factor $\\lambda(f_t)$, as derived in Eq. (11-13). This balances the influence of prompts and pretrained tokens, boosting prompt effectiveness when token embeddings dominate due to length. We agree that adding formal theoretical analysis or empirical complexity comparisons would strengthen this and plan to include them in future revisions.\n\n## Overall Analysis\n\n$$\nf_1 = \\text{Concat}[ (1-\\lambda(p_t)) \\text{Attn}_{v \\rightarrow t}(f_v, p_t) + \\lambda(p_t) \\text{Attn}_{v \\rightarrow t}(p_v, p_t); \n(1-\\lambda(f_t)) \\text{Attn}_{v \\rightarrow t}(f_v, f_t) + \\lambda(f_t) \\text{Attn}_{v \\rightarrow t}(p_v, f_t)]\n= \\text{Concat}[A, B]\n$$\n\nwhere $p_{\\{v, t\\}} \\in \\mathbb{R}^{l \\times d}$ represent the vision and text prompts, and $f_v, f_t$ are the visual and textual features before being fed into $\\text{Attn}_{v \\rightarrow t}$.\n\n$$\nf_2 = (1- \\lambda(f_t)) \\text{Attn}_{v \\rightarrow t}(f_v ,  f_t) + \\lambda(f_t) \\text{Attn}_{v \\rightarrow t}( p_v,  f_t) = B.\n$$\n\nIf $\\text{Attn}_{v\\to t}(\\cdot,\\cdot) \\in \\mathbb{R}^{L \\times d}$, then $A(p_t)$ and $A(f_t)$ belong to $\\mathbb{R}^{L \\times d}$ and $f_1 \\in \\mathbb{R}^{(L+l) \\times d}$, while $f_2 \\in \\mathbb{R}^{L \\times d}$.\n\n$$\nf_2 = \\text{Attn}_{v\\to t}(f_v,f_t) + \\lambda(f_t) \\Delta,\n$$\nwhere $\\Delta = \\text{Attn}_{v\\to t}(p_v,f_t) - \\text{Attn}_{v\\to t}(f_v,f_t)$.\n\n$$\n\\frac{\\partial f_2}{\\partial \\theta} = \\frac{\\partial \\text{Attn}_{v\\to t}(f_v,f_t)}{\\partial \\theta} + \\lambda(f_t) \\frac{\\partial \\Delta}{\\partial \\theta}.\n$$\n\nThis indicates that $f_2$ has a lower-dimensional structure, residual components, and a more direct gradient flow.\n\n## Computation Cost\n\n### Lemma 1: Computation Cost\n\n**Lemma:**  \n$f_2$ is computationally lighter than $f_1$.\n\n**Proof:**  \nThe overall $f_1$ is $f_1 = \\text{Concat}[A;B]$. However, $f_2$ uses only the $B$ branch, $f_2 = B$. The computation cost of branch $A$ is roughly $O_A = O(f_v, p_t) + O(p_v, p_t) =O(A + B)$, while for $B$, $O_{f_2} =O(B)$. Thus, $O_{f_1} >O_{f_2}$.\n\n## Convergence Benefit Analysis\n\n### Lemma 2: Convergence Behavior\n\n**Lemma:**  \nLet $f_1$ and $f_2$ be models that have converged to local minima, with an optimal representation $f^* = B$. If $f_1$'s output is locally linear around the optimum, then $f_2$ achieves the same performance as $f_1$ at convergence.\n\n**Proof:**  \nConsider the loss function $\\mathcal{L}(f_{\\text{out}})$, where $f_1$ outputs $y_1 = h(\\text{Concat}[A;B])$ and $f_2$ outputs $y_2 = h(B)$. At convergence, $\\nabla \\mathcal{L}(f_1) = \\mathbf{0}$ and $\\nabla \\mathcal{L}(f_2) = \\mathbf{0}$, with $f^* = B$ as the optimal representation.  \n\nSince $h$ is locally linear at the optima, there exists a matrix $M$ such that:  \n$$\nh(\\text{Concat}[A;B]) = M \\begin{pmatrix} A \\\\ B \\end{pmatrix} = M_1A + M_2B.\n$$  \nAt convergence, $M_1A = \\mathbf{0}$, so:  \n$$\nh(\\text{Concat}[A;B]) = M_2B = h(B).\n$$  \nThus:  \n$$\ny_1 = h(\\text{Concat}[A;B]) = h(B) = y_2.\n$$  \nHence, $f_2$ performs as well as $f_1$ at the local minima.\n\n# 2. Lacking evidence in generality beyond ODinM-13\nWe thank the reviewer for the suggestion. ODinM-13 already includes tasks across multiple modalities and organs, providing a degree of generalization. To further support generalizability, we conducted additional experiments on polyp datasets from different centers. Results are summarized below:\n| Methods     | Sun   | Kvasir | BKAI  | ClinicDB | FAP ↑ | CAP ↑ | FFP ↓ |\n|-------------|-------|--------|-------|----------|-------|-------|-------|\n| L2P         | 59.22 | 70.01  | 73.16 | 69.24    | 67.91 | 68.69 | 0.35  |\n| DualPrompt  | 62.64 | 71.76  | 75.43 | 72.63    | 70.62 | 69.77 | 1.53  |\n| iDPA (ours)       | 66.10 | 74.33  | 78.77 | 77.93    | 74.28 | 70.92 | -0.03 |\n\n# 3. Failure modes and limitations\nIn our experiments, we observed that the IPG module improves learning in low-resource and hard cases. The DPA module further enhances the IPG's performance and helps reduce forgetting. For example, in the CPM-17 dataset (with only 30 training samples), the naive method achieves an FAP of 1.09. Adding only the IPG module improves performance to 15.64, adding only the DPA module achieves 1.80, and combining both modules increases FAP to 36.54.\n\n## Other notes:\nWe will address the missing definitions in Equation (8). Specifically, $W_k$ and $W_v$ represent the linear projection matrices for keys and values in the attention mechanism. These will be clearly defined in the revised manuscript.", "claims_and_evidence": "**Lacking evidence in generality beyond ODinM-13**: The model is only evaluated on ODinM-13; additional benchmark testing (e.g., public MOD datasets) would strengthen the claim of generalizability.\n\n**Mathematical justification for DPA superiority**: While the scaling factor λ(ft) is well-motivated, additional formal proofs explaining why DPA improves prompt learning over standard Prompt Attention (PA) would be beneficial.", "methods_and_evaluation_criteria": "Strengths:\n- ODinM-13 provides a **diverse, realistic benchmark** for incremental medical object detection.\n- **Comprehensive baselines**: The study includes both prompt-based and non-prompt-based continual learning methods, ensuring a fair comparison.\n\nWeaknesses:\n\n- **Failure modes and limitations**: The paper does not analyze failure cases (e.g., impact of class imbalance, annotation errors in ODinM-13, or domain shifts).", "theoretical_claims": "No theorectical claim found, which may not be that suitable for ICML.", "experimental_designs_or_analyses": "Lack of proof for DPA’s efficiency: a computational complexity comparison between DPA and standard prompt attention is expected.", "relation_to_broader_scientific_literature": "This work uses prompt engineering to improve general VLOD models in terms of the medical domain. Its key contribution is related to the decoupling of prompt attention.", "essential_references_not_discussed": "Not familiar with the latest works.", "other_strengths_and_weaknesses": "1. Wk and Wv in Eq. 8 are not defined.\n2. Lack of theoretical proof for DPA’s efficiency: a formal justification of why DPA outperforms standard prompt attention would improve the rigor of the work.", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NLtAUE6EKY", "reviewer": "Reviewer_Sm8X", "summary": "This paper proposes a novel incremental medical object detection framework called iDPA, which is composed of an instance-level prompt generation (IPG) and a decoupled prompt attention (DPA) module. Comparing existing methods, the instance-level prompt generation provides learnable prompts with fine-grained task-specific knowledge, and the DPA module helps simplify the prompt attention mechanism and mitigates the forgetting issue during the task transfer. The proposed iDPA methods show a consistent improvement in incremental learning settings of 13 different tasks, outperforming existing SoTA in both full-data and few-shot settings.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question 1: Significance of Incremental Learning in Full-Data Settings**\n\nIncremental learning methods like iDPA offer significant advantages in medical settings where models must be deployed incrementally due to regulatory, ethical, or resource constraints. Retraining large models on full datasets is often too costly, but iDPA only updates prompt vectors (1.4% of trainable parameters), which reduces training time. While Medical SAM performs well in zero-shot segmentation, it struggles with long-tail classes and domain shifts. On the other hand, iDPA enables incremental class learning and preserves prior knowledge, effectively handling challenges such as adding new classes over time. It also supports task-specific adaptation, such as joint detection and classification, which SAM is not designed for.\n\n**Question 2: Stability of Instance-Level Prompt Generation in Few-Shot Settings**\n\nThe IPG module ensures stability in few-shot settings by aggregating features from all instances within a class using cross-attention, reducing reliance on noisy instances. It combines local features with global ones, preventing overfitting to sparse annotations. Empirical results show that iDPA outperforms other methods in 1-shot settings and provides a 3.66% FAP improvement compared to naive prompts. IPG also enhances generalization by using a scaling factor (γ) to expand regions of interest (RoIs), capturing contextual information beyond tight bounding boxes. By leveraging pretrained GLIP model knowledge, IPG prevents overfitting even with few samples. To further improve stability, semi-supervised initialization, hard negative mining, and hyperparameter tuning are suggested.\n\n**Question 3: Model Complexity and Writing Clarity**\n\nThank you for the reviewer’s valuable feedback. We appreciate the concerns regarding model complexity and writing clarity. iDPA’s modular design was intentionally chosen for flexibility, with IPG handling instance-level knowledge extraction and DPA optimizing attention. To manage complexity, we have designed hyperparameters like K and λ to be learned adaptively and are exploring simplifications such as hyperparameter-free designs. We also recognize the importance of clear and consistent notation and will standardize symbols and provide clearer explanations for equations, ensuring the content remains both accessible and rigorous.\n\n**Question 4: Technical Details of DPA Module**\n1. **Table 5 Interpretation**:  \n   - The \"X-Attn Number\" represents the number of cross-attention layers in the fusion encoder where DPA is applied. For example, \"6 (all)\" indicates that DPA is enabled in all 6 layers.\n\n2. **Figure 2 Redesign**:  \n   - It should be clarified that DPA is applied to all three encoders (visual, text, and fusion), with emphasis on its role in the fusion stage. The visual and text encoders are optional, as shown in the ablation experiments in Table 4. When all three are used together, continual learning performs best. However, incorporating DPA into the fusion encoder provides a good balance of performance, additional parameter load, memory usage, and training time. Therefore, in subsequent experiments, we default to using the fusion encoder.\n\n3. **Computational Efficiency**:  \n   - **Memory Access**: DPA requires three parallel attention computations (vision→text, text→vision, and original PA). However, this is mitigated by:  \n     - **Reduced Feature Length**: The prompt length (l=10) is much shorter than image/text tokens (L=10,000+), which minimizes memory overhead.  \n     - **Computation Merge**: During testing, we merge the three parallel attention computations, reducing the overall computational cost compared to the original PA. This is further demonstrated in Reviewer rTU6 under the section \"Mathematical Justification for DPA Superiority in Computation Cost.\"", "claims_and_evidence": "Yes, most of the paper's claims are properly supported by the experimental results. The ablation experiments on page 8 help illustrate each module's contribution and provide an empirical reason for the model design. The experiment also validates the DPA module's memory and speed improvement.", "methods_and_evaluation_criteria": "Yes, the proposed IPG and DPA methods are intuitively reasonable and were validated empirically through experiment. Using instance-level prompts can naturally provide fine-grained task-specific information while the DPA module is also proven to be efficient and effective. Based on the evaluation in Tables 1 and 2, it is clear that the proposed method outperforms existing SoTA with a non-trivial gap, and it is also meaningful for the future development of the domain.\n\n## Weakness:\n\n1. The reviewer's major concern here is the significance of the incremental learning setting, especially for the full data evaluation. Since the full data is available, will the performance of the continue learning-based method still be better than the regular task-specific model? Additionally, how about the performance of all-in-one style medical SAM [a] foundation models? But this will not harm the novelty and soundness of the paper.\n2. Another specific concern for the iDPA is the design of instance-level prompt generation. According to section 4.2, the instance-level prompt is generated via the instance-level image feature, which needs training data from the bbox to extract these instance-level features. This is fine for the full data setting, but the reviewer worries that this will be a problem in the few-shot settings, where the training data may not be able to provide a set of high-quality instances. There is no evaluation on the stability of the proposed model on few-shot settings as well.\n3. The last minor concern is the residual connection in each component. Most of the proposed module is connected to the main framework via a residual connection, which can potentially weaken the contribution. Yet, the ablation experimental results show the effectiveness of each component, so it is just a minor issue.\n\n[a] Zhu, Jiayuan, et al. \"Medical sam 2: Segment medical images as video via segment anything model 2.\" arXiv preprint arXiv:2408.00874 (2024).", "theoretical_claims": "N/A, the paper didn't propose a new theoretical claim. The proposed method is evaluated through experiments empirically.", "experimental_designs_or_analyses": "Yes, the evaluation in the paper is generally convincing and thorough.\n1. The improvement of the proposed method is non-trivial compared with the SoTA baseline.\n2. The paper has also provided detailed implementation details of the evaluation and evaluated the proposed method under different settings.\n3. The ablation experiment also helps understand the effectiveness of each method.\n\nYet, the reviewer does notice a small issue in the abstract. The performance improvement reported in the paper is \"5.44%, 4.83%, 12.88%, and 4.59%\" for each setting, but the results in the abstract on the openreview is \"5.44%, 4.83%, 15.39%, and 4.59%\", where the third value is different. But I assume this is just a typo.", "relation_to_broader_scientific_literature": "The proposed method mainly focuses on incremental learning for medical object detection settings. It is developed based on the existing framework of GLIP and swin-transformer. The design of IPG and DPA is also a modification of the existing learnable prompt pool for each task and the prompt attention mechanism in the previous works.\n\nIn this paper, the proposed method mainly focuses on optimizing the prompt generation and attention mechanism, providing a more specific and fine-grained prompt for each task. Also, improves efficiency via using decoupled prompt attention.", "essential_references_not_discussed": "The reviewer didn't find such a missing reference in the paper.", "other_strengths_and_weaknesses": "The reviewer does find two additional weaknesses for the proposed method.\n\n1. The overall complexity of the method. The proposed iDPA is composed of two different modules, each with a set of complex designs and hyper-parameters that can be tuned in application. The ablation experiment helps discuss the effectiveness of each module. The complexity is still concerning.\n2. A major issue, as for the reviewer, is the writing of the paper. There are multiple equations and notations in the paper, but they are not all properly defined. Some of variable even uses repeated notations and make it confusing to follow the paper. Additionally, Figure 2 is also kind of messy. This really hinders the readers from understanding the paper. A few examples are listed below:\n    - Both the centriod of the features and the number of instant-level representations use the letter K as their notation.\n    - The part of section 4.2 between line 200 to 219 is particularly unclear. \n    - The weight of each linear projection layer seems to be represented using W in the equation but never defined. \n    - As for the CCPKI module, it only discussed the creation of prompt $p_i$, where $i$ is the i-th prompt. However, there is no discussion about the difference between $p_v$ and $p_t$, as shown in Figure 2. It is unclear if $p_v$ and $p_t$ are generated using the same mechanism or not. \n    - The activation function $\\sigma$ in equation (8) is not defined, though the reviewer guessed it might be the sigmoid activation. \n    - The notation in equation (9) is also confusing. The double vertical line usually serves for the norm between two vectors, but it seems to be a bracket in equation (9), which is very confusing. \n    - Equation (10-13) uses different colors for some of the components in the equation; it is actually not that clear what the meaning of these colors is.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. The reviewer is not sure what Table 5 is evaluating. According to the method section and Figure 2, the DPA module is only inserted once in each encoder. What is the meaning of the different X-Attn number here?\n2. According to Figure 2, the DPA module is applied for all three encoders, but why is the row with only the fusion encoder colored in red in Table 4?\n3. In the DPA module, is the attention computed separately for each pair of inputs? Will this cause additional memory access during computation? The regular PA seems to need just one large matrix multiplication, while DPA needs three.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "paXuxkHoev", "reviewer": "Reviewer_HWoG", "summary": "This paper aims to tackle the challenge of incremental medical object detection, which adapts to emerging medical concepts and retains prior knowledge. The authors claim that existing works are only designed for classification and fail to capture fine-grained information for detection tasks, which are mainly limited to the (a) coupling between foreground-background information and (b) coupled attention between prompts and image-text tokens. To tackle these challenges, this paper proposes an iDPA framework consisting of an instance-level prompt generation to enhance dense predictions. Besides, It introduces a decoupled prompt attention to enhance the knowledge transfer of prompts. Experiments demonstrate that iDPA achieves superior performance in both full-data and few-shot settings while being efficient regarding trainable parameters and memory usage.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question 1: Conceptual Gap Between Medical and Natural Domains**\n\nWe appreciate the reviewers' feedback and would like to clarify the differences between the medical and natural domains, as well as the limitations of existing methods in medical object detection. Previous studies, such as those by Qin et al. (2022), Ma et al. (2024), and Zhu et al. (2024), highlight key challenges in the medical domain, including data scarcity, modal diversity (e.g., CT, MRI, X-rays), and annotation complexity, as medical annotations require expert input. Existing methods face issues like foreground-background coupling, where background areas may confuse classifiers, and prompt-label attention coupling, which can dilute prompt information and reduce sensitivity to subtle features, such as small lesions. The iDPA framework overcomes these challenges through instance-level prompt generation and decoupling attention, significantly improving medical detection. While designed for medical applications, our method shows strong generalization capabilities, suggesting it can also perform effectively in natural domains due to the greater complexity of the medical domain. We hope this clarifies the necessity and relevance of our method in the medical field, and we appreciate the reviewer’s suggestions, which have helped refine the paper.\n\n**Question 2: Value of Incremental Medical Object Detection**\n\nIncremental medical object detection is crucial for addressing real-world challenges in clinical deployment, offering significant advantages over zero-shot models like Medical SAM. It enables adaptation to new tasks without full retraining, thus overcoming regulatory constraints (e.g., HIPAA, GDPR), and supports the dynamic evolution of medical knowledge, allowing models to adjust to new diseases and imaging technologies without losing prior learning. Incremental learning effectively handles the diversity of medical image modalities (CT, MRI) and annotation inconsistencies, ensuring generalization across different types while reducing the computational and storage demands of retraining large models. While models like Medical SAM excel in zero-shot scenarios, they struggle with rare, out-of-distribution concepts and multitask compatibility. Incremental learning methods like iDPA are better suited for these challenges, particularly in detection tasks. Our method complements models like MedSAM by generating bounding boxes as prompts for precise predictions and enabling continuous learning in out-of-distribution domains. In summary, incremental medical object detection addresses regulatory, data, modality, and learning challenges in dynamic medical settings.\n- Qin, Ziyuan, et al. \"Medical image understanding with pretrained vision language models: A comprehensive study.\" arXiv preprint arXiv:2209.15517 (2022).\n- Ma, Jun, et al. \"Segment anything in medical images.\" Nature Communications 15.1 (2024): 654.\n- Zhu, Jiayuan, et al. \"Medical sam 2: Segment medical images as video via segment anything model 2.\" arXiv preprint arXiv:2408.00874 (2024).\n\n**Question 3: Comparison with Dense Prediction Continual Learning Works**\n\nThe paper focuses on incremental object detection (bounding box regression and classification) in medical images, with an emphasis on dense prediction approaches like DenseBox, which predict relative object positions. This differs from tasks like panoptic or open-vocabulary segmentation. iDPA contrasts with methods such as Eclipse (CVPR 2024) and PanopticCLIP (TPAMI 2024): Eclipse targets panoptic segmentation with dense pixel masks, while iDPA detects discrete objects through bounding boxes. Both methods use prompts, but iDPA’s instance-level prompt generation isolates fine-grained features, which could inspire dense prediction methods, though direct comparison is challenging due to task differences. PanopticCLIP focuses on zero-shot open-vocabulary segmentation, while iDPA handles incremental class learning. Technically, Eclipse uses spatial-semantic cues for segmentation, whereas iDPA decouples instance-level knowledge from background clutter. Architecturally, Eclipse modifies segmentation heads like Mask2Former, while iDPA extends VLOD models like GLIP. Future work includes using iDPA’s decoupled prompt attention to enhance dense prediction and combining it with panoptic prompts for joint detection-segmentation continual learning. We also plan to evaluate iDPA on dense prediction datasets (e.g., Retina, ISIC) to assess its generalizability. \n\n**Question 4: Others**\n1. We will add an efficiency comparison of the method in Table 1 in the revised manuscript, where the high efficiency of our method can already be observed in the current table.\n\n2. During training, the ground truth (gt) boxes are used to generate instance features, and during testing, only the prompts saved in the prompt pool are used.\n\n3. The sentence in Section 4.1 is in blue color to indicate emphasis.", "claims_and_evidence": "No.\n- The mentioned conceptual gap between medical and natural domains is unclear. There is no experimental or theoretical to justify this gap. Besides, I cannot find any convincing designs tailored for the medical imaging, and believe that this method can also work for generic domains.\n- The value of incremental medical object detection is unclear. Many medical foundational models, like medical SAM, can achieve superior zero-shot performance on new datasets. What is the value of designing a handcrafted incremental setting instead of scaling up datasets for the model training?", "methods_and_evaluation_criteria": "Even though the proposed method lacks convening evidence tailored for medical domains, the technical parts of decoupling the prompts and enabling fine-grained representation are reasonable and technically sound.", "theoretical_claims": "The paper does not have proofs or theoretical claims. I have checked the equations in the methodology and did not find significant mistakes.", "experimental_designs_or_analyses": "The experimental design is sound by considering the full-data and few-shot settings, with comparisons against state-of-the-art methods and ablation studies for key components. However, there are many issues:\n\n- There are no comparisons with state-of-the-art methods regarding computational efficiency and memory usage. \n- There is also no discussion about the experiments in Sec.5.4, leading to the limited experimental insight.\n- There lacks a comparison with some latest continual learning works focusing on dense prediction [1,2]\n\n[1] Eclipse: Efficient continual learning in panoptic segmentation with visual prompt tuning CVPR 24\n[2] A survey on continual semantic segmentation: Theory, challenge, method, and application TPAMI 24", "relation_to_broader_scientific_literature": "The proposed method may provide some insights into generic object detection, continual learning, and some scenarios requiring fine-grained visual evidence.", "essential_references_not_discussed": "There is no sufficient discussion about the continual learning in dense prediction [1,2].\n\n[1] Eclipse: Efficient continual learning in panoptic segmentation with visual prompt tuning CVPR 24\n[2] A survey on continual semantic segmentation: Theory, challenge, method, and application TPAMI 24", "other_strengths_and_weaknesses": "Strength\n- The proposed methods exploring box-level prompts make sense and interesting\n- The paper is easy to follow\n\n\nWeakness\n- Lack some comparison with the continual learning works focusing on the dense predictions.\n- Lack the convening evidence and motivation to tackle the issue for medical domains instead of the generic domain\n- The model efficiency may be influenced since the method requires GLIP to generate local-level RoIs. It is necessary to compare this with other methods of efficiency.", "other_comments_or_suggestions": "Why the sentence in Sec 4.1. is in blue color?", "questions_for_authors": "- Since GLIP is trained in the generic domain, how can the quality of generated RoIs be ensured?\n- In Tab.1, using all data to train a unified model gives the best performance. So, how to justify the clinical value of the continual learning setting in the medical domain?\n- How about the model efficiency compared with state-of-the-art works, such as model parameters and inference speed?\n- The mentioned conceptual gap between medical and natural domains is unclear. How does the proposed method address this gap? The natural domain seems more challenging since it consists of more diverse object classes, appearances, and scales. Besides, the authors used GLIP pre-trained in the natural domain, which has a gap with the medical domain if the claimed gap exists.\n- How about the comparison with the dense prediction methods in continual learning?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "cf29UTLI4t", "reviewer": "Reviewer_VMtc", "summary": "This paper proposes a novel framework, instance Decoupled Prompt Attention (iDPA), for incremental object detection in medical images. This task is challenging due to the strong coupling between foreground-background features and the large domain gap between natural and medical images. This work proposes instance-level prompt generation (IPG) and decoupled prompt attention (DPA) to more effectively leverage the medical object information and prompt knowledge. Extensive comparison experiments on full dataset and few-shot settings demonstrate the superiority of the proposed method over the existing incremental object detectors.\n\n## update after rebuttal\nThe authors added the experiments to address most of my concerns. I also read the discussions between the other reviewers and the authors. I appreciate the efforts that authors compare the proposed method with SAM2.1-L and discuss the model efficiency. Overall, I would like to keep my positive score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your detailed feedback and suggestions for improvement. We treasure the opportunity to address your concerns and improve our work.\n\n##  Weakness 1: Domain Shift Robustness with Bounding Box-only Features\nThank you for this important observation. The paper focuses emphasizes instance-level knowledge decoupling to specifically focus on discriminative foreground regions (e.g., lesions or organs), thereby minimizing interference from irrelevant background information. The inherent cross-modal alignment capabilities of VLOD models such as GLIP also help in capturing contextual relationships, which further reduces dependency on background features. Empirical results in Table 1 demonstrate that our proposed iDPA consistently outperforms global prompt-based methods (e.g., L2P, DualPrompt) in full-data settings, indicating that localized feature extraction improves detection accuracy without compromising robustness to domain shifts. Moreover, Figure 5(a) empirically validates that instance-level knowledge surpasses image-level knowledge in our scenario. Nevertheless, we agree with the reviewer that selectively incorporating relevant background context could further enhance the model’s robustness. Thus, in future work, we plan to explore hybrid attention mechanisms that effectively integrate valuable background information during the feature-to-prompt knowledge transfer process.\n\n## Weakness 2: Lack of Ablation on Scaling Factor α in Eq. (6)\nWe would like to thank the reviewer for the suggestion. We have conducted new experiments with respect to $\\gamma$, as shown in the table below:\n| Scaling factor | FAP ↑  | CAP ↑  | FFP ↓ |\n|----------------|--------|--------|-------|\n| 1.00           | 50.07  | 54.01  | 2.57  |\n| 1.30           | 50.28  | 54.10  | 2.48  |\n| 1.50           | 49.99  | 53.40  | 2.77  |\n\n## Weakness 3: Ambiguity in W_k and W_v in Eq. (8)\nW_k and W_v are **linear projection layers** mapping instance features (v_c) into query/key/value spaces for cross-attention. They enable adaptive feature alignment between instance representations and task-specific prompts.  These will be clearly defined in the revised manuscript.\n\n## Weakness 4: Limited Initialization Scope in CCPKI\n\nIncremental initialization (from task $i-1$) balances the stability-plasticity trade-off: recent tasks are given higher priority to mitigate forgetting, while older tasks are retained through the prompt pool. \n\nCompared to using \"all tasks before the $i$-th one,\" this approach reduces the number of additional model parameters. By using the $i-1$-th task for initialization, model training parameters remain consistent, requiring only $1 \\times \\theta^{CCPKI}_{i-1}$.\n\nIn contrast, using all tasks prior to the $i$-th task would require a total of $i \\times \\theta^{CCPKI}_{i-1}$ additional parameters.\n\n## Weakness 5: Unclear Contribution of DPA Components\nTo clarify whether DPA's gains stem from removing the $[\\overline{p_t}]$ learning (Eq. 10) or introducing $\\lambda$ (Eq. 13), we present this in Fig. 5(b). Compared to the naive method, removing $[\\overline{p_t}]$ (i.e., when the scale type is 1.0) results in a +2.79\\% FAP, while our final approach, which introduces $\\lambda$ (dim), leads to a +5.29\\% FAP.\n\n## Weakness 6: Unfair Comparison of Naïve Baseline in Table 3\n\nWe appreciate the reviewer’s concern. However, due to hyperparameter differences, the Naïve baseline in Table 3 operates on $\\Phi_{f}$, which requires fewer parameters compared to $(\\Phi_{v} + \\Phi_{t})$, resulting in lower memory usage and reduced training time. In contrast, the methods in Table 1 operate on $(\\Phi_{v} + \\Phi_{t})$. To further clarify, we have added the Naïve method for $(\\Phi_{v} + \\Phi_{t})$, which achieves 41.72 FAP, 47.30 CAP, and 8.57 FFP. By introducing the iDPA method on top of the Naïve approach for $(\\Phi_{v} + \\Phi_{t})$, we significantly improve these metrics, with FAP increasing by 6.69, CAP by 6.26, and FFP decreasing by 4.19. These improvements, compared to the methods in Table 1, highlight that the gains are due to the effectiveness of our approach, rather than hyperparameter tuning discrepancies.\n\n| $(\\Phi_{v} + \\Phi_{t})$ | FAP ↑  | CAP ↑  | FFP ↓ |\n|------------------------|--------|--------|-------|\n| Naïve | 41.72  | 47.30  | 8.57  |\n| iDPA (ours)            | 48.41  | 53.56  | 4.38  |\n| $\\Delta$               | 6.69   | 6.26   | 4.19  |", "claims_and_evidence": "Yes, the claims in this paper are supported by its experiments.", "methods_and_evaluation_criteria": "This paper proposed a new method for incremental medical object detection. A large dataset for this task is built upon 13 existing datasets to evaluate its method.", "theoretical_claims": "I have checked the formulas in this paper and found no errors.", "experimental_designs_or_analyses": "I have checked the validity of the experiment designs and found them reasonable.", "relation_to_broader_scientific_literature": "This work is meaningful for building stronger tools with continual learning abilities for medical image analysis.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\n1)\tThis paper is written clearly and easy to follow.\n\n2)\tThis method designs the knowledge decoupling from the instance level and generates more precise concept prompts than the mixed knowledge used in classification tasks.\n\n3)\tA large-scale medical detection dataset, including 13 tasks, for incremental learning is proposed to promote the development of this research field.\n\nWeakness:\n\n1)\tDifferent medical image modalities have various visual characteristics, and different diseases appear in various body regions. The image background information is important for the medical detection task, especially when the model weights are transferred from the natural image domain. Why does IPG only use the features around and within the bounding boxes? Will this design decrease model robustness to the slight domain shift?\n\n2)\tAn ablation study needs to be conducted on the scaling factor in Eq. (6).\n\n3)\tIn Eq. (8), what do the symbols of W_k and W_v represent?\n\n4)\tIn IPG, why is the CCPKI for the i-th task only initialized from the i-1-th task instead of all tasks before the i-th one?\n\n5)\tIt remains unclear in DPA whether the performance improvement is primarily attributed to Eq. (10), removing the learning of [¯p_t], or Eq. (13), learning new parameters $\\lambda$. We recommend conducting more detailed ablation studies on the DPA modules.\n\n6)\tIn Table 3, why does the performance of the naïve baseline already outperform most of the methods in Table 1? Do the authors conduct grid search for all comparison methods to choose their best learning rates?", "other_comments_or_suggestions": "No", "questions_for_authors": "Please solve the questions in the weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes iDPA, a novel framework for incremental medical object detection, introducing two core components: (1) Instance-level Prompt Generation (IPG), which learns fine-grained task-specific prompts from localized visual features, and (2) Decoupled Prompt Attention (DPA), which improves prompt-token interactions and alleviates interference during transfer. The authors validate iDPA across 13 diverse medical datasets in both full-data and few-shot settings and show consistent gains over prompt- and non-prompt-based baselines.

The paper received uniformly weak accept scores from all reviewers. Reviewers generally appreciated the paper's novel framework for incremental medical object detection and its practical value in handling continual learning across diverse medical datasets. Key concerns about lack of comparisons to one-for-all baselines and justifications were raised initially, however, the authors provided a thorough and thoughtful rebuttal, addressing nearly all concerns from reviewers.  The overall consensus leans towards acceptance.
Decision: Accept (poster)