Prompt as Knowledge Bank: Boost Vision-language model via Structural Representation for  zero-shot medical detection
Authors: Yuguang Yang
Tongfei Chen
Haoyu Huang
Linlin Yang
Chunyu Xie
Dawei Leng
Xianbin Cao
Baochang Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Vision-language model;  zero-shot enhancement; Structural Representation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Zero-shot medical detection can further improve detection performance without relying on annotated medical images even upon the fine-tuned model, showing great clinical value. Recent studies leverage grounded vision-language models (GLIP) to achieve this by using detailed disease descriptions as prompts for the target disease name during the inference phase.  
However, these methods typically treat prompts as equivalent context to the target name, making it difficult to assign specific disease knowledge based on visual information, leading to a coarse alignment between images and target descriptions. In this paper, we propose StructuralGLIP, which introduces an auxiliary branch to encode prompts into a latent knowledge bank layer-by-layer, enabling more context-aware and fine-grained alignment. Specifically, in each layer, we select highly similar features from both the image representation and the knowledge bank, forming structural representations that capture nuanced relationships between image patches and target descriptions. These features are then fused across modalities to further enhance detection performance.
Extensive experiments demonstrate that StructuralGLIP achieves a +4.1\% AP improvement over prior state-of-the-art methods across seven zero-shot medical detection benchmarks, and consistently improves fine-tuned models by +3.2\% AP on endoscopy image datasets.

Published as a conference paper at ICLR 2025
PROMPT AS KNOWLEDGE BANK: BOOST VISION- LANGUAGE MODEL VIA STRUCTURAL REPRESENTA- TION FOR ZERO-SHOT MEDICAL DETECTION
Yuguang Yang".1:; Tongfei Chen -3 Haoyu Huang 3,5 Linlin Yangl., Chunyu Xiel
Dawei Lengl.? Xianbin Cao', Baochang Zhang
1 School of Electronic Information Engincering, Beihang University, China 2360 AI Research, Qihoo 360, China
3 School of Artificial Intelligence, Beihang University, China
Sttte Key Laboratory of Media Convergence and Communication, Communication University of China, China
National Superior College for Engineers, Beihang University, China
Artifiiial Intelligence Research Center, Lobachevsky State University,
Nizhny Novgorod 603022, Russia
ABSTRACT
Zero-shot medical detection enhances existing models without relying on an- notated medical images, offering significant clinical value. By using grounded vision-language models (GLIP) with detailed disease descriptions as prompts, doctors can flexibly incorporate new disease characteristics to improve detection performance. However, current methods often oversimplify prompts as mere equiv- alents to discase names and lacks the ability to incorporate visual cues, leading to coarse image-description alignment. To address this, we propose StructuralGLIP, a framework that encodes prompts into a latent knowledge bank, enabling more context-aware and fine-grained alignment. By selecting and matching the most relevant features from image representations and the knowledge bank at layers, StructuralGLIP captures nuanced relationships between image patches and tar- get descriptions. This approach also supports category-level prompts, which can remain fixed across all instances of the same category and provide more com- prehensive information compared to instance-level prompts. Our experiments show that StructuralGLIIP outperforms previous methods across various zero-shot and fine-tuned medical detection benchmarks. The code will be available at
INTRODUCTION
Zero-shot medical detection is crucial in healthcare as it enhances detection capabilities without requiring additional annotated medical images, even after model fine-tuning (Badawi et al.) 2024 Mahapatra et al. [2221: Qin et al.] 2022). This is particularly valuable in clinical settings, where doctors often encounter new disease characteristics not previously documented. such cases, clinicians temporarily create custom prompts guide the detection process, allowing models to adapt to novel scenarios more effectively. Recent studies have explored the potential of grounded language-iman image pre-training models (GLIP) (Phan et al. 2024 Tiu et al. 2022 Li al., 2022c Yao et al. 2022) to reduce dependence on annotations by leveraging prior knowledge. These models conduct detection by contrasting image features with descriptive texts, known contextual prompis, generated by visual question-answer models for query objects. To adapt GLIP to the medical domain, recent works QQin et 2022 Wu al. 2023b; Guo et al. 2023 have employed medically enhanced
question-answee answer models like PubMedBERT (Gu et al. 2021 and BLIP (Li et al. 2022a) to create
Co-First Authors. (guangbuaa, tfehen] @ buua..edu.cm t: Corresponding Authors. Iyang@ @ww.eeeu.ccc.. xiechunyu 3600ccn
I: Project Lead.

---

Published as a conference paper at ICLR 2025
Main Branch
Auxiliary Branch
Target Description
Target Description
polyp
Polyp red, round,..
polyp
Polyp red, round....
N
Language Encoder Layers
Image Encoder Layers
Language Encoder Layer
Language Encoder Layer
Image Encoder layer
Mutual Selection
bump
pink
[cls]
red
Structural Representation
RPN
RPN
[x1, x2, y1, y2]
[x1, x2, y1, y2]
(a) Contextual prompt methods
(b) Structural prompt methods
Figure (a) Contextual prompt methods directly concatenate the prompt and target. (b) Our structural prompt method encodes prompts into a latent knowledge bank.
attribute-rich prompts. These prompts capture nuanced characteristics of query targets, improving domain adaptation and performance beyond traditional supervised training.
However, existing contextual prompt-based methods often suffer from coarse alignment between images and target descriptions, resulting two key issues. First, these methods typically treat prompts as contexts that are equivalent to the target, easily causing distribution shift problems to the target's representation. Despite incorporating the prior about the target, they also introduce distracting information about the target. This leads to misalignment between the target and the actual visual cues in the image (see Fig. E). Second, category-level descriptions can not be effectively encoded within the context, which often contain ambiguous vocabularies such "tissue with pink red color, irregular or round shape" for a "bump". This causes that the most relevant prompt can not be precisely matched with the input image.
To address the aforementioned issues, we present StructuraIGLIP. novel zero-shot medical detec- tion model that derives structural representations, which are delicately organized sets of features specifically designed to represent the nuances of the target and the input image. Specifically, as shown in Fig. [ instead of simply concatenating prompts with the target, StructuralGIIIP adopts dual-branch architecture. The main branch processes the target name and input image, while the auxiliary branch encodes the prompts into latent knowledge bank. At each layer, rather than directly performing cross-modal fusion between vision and language features, StructuralGLIP introduces a mutual selection mechanism. This mechanism matches vision features from the main branch with relevant prompt features stored in the latent knowledge bank, where extract latent prompt tokens and latent vision tokens that both highly relevant to the target and the current input image, forming fineggriined structural representations. Once these structural representations the image and language features from the main branch are fused with the selected prompt tokens via cross-modality multi-head attention (Vaswani al. 2017). This enhances the overall feature alignment and improves the fusion process within the main branch. Conceptually, the hierarchical knowledge bank in StructuralGLIP functions like memory system (Bi 202 I Paivio 2013). As the image is processed, relevant knowledge is dynamically retrieved from the bank. This enables the model better align the image features with the prompt information, resulting in more accurate and context-aware detection.
formed,
In this way, StrueturalGI.I can address the challenge of effectively utilizing category-level prompts, which provide broader yet consistent information for all instances within the same category (see Fig. B for visualization). StructuralGLIP's instance-wise selection mechanism ensures that even fixed category-level prompts are dynamically aligned with the specific visual features of each instance. This not only improves detection precision but also enhances efficiency, as category-level prompts remain fixed across instances of the category. To validate the proposed method, bench- mark StructuralGLIP against previous state-of-the the-trt-rt methods on eight datasets under endoscopy, microscopy, photography, and radiology four imaging conditions, and conduct comprehensive
2

---

Published as a conference paper at ICLR 2025
analysis towards StructuralGLIP's structural representations. The primary contributions of our work are as follows:
We introduce StructuralGLIP, novel architecture that achieves adaptive, context-aware alignment between visual features and target descriptions by utilizing a dual-branch structure with mutual selection, enhancing the precision of medical object detection.
We propose the use of category-level prompts, which remain fixed for all instances of the same target. Unlike instance-level prompts, category-level prompts provide more comprehensive prior knowledge about the target disease, reducing the need for prompt generation for each individual image while maintaining strong detection performance.
We explore zero-shot medical detection in more practical settings by demonstrating how zero- shot enhancement can further improve the performance of models fine-tuned on medical data. Structural not only surpasses fully supervised methods such as RetinaNet but also seamlessly integrates into GLIP models fine-tuned on medical datasets, achieving an average improvement of +4% AP.
2 RELATED WORK
Zero-shot medical detection aims to identify and locate pathology concepts in medical images
without relying on annotated data from the target domain (Vilouras al., 2024 Qin et al. 2022
Paul et al. 2021 Mahapatra et al. 2021] Sahasrabudhe et al.) 2020 Le Bescond et al. 2022)
Classical strategies include cross- cross-domain domain generalization (Bian et al. [022 Bansal al. 2024) and
unsupervised learning (Sahasrabudhe et al. 2020 Le Bescond et al. 2022 Paul et al. 2021). Cross-
domain generalization utilizes data from related domains under varied conditions, such as different imaging techniques (Bian al. 2022) demographic differences (Bansal et al. 2024), to adapt models across diverse scenarios. Unsupervised learning methods leverage side information to bypass direct supervision, such using cell nuclei structure for image resolution analysis (Sahasrabudhe et al. 2020), employing GANs with public annotations to enhance mask quality (Le Bescond et al. 2022). and correlating medical reports with disease features to increase detection accuracy (Paul et al. 2021). ). However, these methods are often tightly coupled to specific data priors and exhibit a considerable performance gap compared to supervised models, limiting their clinical significance.
Recent approaches have integrated expert-level knowledge into vision-language models trained on natural images facilitate domain transfer (Liu al., 2023a Lai al.. [0244] Tiu et al.) [2022] Wu
it al. 2023a Zhang et al. 2023). However, most of these efforts focus on medical classification, while the more practical and complex task of medical detection remains underexplored. For example, (Qin et al. (022)) conducted comprehensive study on medical detection using prompts generated by a medically-enhanees
language model, PubMedBERT (Gu et al.) 2021). Follow-up studies (Wul ett al. 2023b Lu et al., [023] Phan et al.) 2024) employed BLIP (Li et al. 2022a) to generate image-specific linguistic attributes, or used GPT (Achiam et al. 2023) to detail target concepts with nuanced descriptions. Recent work (Guo al. 2023) further ad advanced this approach by introducing an ensemble strategy for fusing multiple prompts to improve detection accuracy. However, these methods require unique prompts for each instance, significantly reducing efficiency. Our method, StructuralGI.nn addresses these challenges by introducing a vision-language model that leverages a knowledge bank to store a wide range prompts, enabling instance-dynamie prompt selection in the latent feature space.
prompt method is initially developed for continual learning, which utilizes a prompt pool designed to enhance cross-domain generalization (Wang et al. 2022b a Smith et al. 2023. Wang al., 2023 Du et al. 2022). Previous works (Wang et al. 2002bb] a select top-k prompts aligned with input image features, facilitating domain-specilfs modeling. Recent advances have evolved this strategy, replacing the top-k prompt selection with a more flexible continuous prompt fusion strategy (Smith et al.) 2023), exploring its potential for vision- vision-language model (Wang et al.) 2023), and expanding applications to opea-vocabulary detection tasks (Du et al., 2022). However, these methods typically require an additional training phase and are restricted to prompt retrieval in the input layer. In contrast, StructuralGLIP explores a linguistically accessible avenue by directly utilizing the attributes predefined by the generative models and embeds these attribute prompts into a hierarchy knowledge bank situated within an auxiliary branch to achieve a layer-wise selection process.
Knowledge-bank-ben
3

---

Published as a conference paper at ICLR 2025
3 METHODOLOGY
3.1 PRELIMINARIES
prompt
category
prompt target
category
VQA
target 1
prompt
VQA
prompt
Grounded
Grounded
Finetuned
Finetuned Grounded Detector
Detector
Detector
Grounded Detector
(a) zero-shot detection (instance prompt)
(b) zero-shot detection (c) zero-shot enhancement
(d) zero-shot enhancement
(category prompt)
(instance prompt)
(category prompt)
Figure Experimental settings for zero-shot medical detection and enhancement.
Zero-shot medical object detection means improving the model's medical detection performance without the use of supervised image labels. This formulation emphasizes "further improvement without supervised images", which contains two experimental settings. Firstly, the classical setting, the model, without fine-tuning on medical datasets, uses pre-trained vision-language models with prompts to infer medical concepts (see Fig. (a) and (b)). Secondly, considering the clinical setting prefers supervised models for their excellent performance,
propose zero-shot enhancement setting. This involves fine-tuning the model on medical datasets first, and then using prompts to further improve performance Fig. ( and (d)). This setting mirrors real-world clinical needs, where models can be continuously improved with knowledge without the need for labeled data.
unseen medical images, without requiring additional labels (see
GLIP redefines object detection as phrase-grounding task by employing late fusion dual-tower architecture align image and text features. It uses separate backbones Enc, and Enor extract initial encodings O" and pa for images and text, respectively. These features are then integrated through cross- modal multi-head attention module (X-MHA), enabling fine-grained interaction between the modalities. The integration of image and text features through the deep fusion module (X-MHA) is formalized as follows:
OO P X-MHA(O', Pt); 0 1f + Ozaw), pi+1 = fi P P(2:);
(1)
(2)
where fi and fi the 5th encoder layers for images and text, respectively, and i [1, N]. After N layers of interaction, the final image and text representations are denoted ON and pM respectively. These representations are used as input to the RPN for generating object proposals:
RGLIP = RPN(ON PN),
(3)
where ReGI denotes the set of region proposals of GLIP generated by the RPN. Each proposal E R is characterized by its bounding box coordinates and a confidence score, indicating the likelihood of the region containing the target object.
3.2 ZERO-SHOE HERO
SHOT DUAL- BRANCBRANCH PROMPT FRAMEWORK
In the proposed StructuralGLIP framework, we introduce novel zero-shot architecture to achieve fine- iine-grained alignment between target description and medical images. The overall pipeline is shown in Fig.
Structurally separated auxiliary and main branches. StructuralGLIP adopts a dual-branch architecture. The main branch processes the target name and input image, while the auxiliary branch encodes the prompts into a latent knowledge bank. Given the object target 7 and the prompt Prompt, the initial representations 7 and 3d are obtained as follows:
T = (ncz,((), Dd EEncz, (Prompt),
4

---

Published as a conference paper at ICLR 2025
Main Branch
Auxiliary Branch
take the union
name
prompt
Input
Instance-level
Query What's the shape, color, morphomi netic polyp ?
sample
Contextual Prompts
Language Backbone
Language Backbone
Image Backbone
pink
bump round
VQA
prompt
encoder Layer fi.
encoder Layer
encoder layer fi
sample' 7t
red
trigle
oval
prompt
Top-P P
B
Category-level Prompt Bank [round, polygon, oval, circle] color: [pink, white, red, yellow, red-yellow] morphomimetic: [bump, heart, nodular-like]
0
0
bb
shape:
Top-Q.
RPN [x1, x2. yl,
(b) Medical prompt generation pipeline
StructuratGLIP pipeline
Figure 3: Pipeline of the proposed method and the automatic prompt generation.
where Encz, and Encz, are language backbones with shared parameters for the main and auxiliary branches, respectively. Here, T and BO IR. x represent the initial encoded features the target and prompts, with [PAD] tokens used to pad the input sentences to uniform length Ni. The encoded prompts 13 are then processed through the language encoder layers:
= fi,
(5)
where fi,, is the 7th language encoder layer of the auxiliary branch, and B denotes representation of prompt bank at the 7th layer.
Mutual prompt selection mechanism for structural representation in the auxiliary branch. This mechanism identifies mutually relevant tokens between the visual tokens from the main branch and the linguistic tokens from the auxiliary branch. For selecting the relearet from the latent representation of the input image, we calculate their similarity with latent prompt features. The visual and linguistic representations in the i-th layer are denoted as O E R N. and By IR N. respectively. We have the following:
relevant visual tokens
O = (oi, .., ON.
02..,,
K. Tmmap [key = o',, value o Bi)
(6)
where Top-pmar (key,, value]) denotes selecting the keys with the Top-P maximal values. KÇi is the selected visual tokens in the i-th encoder layer, and N, is the token length of the visual encoder. Similarly, select the Top-Q tokens from the latent representation of the prompt, use the similarity to the selected visual tokens IC;
B bbi D..
by
Ni C = Top-Qmaa ([ky = b),, value = b
where Top-Qmaz (lkey, valuel) denotes selecting the keys with the Top-Q maximal values. XC} is the selected linguistic tokens in the i-th layer, and Ni is the token length of the language encoder. These selected prompt tokens XC) and XCj are highly relevant to the target and the current input image, forming fine-grained structural representations.
Deep fusion with the vision-language prompt in the main branch. Once the structural representa- tions obtained, we serve these selected prompt tokens KJ and KC} as latent prompts for the deep fusion process of GLIP. Instead of using the auxiliary language encoder to enhance the features, the main branch's vision and language encoders leverage the knowledge from the selected tokens K; and KG This ensures that comprehensive knowledge from the prompts be extracted precisely

---

Published as a conference paper at ICLR 2025
and applied in an instance- wise manner enhance the detection process. Specifically, employ multi-head attention (MHA) mechanism Vaswani et al. (2017) for (K;, T2) and (K₁ 0₂):
MHA(Q K., KV TT)) :2n
T MHA(Q KC KV O
(8)
where Q denotes the query item and KV denotes the key and value items for MHA, and 1oo0 T the input image and target representations that incorporate the prior knowledge about the target from the selected tokens, respectively. These representations are then combined with the original layer representation using the following residual connection:
0 = fi(0" OPPP)
T = ff dppp
This deep fusion mechanism ensures that the model dynamically integrates relevant prompts each layer, significantly enhancing instance-specifi adaptation for zero-shot medical detection. After N layers of interaction, denoted as O and TNN, respectively. Here, T represents the target's s representation, which has fused prompt information relevant to the current instance, achieving more precise alignment with ON These representations are then used as input to the RPN for generating object proposals:
obtain the final image and text representations from the main branch,
RsmecturalLLIP RPN(ON .T*),
(10)
where RSmucturatGLII denotes the set of region proposals generated by the RPN in StructuraIGLIP. This process effectively combines the structural representations from both the visual and language modalities to achieve accurate and context-aware zero-shot detection.
3.3 INSTANCE/CATEGORR--EEN
MEDICAL PROMPT AUTOMATIC GENERATION
As shown in Fig. b(),, we propose a dual-level prompt generation mechanism that constructs a com- prehensive prompt repository at both the instance and category levels. This enables StructuraIGLIP to dynamically apply the most relevant knowledge during inference, significantly improving detection accuracy.
Instance-level Prompt Generation. For each medical image, we generate instance-specif prompts to capture unique visual features such as shape, color, and morphology using Visual Question Answering (VQA) model like BLIP Li et al. (2022a). We query the model with targeted questions (e.g., "What the shape of the polyp?"), and the responses form a set of instance-level contextual prompts (e.g., "[pink- "[pink-white, bump- like, round]")). This process ensures that the model can dynamically adapt to the specific characteristics each image, providing fine-grained descriptions that are crucial for precise detection.
Category-level Prompt Generation. parallel, we construct category-level prompt bank con- taining general attributes relevant to each medical category. Using a language model like GPT-4, we generate detailed descriptions for common attributes such as shape, color, and morphology (e.g.. "typical shapes of polyps include round, oval, and nodular-like"), This enriched prompt bank serves as a static reference, enabling the model to capture the broader context of each category and generalize effectively across diverse medical cases. Finally, we gather all attributes from the instance-level prompt and concatenate them with the GPT-4 augmented results to derive the category-level prompt (displayed in Appendix
Application. Category- prompts provide comprehensive information for entire classes of medical images and remain fixed across all images within the same category, offering higher efficiency compared to restance-specine prompts. Despite this advantage, prior methods Guo et al. (023 : Qin
ett al. 22022;; Wu et al. (2023b) have not fully benefited from general prompts (see Tab. 2 due to their lack of adaptive prompt selection. StructuralGLIP however, utilizes instance-wise selection mechanism that supports category-level prompts effectively. This allows the model to dynamically select the most relevant prompts from the prompt bank, achieving performance comparable even better than instance-level prompts on certain datasets. This demonstrates that our method can efficiently leverage general prompts to enhance zero-shot detection without the need for instance- specific generation.

---

Published as a conference paper at ICLR 2025
Table I: Comparative experiment results on zero-shot medical detection across seven datasets, where gray-shaded rows represent the instance-level prompt results, while the unshaded rows represent the category-level prompt results.
Methods
CVC-300 Kvasir ColonDB ClinicDB ETIS ISIC 2016 BCCD
Avg. AP AP50 AP AP50 AP AP50 AP AP50 AP AP50 AP AP50 AP AP50 AP AP50
GLIP
29.8 37.9 25.9 33.6 21.7 32.4 22.1 29.6 6.7 9.7 10.5 20.0 8.9 18.4 17.9 25.9
MIU-VL
36.5 66.6 28.7 36.6 19.8 35.6 28.2 40.6 9.4 15,4 21.7 35.7 11.4 20.4 22.2 35.8
AutoPrompter 52.7 70.6 30.4 39.7 31.9 45.9 22.0 30.6 17.7 26.5 19.9 32.9 12.9 22.3 26.7 38.3
Ours (instance)
54.3 72.8 34.7 43.1 35.3 51.3 28.6 38.2 22.2 31.9 27.7 40.8 13.5 24.1 30.9 43.1
MPT w. WBF 3.27 9.40 12.2 14.4 14.2 19.1 11.2 14.0 12.0 17.0 1.13 5.37 1.22 4.75 7.8 12.0
MPT w. Cluster36.7 47.5 12.0 17.0 11.9 21.4 11.2 14.0 12.0 17.0 19.8 30.9 14.3 33.8 16.8 25.9
Ours (category) 63.9 89.8 42.0 50.5 42.1 66.0 42.0 57.0 30.4 40.3 21.8 33.5 23.6 40.9 37.9 54.0
EXPERIMENT
We illustrate our experiment settings in Fig. _ where we design four distinct settings to evaluate the model's 's performance. In Sec. 4.2 we follow traditional zero-shot setups to evaluate StructuralGLIP in zero-shot setting without any fine-tuning on medical datasets, using both instance-specific and sategory-speciffe prompts (see Fig. aaa) and (b)). In Sec. 4.3 we simulate clinical environments where supervised models are typically preferred. Here, we fine-tune the backbone of the proposed methods, i.e., GLIP, (without using prompts) fine-tuning, we incorporate linguistic prompts for the target disease into StructuralGL.IP to perform zero-shot enhancement, evaluating the model's 's ability to improve performance even after fine-tuning (see Fig. ((c) and (d)). The fine-tuned details are provided in Appendix D
medical datasets to form refined detector. After
4.1 EXPERIMENTAL SETUP
Datasets. We select four types of medical imaging datasets involving eight benchmarks: 1) Endoscopy
datasets for polyp detection: ClinicDB Bernal et al. (2015); Fernández Essarrach et al. (2016),
ColonDB Bernal et al. (2012), Kvasir Jha et al. (2020). ETIS Silva et al. (2014); 2) Microscopy dataset: BCCD shenggan et al. (2018) for blood cells detection; 3) Photography dataset: ISIC-
2016 for skin lesions detection (benign lesion; malignant lesion); 4) Radiology image datasets:
TBX1 1K Liu et al. (2033)) for tuberculosis detection in lung X-rays. Detailed elaboration is given in
the Appendix C
Metric and baseline. To evaluate our approach, we primarily benchmark against recent studies, mainly following Qin et al. Qin al. (2022) (2023) and Wu et al. Wu et al. (2023b) (2023). Our baselines include recent based methods (vanilla GLIP Li et al. (2022b), MIU-VL Qin ct al. (2022). and AutoPrompter Wu al. (2023b)) for mustance-speciiee
prompt generation setting, and works attempt to use category-speciff prompt for detection (MPT Guo et al. (2023), and its variants MPT+SoftNMS Bodla et al. (2017), MPT+WBF Solovyev et al. (2021), ))). For zero-shot enhancement experiments, fully supervised detection models (RetinaNet Lin et al. (2020) and DyHead Dai et al. (202I) are also included to provide a comprehensive evaluation landscape for our zero-shot enhancement experiments. The training details of the GLIP are elaborated in Appendix D
4.2 RESULTS OF ZERO-SHOT MEDICAL DETECTION
Superior transfer performance across various medical scenarios. For For fairness, we ensured that all methods used consistent prompts for a fair comparison. For instance-specific prompt methods, we utilized BLIP (Li et al. 2022a) as the vision -questine answering model for all approaches, except for vanilla GLIP (Li et al. 2022bj, which directly used the target name as text input. For MIU-VL (Qin) ctt al. 2022), we additionally used PubMedBert Gu et al. (022I) to generate prompts specific to the target discase. AutoPrompter (Wu et al.) 2023b) uses GLIP to produce the initial bounding box with instance- -sauuncc- prompt and refine them with a self-training process with Yolo-X (Zheng et al.) 2021). prompt), where all prompt methods enhance the original GLIP model's performance by providing additional descriptions. Among them, Structural II achieved the greatest improvement, with average +4.2% AP 1. +4.8% AP50
The experimental results are shown in Tab.
7

---

Published as a conference paper at ICLR 2025
Table 2: AP% of vanilla GLIP and the proposed methods with instance-specific (I) and category-specific (C) prompt under zero-shot detection setting.
Table 3: AP% of fine-tuned GLIP and the proposed methods with instance-specific (I) and category- specific (C) prompt under zero-shot enhancement setting.
CVC-300 ClinicDB
Kvasir
Avg. C
CVC-300 ClinicDB
Kvasir
Avg.
C
C
I
C
I
C
C
I
C
I
C
I
GLIP 34.3 29.8 17.9 22.1 22.3 25.9 24.8 25.9
GLIP 70.0 67.5 54.3 63.0 44.5 51.1 56.2 60.5
ours 63.9 54.3 42.0 28.6 42.0 34.9 48.3 39.2
ours 77.2 74.9 70.4 68.4 71.3 69.6 72.9 70.9
across seven datasets. We do not exhibit the results for the radiology dataset TBX-11K here, as the initial performance of the GLIP model on this dataset was poor, and the performance improvement for each prompt method is not distinguishable.
Knowledge bank facilitates category-level prompts. In this experiment, we focus on the effective- ness of category-level prompts generated by BLIP and GPT-4, which expand attributes related the target across different dimensions such as colors, shapes, textures, and locations. These category- prompts, being about 10 times longer than instance-speciffc prompts, remain consistent across all instances within the same class, and their details are provided in Appendix G To benchmark against other methods, we include the MPT (Guo et al. 2023) approach, which is built upon the GLIP backbone and designed specifically to handle category- prompts. MPT employs different prompt ensemble strategies, such as Weighted Box Fusion (WBF) and clustering, to split the category prompts into multiple groups and fuse the outputs for improved performance. Tab. shows the performance comparison under different ensemble strategies. As seen in the results, StructuralGI.IP achieves supe- rior average performance across seven datasets compared to MPT. More importantly, StructuralGLIP consistently outperforms instance-specific prompt methods when utilizing category-level prompts (a +7% average AP 1 across seven datasets). This suggests that StructuralLLIP can effectively harness the richer and more comprehensive information encoded in the category prompts.
We attribute this advantage the dual-branch architecture of StructuralGLIP. where the prompts and image features are separated into auxiliary and main branch, respectively. By introducing an instance-win wise selection mechanism, StructuralGLiP can dynamically select the most relevant parts of the category prompt based on the input image. To further verify this, we directly feed the category prompt for GLIP to obtain GLIP's performance under the category prompt and follow MIU-VL to obtain its performance under the instance prompt, As shown in Tab. 2 The results demonstrate significant improvement (average AP of 24.8 - 49.3) in StructuralGLIP performance compared to the vanilla GLIP with category-level prompts. Interestingly, by comparing the performance of GLIP between using category-level prompt (see Tab. 2 and instance-level prompt (see Tab. [ vanilla GLIP exhibit performance degradation when category prompts employed (average AP of 25.7 - 24.8). In contrast, StructuralGLIP shows a significant AP improvement (39.2 + 49.3). This highlights the advantage of StrueturalGLIIPs knowledge modeling and its ability to dynamically extract the most relevant prompt information for each instance, effectively leveraging the comprehensive knowledge provided by category- prompts.
4.3 RESULTS OF ZERO-SHOT ENHANCEMENT FOR MEDICAL DETECTION
StructuralGLIP surpasses the fully-supervised methods. In this experiment, we evaluate zero-shot enhancement and also compare fine-tuned GLIP-based models with classic object detection models, such as FasterRCNN Ren et al. (2015) and RetinaNet et al. (2020), which were fully supervised. As shown in Tab. 4 while the refined GLIP performs similarly to the supervised RetinaNet (55.2 vs. 56.6 average AP),, incorporating instance-level prompts with StructuraIGLIP raises the performance to 59.3 AP, Tab. +2.7% improvement. For category-level prompts, StructuralGLIP achieves an average AP of 60.6, showing a slight improvement over instance-leve prompts. However, given that category-level prompts remain fixed across all images of the same class and can be pre-encoded in our auxiliary branch, this performance boost comes with only the inference cost for calculating the attention matrix of prompt, further demonstrating the efficiency of our approach.
8

---

Published as a conference paper at ICLR 2025
Table 4: Comparative zero-shot enhancement experiment results across datasets, s, where gray-shaded rows represent the instance-level prompt results, while the last unshaded block repre- sents the category-level prompt results.
Methods
Kvasir ColonDB ClinicDB
ETIS CVC-300ISIC 2016
BCCD TBX-11K
AP AP50 AP AP50 AP AP50 AP AP50 AP AP50 AP AP50 AP APS0 AP AP50
FasterRCNN
63.4
44.1
71.6
44.5
59.4
50.3
56.9
33,9 73.9
RetinaNet
64.1
49.8
71.9
46.6
61.6
54.0
56.7
37.0 77.9
GLIP
64.8 82.2 56.8 79.1 65.1 82.6
60.4 77.0 75.2 95.9 39.9 50.9 55.4 78.2 35.2 75.3
MIU-VL
67.7 86.2 48.8 75.2 63.0 82.6 48.9 68.8 67.5 97.2 29.7 38.7 44.5 58.9 35.5 76.7
AutoPrompter
70.0 87.5 57.8 81.3 67.5 85.3 59.6 76.8 75.2 97.1 37.3 49.0 23.4 33.2 35.7 76.5
Ours (instance) 69.6 87.9 58.1 81.0 68.4 87.5 60.3 77.0 74.9 96.3 49.5 62.7 56.9 80.2 37.3 78.2
MPT+Cluster 25.1 30.0 22.3 29.5 24.8 29.3 24.7 29.8 33.4 41.5 25.6 33.7 22.8 30.6 31.4 68.2
Ours (category)71.3 89.0 62.0 85.3 70.4 88.2 62.4 79.5 77.2 96.5 45.9 58.3 57.8 82.4 37.8 79.2
Table 5: Ablation results for Top-Q Q (y-axis) and Top-P (x-axis) on CVC-300 dataset with zero-shot medical detection seting.
Table 6: Ablation study results (AP%) the genera- tion of category prompt using VQA and GPT across four datasets.
15.1
50.1
53.4
52.4
52.4
51.9
52.4
19.9
47.1
56.5
55.9
55.3
55.5
55.8
Methods
Kvasir ColonDB ClinicDB ETIS
19.9
47.1 1
56.0
55
54.9
55.2
55.3
MPT+VQA+GPT
12.2
14.2
11.2
12.0
17.3
48.2
55.2
54.9
54.9
55.2
Our
37.6
38.9
38.8
26.3
Ours+VQA+GPT
42.0
42.1
42.0
30.4
18.8
50.4
56.4
55.6
55.9
56.1
56.1
50.2
55.
56.2
10
15
30
40
StructuralGLIP facilitates further improvement on fine-tuned models. Interestingly, we observe that not all prompt-based methods effectively enhance fine-tuned GLIP model. As shown in Tab, 4 methods like MIU-VL and AutoPrompter experience performance degradation when applied to the refined GLIP (MIU-VLL VL: 56.6 50.7 AP, AutoPrompter: 56.6 > 53.3 AP). This decline likely occurs because these methods treat prompts simple contextual information for the target name. During fine-tuning, only the target name used as the linguistic input, causing a significant distribution shift when prompts are introduced during inference. In contrast, StructuralGI.I encodes prompts into a latent knowledge bank via the auxiliary branch, where prompts are used to construct structural representations during vision- vision-language fusion. However, the final RPN inference still relies the target name representation. In this way, StructuraIGLIP incorporates additional knowledge about the target and alleviates the distribution shifting problem the same time. This approach allows StructuralGLIP to achieve further performance gains on fine-tuned GLIP (56.6 - 59.3 AP).
4.4 ABLATION AND ANALYSIS
Prompt as Knowledge Bank. StructuraIGLIP uses a dual-branch architecture and mutual selection mechanism encode prompts into latent knowledge bank. effectively supporting category-level prompts with rich attribute knowledge. To validate this, we directly feed category-level prompts of StructuralGLIP and instance-level prompt of MIU-VI for vanilla GLIP to gain its performance with category-level prompt and instance-level prompt. respectively. As shown in Tab, 2 when employing category-level prompt, GLIP suffers performance degradation (25.9 -4488) while StructuralGI.IP gains additional performance improvement (39.2->48.3) This indicates that mutual selection helps StructuralGLIP effectively leverage category prompts by selecting the most relevant information for each image. Besides, another important advantage of embedding prompts as a knowledge bank is that this design enables the precise integration of additional knowledge without affecting the distribution of the target representation. To validate this, conducted ablation studies the fine-tuned GLIP model, where only the target name is used during the training phase. Then, we evaluate the performance of GLIP and StructuralLI.I under zero-shot enhancement setting. We present the experimental result

---

Published as a conference paper at ICLR 2025
0.7
0.7
0.6
0.6
0.5
0.5
0.4
0.4
0.3
0.3
0.2
0.2
1
2
3
5
-+t- CvC300
-o-oooooo clinicdb
kvasir ---- etis avg min
(a) Fusing at specific layer
(b) Starting from a Specific Layer to the Layer
Figure 4: Ablation towards the fusing layer of the proposed method
in Tab. Similar to the analysis in Tab. 2 the proposed StructuralGLIP effectively incorporates the knowledge from the bank without a performance degradation. As discussed in Sec. Sec. 4.3 with dual- branch architecture, the knowledge bank functions as a residual feature in thee modality fusing phase, which prevents the distribution shift by encoding prompts separately from the target representation, ensuring smooth integration of prompt knowledge during inference.
peregot--prompt generation methods. To validate the effectiveness of incorporating the prompt generated with the large language model, we exhibit the comparison of only using the VQA model and combing the results of VQA and GPT without fine-tuning the GLIP model in Tab. 4.4 Our experimental results show that GPT provide more comprehensive knowledge about the target and further improve the performance.
Ablation on the fusing layer. We performed ablation study to analyze how the layer which the latent knowledge bank is fused impacts the performance. Fig. 4 (illustrates two fusion strategies. The first strategy, shown in Fig. a), explores the effect of fusing at specific layers. The results indicate that fusion at Layer 5 yields the highest performance across multiple datasets, suggesting that this layer contains the most relevant features for effectively incorporating prompt knowledge. In contrast, earlier layers such as Layer and Layer 2 exhibit lower performance, likely due to their focus on low-level features that are less compatible with the semantic richness of the prompts. The second strategy, depicted in Fig. 4b)), investigates the effect of starting from a specific layer and fusing through the last layer (Layer 6). The results reveal a hierarchical pattern (Fusing Layer4- Layer6>Layene Layer5-Layer6>1......
where starting fusion from Layer 4 and continuing to Layer 6 achieves the best results. This indicates a progressively integrating the prompt knowledge at deeper layers allows the model to better utilize the information from the knowledge bank, rather than directly fusing at the last layer. A further analysis of this hierarchy characteristic is conducted in Appendix and more insights into the improvement of StructuralGLIP
shown in Appendix H
Ablation on Q andP. We explore the joint effects of apper-parameters of the selected visual tokens and prompt numbers Q and P on CVC-300 under zero-shot detection without the fine-tuned model. As shown in Tab, 4.4 the model achieves the optimal performance with = 10 and Q = 10. Further increasing P or introduces redundant information and degrades performance.
More important analysis towards the category-specife prompt, selection mechanism, the effect of the selected LLM to generate prompts are attached in Appendix A
CONCLUSION AND LIMITATION
We introduced Structural IP, novel zero-shot medical detection model that achieves fine-grained alignment between target descriptions and medical images. Unlike prior works that directly transfer vision- usioo-languagg models to the medical domain, extended zero-shot medical detection more practical setting by exploring both category-Ieve prompts and zero-shot enhancement. Through ex- tensive experiments, demonstrated that StructuraGGLIP excels under these conditions, significantly outperforming existing methods. In future work, we aim to extend the applicability of StructuralGLIP more diverse medical and non-medical domains, potentially improving its adaptability varied visual conditions and more complex multimodal tasks.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGE
The work was supported by the National Key Research and Development Program of China (Grant No. 2023YFC3306401), and the Beijing Natural Science Foundation (No. L.244043), and the National Natural Science Foundation of China (No. 62406298); Result in Section 4.4 was sup- ported by the Analytical Center for the Government of the Russian Federation (agreement identifier W00000D7303245540000 grant No 2023-001320 dated 27. 12.2023...
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Maged Badawi, Mohammedyahia Abushanab, Sheethal Bhat, and Andreas Maier. Review of zero- shot and few-shot algorithms in the medical domain. arXiv preprint arXiv:2406. 16143, 2024.
Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, and Ajay Divakaran. Zero-shot pediatric tuberculosis detection in chest X-rays using self-supervised learning. In ISBI, 2024.
Jorge Bernal, Javier Sánchez. and Fernando Vilarino. Towards automatic polyp detection with polyp appearance model. Pattern Recognition, 45(9):3166-3188 2012.
Jorge Bernal, Javier Sánchez, Gloria Fernández: Fernandez-Esparrachh Debora Gil, Cristina Rodríguez, and Fernando Vilariño. Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Computerized medical imaging and graphics, 43:99-111, 2015.
Yanchao Bi. Dual coding of knowledge in the human brain. Trends in Cognitive Sciences, 25(10): 883-895, 2021.
Cheng Bian, Chenglang Yuan, Kai Ma. Shuang Yu, Dong Wei, and Yefeng Zheng. Domain adaptation meets zero-shot learning: An annotation-efficicnn approach to multi-modality medical image segmentation. IEEE Transactions on Medical Imaging, 4155):0443 2022. doi: 10. 109/ TMI.2021.3111245.
Navaneeth Bodla, Bharat Singh, Rama Chellappa, and Larry S Davis. Soft-nns-improving object detection with one line of code. In ICCV, pp. 5561-5569, 2017.
Xiyang Dai, Yinpeng Chen, Bin Xiao, Dongdong Chen, Mengehen Liu, Lu Yuan, and Lei Zhang, Dynamic head: Unifying object detection heads with attentions. In CVPR, pp. 7373-7382, 202 1.
Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open- vocabuocyyylary object detection with vision-language model. In CVPR, pp. 14084-14093, 2022.
Glòria Fermández-Esparracch Jorge Bernal, Maria López-Cerón, Henry Córdova, Cristina Sánchez- Montes, Cristina Rodríguez De Miguel, and Francisco Javier Sánchez. Exploring the clinical potential of an automatic colonic polyp detection method based on the creation of energy maps. Endoscopy, pp. 837-842, 2016.
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Nomain-specifit language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH), 3(1):
1-23, 2021.
Miaotian Guo, Huahui Yi, Ziyuan Qin, Haiying Wang, Aidong Men, and Qicheng Lao. Multiple prompt fusion for zero-shot lesion detection using vision-language models. In MICCAI, pp. 283-292, 2023.
Debesh Jha, Pia H Smedsrud, Michael A Riegler, Pål Halvorsen, Thomas De Lange. Dag Johansen, and Hävard Johansen. Kvasir-seg: A segmented polyp dataset. In MMM, pp. 451-462, 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 42226990, 2014.
11

---

Published as a conference paper at ICLR 2025
Haoran Lai, Qingsong Yao, Zihang Jiang, Rongsheng Wang. Zhiyang He, Xiaodong Tao, and S Kevin Zhou. Carzero: Cross- attention alignment for radiology zero-shot classification. In CVPR, pp- 11137-11146, 2024.
Loic Le Bescond, Marvin Lerousseau, Ingrid Garberis, Fabrice André, Stergios Christodoulidis, Maria Vakalopoulou, and Hugues Talbot. Unsupervised nuclei segmentation using spatial organization priors. In MICCAI, pp. 325-335, 2022.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In ICML, pp. 12888-12900, 2022a.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang. Jenq-Neng Hwang, al. Grounded language-image pre-training. In CVPR, pp. 10975, 2022b.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In CVPR, pp. 10965-10975, 2022c.
Tsung-Yo Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar. Focal loss for dense object detection. IEEE transactions on pattern analysis and machine intelligence, 42(2):318 -27,, 2020.
Jiaxiang Liu, Tianxiang Hu, Yan Zhang, Xiaotang Gai, YANG FENG, and Zuozhu Liu. A chatgpt aided explainable framework for zero-shot medical image diagnosis. In ICML, 2023a.
Yun Liu, Yu-Huan Wu, Shi-Chen Zhang, Li Liu, Min Wu, and Ming-Ming Cheng. Revisiting computer-aided tuberculosis diagnosís. IEEE Transactions on Pullern Analysis and Machine Intelligence, 2023b.
Ming Y Lu, Bowen Chen, Andrew Zhang, Drew FK Williamson, Richard Chen, Tong Ding. Long Phi Le, Yung-Sung Chuang, and Faisal Mahmood. Visual language pretrained multiple instance zero-shot transfer for histopathology images. In CVPR, pp. 19764-19775, 2023.
Dwarikanath Mahapatra, Behzad Bozorgtabar, and Zongyuan Ge. Medical image classification using generalized zero shot learning. In ICCV, pp. 3114-3344555 3353, 2021.
Allan Paivio. Imagery and verbal processes. Psychology Press, 2013.
Angshuman Paul, Thomas C Shen, Sungwon Lee, Niranjan Balachandar, Yifan Peng, Zhiyong Lu, and Ronald M Summers. Generalized zero-shot chest x-ray diagnosis through trait- guided multi-view semantic embedding with self-training. IEEE Transactions Medical Imaging, 40 (10):2642-2655, 2021.
Vu Minh Hieu Phan, Yutong Xie, Yuankai Qi, Lingqiao Liu, Liyang Liu, Bowen Zhang, Zhibin Liao, Qi Wu, Minh-Son To, and Johan W. Verjans. Decomposing disease descriptions for enhanced pathology detection: A multi-aspeet vision-language pre-training framework, 2024.
Ziyuan Qin, Hua Hui Yi, Qicheng Lao, and Kang Li. Medical image understanding with pretrained vision language models: A comprehensive study. In ICLR, 2022.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances neural information processing systems, 28, 2015.
Mihir Sahasrabudhe, Stergios Christodoulidis, Roberto Salgado, Stefan Michiels, Sherene Loi, Fabrice André, Nikos Paragios, and Maria Vakalopoulou. Self-supervieed nuclei segmentation in histopathological images using attention. In MICCAI, pp. 393-402, 2020.
shenggan, Nicolas Chen, cosmicad, and akshaylamba. Beed: Blood cell count and detection, 2018.
URL http::tttt://ggtttt hhtp:///ttttb...c cww Swwnsgaannnaaee..e BCCD BBSSCCDDDataase
Juan Silva, Aymeric Histace, Olivier Romain, Xavier Dray, and Bertrand Granado. Toward embedded detection of polyps in wee images for early diagnosis of colorectal cancer. International journal of computer assisted radiology and surgery, 9:283-293 2014.
12

---

Published as a conference paper at ICLR 2025
James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante- Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention-basee prompting for rehearsal-fin -ree continual learning. In CVPR. pp. (1909-11919) 2023.
Roman Solovyev, Weimin Wang, and Tatiana Gabruseva. Weighted boxes fusion: Ensembling boxes from different object detection models. Image and Vision Computing, 107:104117 17, 2021.
Ekin Tiu, Ellie Talius, Pujan Patel, Curtis P Langlotz, Andrew Y Ng. and Pranav Rajpurkar. Expert- level detection of pathologies from unannotated chest x-ray images via self-supervised learning. Nature Biomedical Engineering, 1399-14666 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Konstantinos Vilouras, Pedro Sanchez, Alison Q O'Neil, and Sotirios Tsaftaris. Zero-shot medical phrase grounding with off-the-shelf diffusion models. arXiv preprint arXiv:2404. 2024.
Runqi Wang, Xiaoyue Duan, Guoliang Kang. Jianzhuang Liu, Shaohui Lin, Songcen Xu, Jinhu Lü, and Baochang Zhang. Attriclip: A non-incremental learner for incremental knowledge learning. In CVPR, pp. 3654-3663, 2023.
Zifeng Wang. Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In ECCV, pp. 631-648, 2022a.
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In CVPR, pp. 139-149, 2022b.
Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Medklip: Medical knowledge enhanced language-image pre-training for x-ray diagnosis. In CVPR, pp-
2023a.
Yongjian Wu, Yang Zhou, Jiya Saiyin, Bingzheng Wei, Maode Lai, Jianzhong Shou, Yubo Fan, and Yan Xu. Zero-shot nuclei detection via visual-language pre-trained models. In MICCAI, pp. 693-703, 2023b.
Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detelip: Dietionary-enriched visual-concept paralleled pre-training for open- world detection. Advances in Neural Information Processing Systems, 35:9125-9138,, 2022.
Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Weidi Xie, and Yanfeng Wang. Knowledge-en visual-language pre-training chest radiology images. Nature Communications, 14(1):4542 2023.
Ge Zheng, Liu Songtao, Wang Feng, Li Zeming, Sun Jian, et al. Yolox: Exceeding yolo series in 2021. arXiv preprint 2100:22077.08400 2021.
13

---

Published as a conference paper at ICLR 2025
A MORE IMPORTANT ANALYSIS AND ABLATION.
Can category-specífic prompts be applied to all methods? We compare the performance of several methods when category-speciff prompts, generated by BLIP and GPT4, are used. Tab. pressenss the AP and AP@50 50 metrics across multiple datasets, including CVC300, ClinicDB, and Kvasir.
Table A: AP and AO@ Performance comparison of different methods when applied category-level prompt (BLIP- GPT4).
Method
CVC300 AP & AP@50
ClinieDB AP & 50@
Kvasir AP & AP@50
StructuralGIIP
63.9, 89.8
42.0, 57.0
42.0, 50.5
MIU-VL AutoPrompter
34.3, 53.2
17.9, 26.5
22.3, 39.4 17.5, 26.2
30,9, 37.4
15.8, 31.3
The results presented in Tab. A clearly demonstrate the superior performance of StructuralGLIP when applied with capegory-specific prompts, especially in terms of both AP and AP@ 50 metrics. StructuralGLIP achieves significantly higher detection performance across all datasets compared to MIU-VL and AutoPrompter. For instance, on the CVC300 dataset, StructuralGLIP reaches an AP@50 of 89.8, which is nearly 36% higher than MIU-VL and more than 50% higher than AutoPrompter.
Additionally, StructuralGLIP's performance on the ClinicDB and Kvasir datasets also outperforms the other methods by substantial margin. These findings suggest that StructuralLLIP is particularly effective adapting to dategory-specifie prompts and achieving high-quality results across a variety of medical image datasets. On the other hand, both MIU-VL and AutoPrompter show much lower performance, particularly in the AP@ 50 metric, indicating that their methods do not leverage category- specific prompts as effectively as StructuralGLIP. This underperformance highlights the robustness of StrueturalGLIP's approach in utilizing detailed prompts to enhance model accuracy.
In conclusion, these results further reinforce the importance of StructuralGLIP's capability utilize eategory-specifie prompts effectively. making it the method of choice for high-performance detection tasks in medical imaging.
Can the selection mechanism help against the noisy knowledge? In practice, we can not guarantee that the used prompt precise and clean. Therefore, how well method can perform on noisy prompt is very important. To evaluate the robustness of the mutual selection process, we conducted additional experiments on the BCCD dataset, which includes red blood cells, white blood cells, and platelets. In these experiments, we introduced noisy knowledge by mixing attributes from unrelated categories and tested the performance of StructuralGLIP. The results are shown in Tab. B In this experiment, "(X, X)" denotes using prompts solely for category X during detection of category X, while "(X, Y)" indicates the introduction of prompts from category when detecting category X.
Table B: StructuralGLIIP's AP@50 performance with noisy knowledge under zero-shot detection setup BCCD datasets.
Model
Red Blood Cells
White Blood Cells
StructuraIGLIP
Red Blood Cells
32.7
32.4
White Blood Cells
60.5
61.0
GLIP
Red Blood Cells
21.1
15.6
White Blood Cells
28.3
38.7
The results show that GLIP suffers significant performance degradation when noisy knowledge is high accuracy. This demonstrates the robustness of the
introduced, while StrueturalGI.IP maintains mutual-selection mechanism, which effectively filters out irrelevant information and selects the most relevant prompts for the task at hand.
How to assess the quality of the generated prompt? We evaluate the quality of prompts using CLIP-Score, which measures the cosine similarity between the embeddings of cropped regions (e.g., disease regions) and their corresponding prompts. This evaluation the prompts generated by different modal laniuaelanguage models (MLLMs), such as BLIP, I.La Va-7b, and DWw-2-VLL-7b, where the MLLMS are used as Vision Question Answering (VQA)
extended to compare
14

---

Published as a conference paper at ICLR 2025
The results are summarized in Tab. C prompts.
models.
comparing both instance-level and category-level
Table C: CLIP-Score and detection performance under zero-shot enhancement setting on TBX-
Model
BLIP
LLaVa-7b
Qwen2- VL-7b
Instance-level prompt's CLIP-Score
0.284
0.290
0.294
Instance-level AP @0
0.780
0.783
0.792
Category-level prompt's CLIP-Score
0.264
0.262
0.288
Category-level AP@
0.782
0.765
0.790
Table D: CLIP-Score and detection performance under zero-shot setting on CVC-300.
VQA Model Instance-level CLIP-Score Instance-level AP@50
BLIP
0.259
72.8
LLaVa
0.272
73,9
Qwen2-VL-7b
0.270
74.3
The results show clear positive correlation between CLIP-Score values and detection performance (AP@50). This validates the effectiveness of CLIP-Score reliable metric for evaluating the quality of prompts. The analysis underscores the utility of CLIP-Score as an additional evaluation metric for VQA-generated prompts.
Ablation on the choice of LLMs for Category Prompts. To assess the impact of different LLMs on performance, evaluated StructuraGGLIP using category-level prompts generated by GPT-4, VL-7a- and Qwen2-VL-7b. The results are summarized in Tab. comparing their performance multiple datasets.
Table E: Ablation the choice of LLMs when generating category prompts for zero-shot enhance- ment (AP@ 50).
LLM Model
TBX-1Ik
CVC-300
GPT-4
79.2
96.5
LLaVa-7b
76.5
73.9
Qwen2-VL-7b
79.0
89.0
The experimental results indicate that the choice of LLM significantly influences the final performance. In contrast, as shown in Tab. C the selection of different VQA models has relatively minimal impact on the quality of generated prompts. This is because the LLM-based prompt expansion process primarily relies the model's internal knowledge and memory of the appearance attributes related to the target lesion, rather than utilizing example images.
Therefore, when using LLMs for prompt expansion, selecting peeefoeeffrming models crucial to ensure the generation of reliable prompts. Additionally, for detecting rare diseases, prompts generated through LLM expansion may be applicable, as these models might lack sufficient domain knowledge of less common conditions. This experiment provides valuable insights into the scenarios where LLM-based prompt expansion can effectively utilized.
Ablation on the LLM model to generate prompts. The results in C shows that the perfor- mance of StrucutralGI.I can be largely affected under different choices L.L.M to generate prompt. Therefore, we use Qwen2-VL-7B generating, to generate instance-level prompts and compare the performance of StructuralGLIP with AutoPrompter. The results shown in Tab.
the VQA model, which has the best performance in prompt
The results in Tab. demonstrate that StructuralGLII consistently outperforms AutoPrompter in terms of both AP 50 and AP metrics across all datasets. The results in Tab. demonstratee that StructuralGLIP consistently outperforms AutoPrompter with different vQA models for instance-level prompt generation.
15

---

Published as a conference paper at ICLR 2025
Table F: AP and AP@50 performance of StructuralGI.IP and AutoPrompter with different VQA models for Instance-Level Prompts.
Colondb
Kvasir
Etis
Clinicdb
AutoPrompter AP@50
0.513
0.431
0.240
0.318
AutoPrompter AP
0.353
0.347
0.178
0.233
StructuralGLIP A 50
0.549
0.440
0.288
0.376
StructuralGLIP AP
0.373
0.359
0.193
0.291
These results confirm that the effectiveness of SturcturalGI.PP can be transferred well to different quality of prompts.
Table G: AP and AO@ Performance of StructuralGI.PP and AutoPrompter with different VQA
Model for Instance-Level Prompts.
Method
CVC-300 AP@50
ClinicDB AP@50
StructuralGIIP (BLIP)
72.8
38.2
StructuralGI.IP (LLaVa-7B)
73.9
37.9
StructuraGGLIP (Qwen2-VL-7B)
74.3
37.6
AutoPrompter (BLIP)
70.6
30.6
AutoPrompter (LLaVa-7B)
70.9
30.9
AutoPrompter (Qwen2-VL-7B)
75.0
31.8
B COMPARATIVE VISUALIZATION
We illustrate the detection results the ColonDB and BCCD dataset in Fig. E where we employ the vanilla GLIP for these prompt- based methods. Intuitively, both AutoPrompter and MIU-VL struggle with either over-detection missing critical targets. This is likely due to the coarse alignment between vision and target representations, leading to false positives and missed detections. For example, in ColonDB, both methods produce inconsistent bounding boxes, failing accurately localize the polyp. On the other hand, category-level prompts, leading fewer missed targets and improved confidence scores.
more precise localization with
We demonstrate example detection results on the ISIC2016 and TBX11K datasets in Fig. below, where we employ the fine-tuned GLIP and AutoPrompter for comparison. As shown in Fig. a), it is evident that vanilla GLIP and AutoPrompter fail to produce correct classification results for lesion detection. In contrast, our method, benefiting from the category-level prompt, makes corrects classification. For radiographic datasets, our instance method achieves higher confidence scores using the same prompts with AutoPrompter.
C DATASET INTRODUCTION
We select four types medical imaging datasets involving eight benchmarks:
1) Endoscopy datasets for polyp detection: ClinicDB Bernal et al. (2015), [ernández7 jet al. (2016), ColonDB Bernal et al. (2012), Kvasir Tha et al. (2020). ETIS Silva et al. (2144. There 2,248 images and 2,374 bboxes total. The complete training and validation images for the entire benchmark are 1 160 and 290, respectively. And the number of test set images for CVC-300, CVC-ClinicDS CVC-ColonDB, Kvasir, and ETIS datasets are 60, 62, 380, 100, and 196 respectively. The primary challenge involves highly variable polyp appearances, obscured views due to mucus and bleeding, and low contrast against surrounding tissues.
2) Microscopy dataset: BCCD shenggan et al. (001 for blood cell detection (white blood cells, red blood cells, and platelets). The BCCD dataset is designed for blood cell detection tasks, including
16

---

Published as a conference paper at ICLR 2025
ours (category)
AutoPrompter (instance)
MIU-VL (instance)
pump:0.55
mm
Dump:0.50
guspe 0.4
bump::0 42 pump:0..3 d
bump:0.51
00.p:00.0
bumbo0
bumpio 48 f
0.43
0.43
0.53
But3
0.41
Figure Visualization Results on the ColonDB and BCCD datasets. three classes: white blood cells, red blood cells, and platelets. There are 874 images with 11,789 bboxes for the entire BCCD dataset.
3) Photography dataset: ISIC-2016 for skin lesions detection (benign lesion, malignant lesion). The ISIC-16 16 dataset consists of 1,279 images with 1,282 bboxes for benign skin lesions and melanoma detection, divided into 720/1 80/379 images for training, validation, and testing. This dataset pose difficulties due to the small size and high density of the targets, and variations in staining which affect visual clarity and consistency.
4) Radiology image datasets: TBXI Liu al. (2023b) for tuberculosis detection in lung x-rays. These datasets are challenging due to the subtle nature of disease indicators, which can obscure key features. The TBXI dataset used for tuberculosis detection in the lung, including 799 images and 1,211 bbox labels. Moreover, this dataset is divided into 779/120/200 images for training, validation, and testing sets, respectively.
We demonstrate some example images of these datasets in Fig. below.
D TRAINING DETAILS FOR GLIP's ZERO-SHOT ENHANCEMENT EXPERIMENT
The zero-shot enhancement aims to further improve the performance of models after supervised training on the downstream datasets. We follow Qin et al. (202)) to use a fine-tuned GLIP model optimized with the Adam Oinimize & Ba 2014),, where the initial learning rate is set to 1 x 10-4 (1 x 10-5 for the BERT text encoder). weight decay of 0.05 is applied to prevent over- fitting, and the bottom two layers of the image encoder are frozen to preserve fundamental features. Our expressive prompts tailored to the characteristics of the target's appearance are generated using GPT-4 Achiam et al. (2023). Full details of these prompts available in the Appendix.
17

---

Published as a conference paper at ICLR 2025
Fec lok tuberculoa
GLIP
Ours(category)
Ground Truth
AutoPrompter (0.67) gray rectangle tuberculosis
benign lesion; malignant lesion
benign lesion Symmetrical, Regularshape, Well- defined borders, Smoothedges, Uniform color:melignant lesionnot Symmetrical, Regular shape, Well-defined borders, Smooth edges, Uniformcolor
Jubereal
gray peclongie
AutoPrompter brown circle benign lesion;
GLIP (0.66) tuberculosis
Ours (0.69)
circle malignant lesion
rectangle tuberculosis
(a)
(b)
Figure B: Visualization Resultson the ISIC2016 and TBX11K datasets.
E ANALYSIS FOR THE HIERARCHICAL CHARACTERISTIC OF THE STRUCTURAL REPRESENTATION.
To validate the hierarchical nature the structural representations derived through layer- prompt retrieval, we analyzed the distribution of selected prompts across six GLIP model encoder layers. Specifically, we conducted this analysis using StructuralGI.IP with category-level prompts on two medical detection tasks: BCCD red blood cell detection and ClinicDB polyp detection. Prompts were categorized into four types: color, location, shape, and texture (see detailed prompt categories in Appendix G The results, shown in Fig. D reveal distinet differences in the frequency of selected prompt types across layers (see Tab. H for the concrete value of the figure). In both tasks, color prompts were consistently selected across all layers, highlighting its importance in medical detection. For ClinicDB, the frequency of shape and texture prompts increased deeper layers, indicating that these features become more relevant the model abstracts more complex attributes. In contrast, for red blood cell detection, color remains the predominant feature across layers, while the selection of shape and texture prompts decreases. This analysis demonstrates that StructuralGLIP can dynamically retrieve rese-rele prompts from the knowledge bank at different layers, confirming the adaptable nature of our zero-shot medical detection framework.
Attribute
colors
locations
shapes
texture
Dataset
polyp red blood cell polyp red blood cell polyp red blood cell polyp red blood cell
layerl
1436
1192
183
233
770
474
467
233
layer2
1468
1073
473
17
690
562
572
149
layer3
1475
1109
158
122
702
499
223
136
layer4
1505
1160
436
133
1131
653
883
245
layer5
1498
1062
416
59
1163
565
779
269
layer6
1450
1117
182
21
904
433
452
48
Table H: The selected frequencies of different types of prompts across GLIP's different layers
18

---

Published as a conference paper at ICLR 2025
(a) endoscopy
(b) radiology
benign
benign
benign
melanoma
melanoma
(c) photography
red blood cell
platelets
whit blood cell
(d) microscopy
Figure C: Examples of medical images under different imaging conditions.
layer index
BCCD red blood cell detection
ClinicDB polyp detection
70
color
15
location
32
shape
25
texture
20
60
10
28
15
50
10
5
24
40
5
30
0 1234
20
0
12
5
6
1
5
123
5
Figure D: Results of the selected frequency (y-axis) of different types of prompts across the network's layer (x-axis).
F EFFECT OF PROMPT QUALITY.
As shown in Tab. we present the detailed target description (query name with prompt) for red blood cells detection task, and only using the category name "red blood cells" as input to the model of GLIP resulted in poor detection performance, with an AP of just 1.7%. From Tab. [ it is evident that the design prompt methods significantly affects the ability of vision-language models to utilize prompts for domain enhancement. For instance, our method, with the simple prompt, "pink oval", outperforms MIU-VL, which uses multiple types of attributes, achieving a +7.4%AP50 and +4.3%AP improvement. This improvement is attributed to structural representation, which achieves hierarchical vasion-language alignment, thereby enhancing the utilization of prompts for medical image analysis. Additionally, the structural representation involves fine-grained vision-
19

---

Published as a conference paper at ICLR 2025
language alignment, enabling precise selection of attribute tokens from prompts. This capability allows our method to effectively incorporate more comprehensive prompts, leading to a further improvement of 5.1%AP50 based on the simple short prompt. This demonstrates the effectiveness and robustness of our approach in generating and utilizing prompts for zero-shot detection tasks, showcasing its superiority in achieving efficient and accurate medical image analysis.
Table I: Comparison of different methods with different prompts based the red blood detection task of BCCD dataset (complete prompts are shown in Appendix G
Methods
Target Description pname+prompt)
AP
AP50
GLIP
[name]] red blood cells
1.7 4.3
MIU-VL
[name] + red color + spherical shape in birth
12.0
24.7
AutoPrompter
[name] + pink oval
12.6 27.0
Ours
[name] + pink oval
16.3 34.4
[name] + (four prompts below) [flesh-colored, pink, round, blood]
MPT+Cluste
12.5 25.6
[name]+ [color] pale bright, et al.+ [shape] oval round, et al. Ours (category) [texture] smooth rough al.+ [location] peripheral central, et al.
19.3 39.5
G DETAILED CATEGORY-PROMPT FOR THE MEDICAL DATASETS
Colors
Shapes
Textures
Locations
Polyp
white, yellow, orange, red, brown, pink, pale, tan, ghit- gold, cream, ruby. turquoise, indigo, violet
octagon, circle, round, heart, oblong, oval, small, rounded. jagged, wide, large. bulbous, spherical, circular, irregular, diamond
smooth, textured, cracked, striped, shiny, dull, speckled, raised, rough, gran- ular, grooved, glossy, veined, pigmented, uneven, mottled, interwoven, lines, patches, complex, reticular, structure smooth, rough, grun- ular, fibrous, glossy, matte, sticky, velvety, spongy, creased, crys- talline, jelly-like, pit- ted, wrinkled, spiny, bumpy, flaky, mucous, papillary, striated
rectal, mucosal, el- evated, demarcated, creased, folded, iso- lated, clustered, soli- lary, honeycombed
Red Blood
pale, bright red, dark red, pinkish. crimson, ruby, coral, salmon, cherry, scarlet, rusty, maroon, wine, bur- gundy, rosy, flamingo, peach, copper, ma- hogany, terracotta
disc-shuped, oval, round, elongated. spherical. like, bean-shaped, crescent, irregular. biconcave, elliptical. cuboidal, gular, squamous. fusiform. polygonal. rod-shaped, fibrillar. amorphous, lobed round, oval, irregular. lobed, segmented, spherical, kidney- shaped, amoeboid, polymorphous, triangular. elon- gated, bean-shaped. cuboidal, crescent. spindle-shaped, fusiform, irregularly- shaped, star-shaped, flattened, discoid
peripheral, central, upper. lower, me- dial, lateral, distal, proximal, anterior, posterior, cervical, thoracic, abdominal, pelvic, inguinal, axillary, oral, nasal, occipital parietal
Cells
ring-
trian-
White Blood Cells
purple, white, pink, gray, blue, translu- cent, lavender, milky, yellow, pale, clear, light purple, ivory, cream, faint blue, sil- ver, off-white, light gray, opalescent
granular. rough, smooth, wrinkled, spongy, matte, glossy, fibrous, pitted, veined, speckled, lobulated, ridged, reticular. grooved, folded, striated, flaky, nodular, uneven
circulating, ripheral, abdominal, pelvic, cervical, axillary. lymphatic, spleen, marrow, mediastinal,
pe- thoracic,
raised,
proximal, distal, inguinal, occipital, parietal, cranial, vertebral, lumbar.
sacral
20

---

Published as a conference paper at ICLR 2025
Platelets
yellow, gray, pink.
small, round, oval, ir- regular, disc-shaped, spiked, star-shaped, elongated, granular. fragmented, jagged, ring-shaped, crescent, cuboidal, polygonal. fibrillar, amorphous. fusiform. spherical. irregularly-shape
smooth, rough, spongy, fibrous. wrinkled, glossy, veined, lobu- lated, striated, flaky, nodular. reticular, ridged, bumpy, raised, speckied. uneven, lumpy smooth, glossy, matte, uniform, fine, clear, unbroken, even, pol- ished, soft, thin, flat, reticular, striated, nodular. shallow, gramular, homoge- neous, light-textured, delicate
granular.
circulating. periph- eral, marrow, spleen, liver, thoracic, ab- dominal, lymphatic, distal, proximal, cer- vical, axillary, cra- nial, vertebral, sacral, pelvic, mediastinal, inguinal, parietal, oc- cipital
translucent, clear,
beige, orange, white,
pitted, matte,
pale yellow, light gray, light pink, golden, amber, straw; ivory, light orange. peach, tan. light brown, opalescent
Benign Lesion
light brown, tan, pale pink, beige, ivory, light yellow, flesh-colored, clear, translucent, white, pink, light red, off- white, cream, soft yellow, gray, peach, faint brown, faint yellow, light orange
round, oval, smooth- edged, well-defined, regular. flat, slightly raised, small, lobu- lated, dome-shaped, circular, symmetticcal uniform, elongated. flat-topped, irregu- lar, semi-spherical. oblong, disc-shaped. heart-shaped
superficial, epi- dermal,
dermal, non-invasive, iso- lated, peripheral, central, facial, limb, torso, scalp, back, upper, lower, anterior posterior, lateral, abdominal, neck, arm
Malig- nant Lesion
dark brown, black, red, purple, blue, gray, deep red, ma- roon, dark purple, crimson, burgundy, dark gray. navy, violet, yellowish, pale gray. dark pink, reddish-brown, orange, lan
irregular, asymmetric. poorly-defined, multi- lobed, jagged, raised, ulcerated, irregular- edged, large, deep. multi-colored, nodu- lar. star-shaped. rough-edged, uneven. angular, oblong,
rough, scaly. gran- ular, ulcerated, cracked, irregular, firm, thick, pitted, fibrous, crusty, glossy, uneven, speckled, reticular, indurated, papillary, pigmented, veined rough, distorted, fragmented
invasive. dermal, subcutaneous, nodal, systemic, spread, clustered, axial, limb, facial, scalp, back, chest, abdominal, upper, lower, lateral, posterior, anterior, proximal, distal
bumpy,
Tuber- culosis
white, gray, patchy, cloudy, translucent, opaque, pale, faint, bright, shadowed. dull, smoly, hazy. diffused, misty, dense, light gray, speckled, milky, gray-white
irregular, nodular, patchy, lobular, diffuse, multi-focal, rounded, asymmet- rical, large. small. streaked, segmented. thickened, elon- gated, fragmented, scattered, spotty. uneven, consolidated, granular
rough, fibrotic, granular, nodular, scarred, thick, tex- tured, coarse, uneven, reticular. banded, streaked, fibrous, pit- ted, grooved, layered, striated, indurated, dense, veined
apical, upper lobe, lower lobe, central, peripheral, posterior anterior, lateral, mediastinal. pleu- ral, diaphragmatic, trucheal, hilum, bronchiolar thoracic, cervical, upper, lower, rib, clavicle
H ANALYSIS TOWARDS THE FEATURE DISTRIBUTION OF STRUCTURALGLIP.
To provide more insight into the improvement brought by StructuralGLIP we focus vision and language features input into the RPN detection model before and after applying our proposed approach. We conduct experiments on five datasets (cvc300, colondb, clinicdb, kvasir, etis) on the vanilla GLIP without fine-tuning and calculate the average value vision and target representation's attention matrix (termed "average attention strength"). The prompt we used is category-level prompt. Then, we employ a kernel density estimation method to estimate the distribution of average attention strength. We found that there is significant increase in average attention strength for the proposed StructuralGLIP compared with the vanilla GLIP, indicating better alignment between vision and language representations.
21

---

Published as a conference paper at ICLR 2025
Layer1
Layer2
Layer3
Layer4
Layer5
Layer6
Figure E: Feature distribution of the average attention strength using KDE estimation five datasets (cvc300, colondb, clinicdb, kvasir etis). The x-axis is the value and the y-axis is the density.
I VISUALIZATION ON NATURAL IMAGES
Here we demonstrate the visualization of the proposed StructuralGLIP and GLIP on natural images. The prompt for each category is follows:
1. Cosmos:
broad, delicate, slightly ruffled petals radiating symmetrically around vibrant yellow center, with a lightweight and airy appearance that contrasts beautifully against the surrounding colors
2. Peony:
lush, voluminous, soft, delicate, vibrant, radiant, layered, rounded, ruffled, full, graceful, elegant, eye-catching, rich, luxurious, intricate, symmetrical, silky, and captivating
3. Hematite: flat and irregularly shaped, with coarse and slightly grainy surface indicative of its iron-rich composition
22

---

Published as a conference paper at ICLR 2025
Peony
Cosmos
Hematite
nel
Nematite:0.55
StructuralGLip
0.55
0.60
hematit dem hematitaemaiiee aigtes
GLIP
Figure F: Examples of natural images with GLIP and StrUcturalGLIP..
23


Reviews:
[{"review_id": "TUsPWtsU3a", "reviewer": "Reviewer_cerF", "summary": "In this study, the authors introduce StructuralGLIP, a novel approach to zero-shot medical detection using vision-language models (VLMs). This method leverages structured representations within a dual-branch architecture that enables nuanced alignment between images and textual prompts, significantly enhancing the model's adaptability to new medical scenarios without needing annotated data. StructuralGLIP uses category-level prompts, maintained in a latent knowledge bank, and a mutual selection mechanism for precise cross-modal fusion, thus improving accuracy across diverse medical imaging datasets.", "questions": "(1)\tTo what extent can the findings be generalized to other medical imaging modalities or less common diseases? Are there plans to evaluate the model on broader datasets?\n(2)\tBesides Average Precision, what other metrics were considered for evaluating model performance? Are there plans to incorporate user feedback or clinical outcomes in future evaluations?\n(3)\tThis paper focuses on zero-shot medical detection, whereas GLIP was initially developed for natural images. Can the proposed method also be applied effectively to natural image datasets?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "(1)\tThe paper introduces an effective structural representation by encoding prompts into a knowledge bank and utilizing a dual-branch structure. This approach enables adaptive and context-aware alignment, which is particularly advantageous for complex medical detection tasks.\n(2)\tStructuralGLIP outperforms traditional zero-shot models by effectively handling both instance-level and category-level prompts, achieving significant improvements across various benchmarks in endoscopy, microscopy, radiology, and more.\n(3)\tBy allowing for zero-shot enhancement, the model can be fine-tuned and then further improved with category-level prompts, a feature well-suited for dynamic medical settings where data annotation is scarce.", "weaknesses": "(1)\tThe proposed dual-branch structure with a knowledge bank requires complex engineering and computational resources, potentially limiting its accessibility for practitioners in less resource-rich environments.\n(2)\tThe paper may not adequately address the potential data imbalance present in the datasets used for evaluation. Some diseases or conditions may have significantly fewer examples, which could impact the model's performance and generalizability.\n(3)\tThe model's inner workings, particularly regarding how it selects and utilizes prompts, may be difficult for practitioners to interpret, limiting trust in its decisions and making it harder to diagnose potential failures.\n(4)\tDespite improvements in alignment, there may still be instances of misalignment between visual features and prompts, especially in cases of atypical presentations, which could lead to missed detections.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DFJ1PNhrUe", "reviewer": "Reviewer_wdRS", "summary": "This paper introduce a new zero-shot method for medical Vision-Language Models(VLMs) for detecting unknown targets. To address the prompt umatched with the variations in the medcial images, the authors propose a StructualGLIP desgin with a main and an auxiliary branch encoders for text input and introduce a mutual selcetion mechanism. \nThe author explain that the auxiliary branch would work as a knowledge bank where the main branch can extract latent prompt tokens, while the tokens in the knowledge banked are filtered by the mutual selection process.\nOverall, the motivation, method, and performance of this work is good enough, but I still need some explaination for some detail, please refer to the weakness and question section. I will consider adjust my rating based on the authors response.", "questions": "1. How to evaluate the quality of the generated prompts by VQA method. As some previous work pointed out, the VLM without medical domain adaptation perform poorly on some medical datasets, especially for radiology datasets.\n2. For Prompt as Knowledge Bank ablation study seciton, I would like to see an experiment on the whether adding noisy knowledge (for example, knowledge for another target) would sharply downgrade the StructureGLIP performance. This experiment would the the robustness of the mutual-selection process.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work is aiming for a vital issue in medical image understanding field, which is the generalization capability of foundation models with limit data access. The motivation of improving the existing work is clear and strong, which is the lack of object level prompt and fail to capture the various feature of images during prompt desgin.\n2. This work present a novel but efficient method, called StructualGLIP, to increase the model's zero-shot/few-shot performance on various datasets. The desgin of StructualGLIP introduce the knowledge bank and mutual selection process to help prompt design process. This method address several shortcomings of current method and is novela and effective.\n3. This method largely increased the zero-shot performance on different medical image datasets across different modalities.", "weaknesses": "1. Line 234, the sentence seems not finished. \n2. One of the major problem of the proposed method is not trainable as the Top-K selection operation is non-differentiable, while previous work is differentiable and thus finetuning would result in better performance. I would suggest include Reparameterization Trick for Gumbel-Softmax to improve your method. Though this work is good enough as a stand-alone method for zero-shot detection. But I see a potential to achieve better performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Js2jizqooM", "reviewer": "Reviewer_RBUs", "summary": "This paper presents Prompt as Knowledge Bank, a method that encodes prompts into an adjustable knowledge bank, enhancing multimodal models’ accuracy in zero shot tasks by dynamically selecting prompts from prior knowledge. This approach achieves robust performance in high-precision fields like medical imaging, even with limited data.", "questions": "1. How does the prompt generation module ensure the creation of valuable prompts for unseen classes without introducing distracting information?\n2. How is the accuracy of instance-level contextual prompts generated from visual input for VQA ensured, as this seems crucial to the final experimental performance?\n3. How does using alternative LLMs affect the experimental results?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. By introducing a dynamic knowledge bank that selects the most relevant prompts, the model achieves more flexible and accurate vision-language alignment.\n2. This method performs exceptionally well in unsupervised and few-shot scenarios, maintaining high detection accuracy even with limited labeled data.\n3. The prompt generation module extracts information from prior knowledge of unseen classes, enhancing the model’s adaptability and making it suitable for tasks across various domains.", "weaknesses": "1. The introduction of the knowledge bank and prompt generation module increases computational costs, raising demands on hardware resources.\n2. The model heavily relies on prompt quality, and low-quality prompts may negatively impact its performance.\n3. The impact of the prompt’s LLM on performance is apparent; however, the paper does not analyze how the choice of LLM (e.g., LLaMA, Gemini) affects the results.\n4. The complex prompt selection and knowledge bank structure reduce the transparency of the model's decision-making process, posing challenges for clinical applications.\n5. There are formatting errors in the paper, such as an unexpected horizontal line near the number 15 in the table 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DQgkN2SMzh", "reviewer": "Reviewer_Ky5o", "summary": "This paper introduces a zero-shot object detection model, StructrualGLIP, for medical image-based object detection tasks. The method uses a knowledge bank that stores encoded text prompts, which are later used to select and match relevant image features to form fine-grained structural representations to allow better alignment of image features with the prompt information, achieving accurate, context-aware detection.", "questions": "Please address Weaknesses #1,5,6.\n\n\n------------------\n04/12/2024:\nI thank the authors' detailed responses. Since the reviewer's post deadline has passed so I am writing my final comments here. \n\nMy conclusion is that I will retain my original score. I believe the paper should improve clarity as many new arguments and technical details surfaced during the discussion period which should have been included in the main manuscript. The novelty and settings should be explored in more depth.\n\nMy comments are:\n\n1. Novelty. \n\nI agree with the authors' argument the image and language encoders do not need to be fine-tuned in your zero-shot detection setting. This I already mentioned in my last comment: \"I can understand if the auxiliary branch's language encoder is untrained\". My question was whether the modules in Eq. 8 to 10 need training or not? Especially the MHA module in Eq. 8, GLIP has the X-MHA module, did the author reuse the X-MHA weights for the MHA without retraining? The authors mentioned in L263 \"we employ a multi-head attention (MHA) mechanism\" which reads like it is a new module by the authors' own design. Given an MHA module (as far as I know) should have learnable weights but the authors claim the zero-shot detection setting is training-free, then I find the logic here is contradicted. Furthermore, the authors quoted L210 to demonstrate the training-free paradigm, but that is immediately after Eq.4 and the sentence only describes the encoders. This is an imprecise response to my question so I'm not making a judgment here, but this does make me think the paper should improve clarity. \n\nThe above was not the main point of my original novelty concern, but a spin-off. My original point was that if the authors wanted to demonstrate the \"preventing/addressing domain shifting\" argument for the novelty, you should consider measuring the actual feature distribution difference between GLIP and StructureGLIP.  The authors revealed in the last comments that the domain referred to the features before RPN, and that is what I think you should demonstrate. To recap the authors' comments, the authors claimed the other methods would concatenate prompt so the feature to RPN becomes CLS token of [target_name, prompt] which causes the domain shift, whereas StructualGLIP did the prompts integration in early layers so StructuralGLIP only has CLS of [target_name] to the RPN, and that addresses the domain shift. Conceptually, maybe it is true but please provide empirical evidence. The final performance is a surrogate measure, which does not directly support your domain shift argument. Finally, StructrualGLIP also fuses additional information in the process, so the CLS token should exhibit some domain shift as you are trying to do domain adaptation, these two arguments also have contradictions to each other. \n\nRegarding the \"addressing noise prompt\" argument, I'd imagine the top-P/Q has a certain capability of limiting noise prompts as you only choose the top tokens, so even when you add more prompts, they won't be selected if the network was already trained to have higher attention for the target classes tokens, but that on the other hand, make it trivial argument rather than a novelty.\n\n7. Zero-shot enhancement. After reading the authors' most recent comments, I strongly feel the setting should not be called zero-shot. When the target dataset's labelled data are already used to fine-tune your GLIP base model, then this is weakly supervised. In L175-176, the authors state \"We propose a zero-shot enhancement setting. This involves fine-tuning the model on medical datasets first, and then using prompts to further improve performance on unseen medical images\", initially I considered this as the model was trained on relevant datasets such as same/similar imaging modalities but different tasks or different datasets. However, if the target dataset itself was used for finetuning for training, then the \"further improve performance on unseen medical images\" does not hold.\n\nOther comments:\nThe experiments I requested were for the understanding of how much performance gain was attributable to the design/novelty of StructrualGLIP, i.e., the introduction of the auxiliary branch. The zero-shot enhancement setting (i.e., fine-tuned GLIP) has boosted the performance by a larger margin than the StructualGLIP can bring to GLIP. Also, the choice of the VQA can sometimes affect the AP/AP50 by a noticeable margin (i.e., that 75 AP) but it appears not stable. Then, at least for me, the viability of zero-shot medical detection unfortunately remains questionable.\n\nOnce again I appreciate the authors' effort to address my comments but I'm sorry I will retain my original score based on my thoughts above. The authors are encouraged to delve into the details of the novelty and concepts as well as improve clarity of the manuscript.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. the paper demonstrates a comprehensive evaluation of four medical image modalities.\n2. the paper's representation and structure are mostly clear and easy to follow (although with some language and word choice issues, which will be discussed below).", "weaknesses": "1. The claimed novelty appears to be on the latent knowledge bank and its function to store encoded prompt tokens and later be used at each encoder layer as a vision token selector and vice versa. This is coined as a mutual selection process. The selected information is then merged into the original image and text encoder back at each layer. The entire process is different to a standard contextual prompt method (Fig. 1(a)) but it feels like a quite incremental difference which I don't see much novelty.\n2. The word choice of \"medical detection\" bugs me, in medical science detection refers to \"detecting diseases\" whereas here this is object detection in medical images so this may cause confusion to certain readers.\n3. Line 234 looks unfinished.\n4. Eq. 6&7, perhaps the Top-P/Q^{max} function can be simplified by using argmax/argsort function?\n5. L201/534: \"... fine-grained alignment between target descriptions and medical images\", L:256 \"forming fine-grained structural representations\", can the authors clarify what \"fine-grained\" refers to in those places?\n6. L294: \"...like BLIP Li et al. (2022a)\", is the VQA model BLIP or not? Have you considered other VQA models and would the performance of other VQA models fluctuate your detection results?\n7. L177: I don't entirely agree that zero-shot detection has a real-world clinical need as the clinicians I've encountered would not trust zero-shot settings, in the medical domain, accurate detection/segmentation/diagnosis is the the most important thing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)