DETR-ViP: Detection Transformer with Robust Discriminative Visual Prompts
Authors: Bo Qian
Dahu Shi
Xing Wei
Venue: ICLR 2026 Poster (2026)
Keywords: object detection
prompt-based detection
open-set object detection
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: This paper presents the DETR-ViP framework, which enhances visual prompt detection by improving the semantic consistency of visual prompts and introducing a selective fusion strategy.

Abstract: Visual prompted object detection enables interactive and flexible definition of target categories, thereby facilitating open-vocabulary detection. Since visual prompts are derived directly from image features, they often outperform text prompts in recognizing rare categories. Nevertheless, research on visual prompted detection has been largely overlooked, and it is typically treated as a byproduct of training text prompted detectors, which hinders its development. To fully unlock the potential of visual-prompted detection, we investigate the reasons why its performance is suboptimal and reveal that the underlying issue lies in the absence of global discriminability in visual prompts. Motivated by these observations, we propose DETR-ViP, a robust object detection framework that yields class-distinguishable visual prompts. On top of basic image-text contrastive learning, DETR-ViP incorporates global prompt integration and visual-textual prompt relation distillation to learn more discriminative prompt representations. In addition, DETR-ViP employs a selective fusion strategy that ensures stable and robust detection. Extensive experiments on COCO, LVIS, ODinW, and Roboflow100 demonstrate that DETR-ViP achieves substantially higher performance in visual prompt detection compared to other state-of-the-art counterparts. A series of ablation studies and analyses further validate the effectiveness of the proposed improvements and shed light on the underlying reasons for the enhanced detection capability of visual prompts.

Published as a conference paper at ICLR 2026
DETR-VIP. DETECTION TRANSFORMER WITH ROBUST DISCRIMINATIVE VISUAL PROMPTS
Bo Qian 1.
Dahu Shi2.3
Xing Wei
1 School of Software Engineering, Xi'an Jiaotong University CCAL, Zhejiang University 3 Hikrobot Co., Ltd.
ABSTRACT
Visual prompted object detection enables interactive and flexible definition of target categories, thereby facilitating open-vocabulary detection. Since visual prompts are derived directly from image features, they often outperform text prompts recognizing rare categories. Nevertheless, research on visual prompted detection has been largely overlooked, and typically treated as byproduct of training text prompted detectors, which hinders its development. To fully unlock the poten- tial of risual-prompted detection, we investigate the reasons why its performance is suboptimal and reveal that the underlying issue lies in the absence of global discriminability in visual prompts. Motivated by these observations, we propose DETR- -VI, robust object detection framework that yields class- diasssisttinuushablle visual prompts. On top of basic image-text contrastive learning, DETR- ViP incor- porates global prompt integration and visual-textual prompt relation distillation learn more discriminative prompt representations. In addition, DETR- ViP employs selective fusion strategy that ensures stable and robust detection. Extensive exper- iments on COCO, LVIS, ODinW, and Roboflow 100 demonstrate that DETR-ViP achieves substantially higher performance visual prompt detection compared to other state-of-the-ar counterparts. A series of ablation studies and analyses further validate the effectiveness of the proposed improvements and shed light on the underlying reasons for the enhanced detection capability of visual prompts.
INTRODUCTION
Compared with traditional closed-set object detection( Girshick (2015): Redmon al. (2016): Zhang ct al. (2022a)), open-set object detection breaks through the limitation of predefined categories, thereby demonstrating greater potential in real-world applications. Prompt-based object detection has further inspired the development of open-set detection by enabling flexible identification of target objects. Several prominent approaches( [Giias al. (2022); ; Gu et al. (202T) Kamath et al. (202T): :
Li et al. (2022b): Liu et al. (2024); Minderer et al. (2022 Zhou et al. (2022) support promppr
open-set detection by distilling knowledge from langunee models such as CLIP Radford et al. (202T) or from language models like BERT( DDvlin et al. (2019) through aligning visual representations with textual descriptions. The training paradigm of rext-prompted model centers aligning visual and textual feature spaces, with zero-shor shot generalization largely attributed to large-scale, generic image-text pairs. However, in specialized domains, relying solely on textual descriptions of target categories attributes often proves insufficient for reliable detection.
Some methods( Jiang et al. (2023); Kirillov et al. (2023): ; Zou et al. (023)); Ravi et al.] (2024);
Li et al. (2024)) employ visual prompts to define target objects of interest, further enhancing the interactivity and adaptability of object detection models. Recently, several methods( [Jiang et al. 20024 Cheng al. (2024) have begun to explore object detection frameworks that support both textual and visual prompts, achieving promising progress. [Jiag et al. (2024) empirically find that, compared with text prompts, visual prompts are more effective in recognizing rare categories. This is
WWok done during the internship at Hikrobot Co., Ltd. Projecc page: http:://gihuu..ommMMMMXXUU
Corresponding author.

---

Published as a conference paper at ICLR 2026
1.6
40
Similariit
50000
intrr: miniiaa
1.4
38
25
25000
20
20-
20000
11SR
1.2
1.5
15000
1.0
5000
Frc.
0.8
VODINO
+Align
GGoohal.
+Distill
14
-10 -30
40
0.0
26
64
Similarity Range
0.4
0.
08
100
(a) I-SNE Visualization
(b) Similarity Distribution
(c) IISR vs. MAP
Figure I: Analysis of visual prompts. (a) t-SNE visualization of VIS- GDSNO prompts sampled from 10 COCO categories. (b) Similarity distribution between VIS-GDINO prompts of the same category and across different categories. (c) Trends of Intra-Inter Similarity Ratio (IISR) and mAP. expected because visual prompts, being sampled from the visual domain, naturally compatible with image features, thus possessing stronger generalization ability than text prompts, which require cross-modal (text-to-vision) alignment. Nevertheless, visual prompts still underperform text prompts overall, which limits their practical applicability.
To systematically investigate the reasons behind this suboptimal performance and enhance detection capability, we build baseline model incorporating visual prompts on top of Grounding DINO, termed VIS-GDINO, and conduct a detailed analysis. VIS-GDINO achieves mAP scores of 21.1 and 17.2 on COCO and LVIS, respectively. which are lower than those of Grounding DINO with text prompts. For each category, we sample 128 images to extract visual prompts, and perform a t-SNE analysis on all visual prompts, as shown in Fa) It can be observed that the visual prompts of VIS-GDINO lack clear semantic boundaries, which may contribute to frequent misclassifications when using visual prompts for categorization. To illustrate this more clearly, we compute the pairwise cosine similarities among all visual prompts and plot the distributions of similarities for same-category and lifferent-categone prompts as histograms, shown in Figure (b)) Same-category and different- category prompts share a relatively wide similarity range, which underscores the disorganization of the prompt embedding space. Taken together with these observations,
attribute the suboptimal performance of visual prompts to two factors: (I) visual prompts from different instances of the same category exhibit large variance; and (2) visual prompts from different categories are heavily entangled in the global embedding space, making them difficult to distinguish. In summary, visual prompts suffer from a lack of sufficient semantic discriminability.
Building upon the above insight, we improve VIS-GDINO into a detection model with more robust and discriminative visual prompts, termed DETR- ViP, by simultaneously reducing the intra-class variance and enlarging the inter-class distance of visual prompts. DETR-ViL ViP introduces a global prompt integration strategy that simulates cross-image prompt detection during training while augmenting negative samples, thereby encouraging image features to encode more distinctive global semantics. In addition, we design visual-texi prompt relation distillation loss. This loss transfers the semantic priors from the similarity matrix of textual prompts to the visual prompts, encouraging them to capture semantic proximity. Finally, we propose a selective fusion strategy. The fusion of each visual prompt is conditioned on the model's judgment of whether the corresponding category instance exists in the target image, resulting in more stable and robust detection. To quantitatively evaluate the quality of visual prompts by measuring intra-class variance and inter-class separability, we further introduce the Intra-Inter Similarity Ratio (IIR). A larger IISR indicates stronger semantic consistency of visual prompts. As shown in Figure Ie). both IISR and mAP on COCO consistently improve with the proposed modifications, validating that the performance gains stem from the optimization of visual prompt distributions.
fusion
We conduct comprehensive evaluations on COCO Lin et al. (2014)) LVIS( Gupta et al. (019)) )
ODinW al. (002a)), and Roboflow 100( Ciaglia et al. (2022)) Under comparable backbone settings, DETR- ViP consistently outperforms T-Rex2. Specifically, DETR-ViP-1 outperforms T- Rex2-T by +4.4 MAP on COCO, and achieves +3.7 mAp over T-Rex2-T and +6.9 mAP over YOLOE-v8-L on LVIS. In terms of AP, and AP,, our model surpasses T-Rex2 by +9.4 and +5.2, and exceeds YOLOE-V8-L by +8.7 and +1.9, respectively. On ODinW and Roboflow 100, which contain category distributions substantially different from the training datasets, DETR-ViD -VP-- surpasses T-Rex2-L by 3.4 and 5.1 mAP, respectively. DETR-ViP-L further improves upon DETR- ViP-T, surpassing T-Rex2-L by 3.7 and 1.5 mAP on COCO and LVIS, respectively. Beyond fair comparisons
2

---

Published as a conference paper at ICLR 2026
with existing methods, also conducted extensive ablation studies to validate the effectiveness of each modification from VIS-GDINO to DETR-ViP. Moreover, both the IISR metric and t-SNE visualizations of visual prompts provide intuitive evidence that these improvements progressively optimize the structure of the prompt embedding space.
2 RELATED WORK
Prompt-based object detection provides a promising pathway toward open- vocabulary detection. Text- prompted approaches( Gu al. (202T): Kamath al. (2021): Li et al. (2022b); Liu al. (2024);
Minderer et al. (2022); Zang et al. 2022); Zhang et al. (2023); Zhong et al. (2022) have achieved remarkable progress, exhibiting strong zero- zero-shot aew shot recognition capabilities. These methods mainly align visual features with text embeddings, often leveraging pretrained language models such as CLIP
[Radford et al. (202Th) or BERT( DDvlin al. (019)). Representative advances include GLIP (022b)), which unifies detection and phrase grounding through large-scale image-text pretraining; DeIC Yao et al. (2022)), which introduces richly descriptive concepts; Grounding DINO Liu et al. (2024)), which enhances early cross moosl fusion; and Region ZZhong et al. (2022)
which transfers regional knowledge via pseudo boxes improve generalization.
However, in many scenarios language alone is insufficient to precisely describe target objects, motivating research on prooale detection for greater flexibility and contextual awareness.
Early methods( Minderer et al. (2022); ; Xu et al. (2023); Zang et al. (2022); Jiang et al. (202))
Ravi Li et al. )2024) explored more compact formats such keypoints or bounding boxes. DINOv(
adopted full reference images as prompts, while later works( Kirillov et al. (2023); Ravi et al. (2024);
ct al. (2024)) treats visual prompts as contextual exemplars for open- open- neen-vocbbulary detection, and T-Rxx (Rix(Jiangee al. (2023) applies deformable attention extract region-level features from point- box-based prompts. Recent methods combine visual and textual prompts: T-Rex2( [iing et al. (2024)
incorporates usioo-llnguage contrastive learning, and YOLOE Cheng et al. (2024) introduces RepRTA and SAVPE for unified prompt handling. Despite these advances, the performance of visual-prompted detection still lags behind that of text-prompted counterparts.
METHOD
3.1 PRELIMINARY
A closed-set detector computes the score of each proposal for predefined categories by training a linear layer, whereas an prompt-based open-set detector derives the score by measuring the similarity between the proposal features O and the prompt embeddings P, as formulated in Equation O
Score TOW Closed-set
+ b)
Score o b).
(I)
Prompt-based
Here, W denotes the weights of the linear layer, is a learnable bias, and a represents the sigmoid function. In this work, we adopt the prompt-based method shown in Equation (1) (right), where neither O nor P is subjected to L.2 normalization.
Given an image-text pair TI, Grounding-DINOO Liu et al. (024 extracts multi-scale visual features X, E R using a visual backbone (e.g., Swin Transtormer( Liu et al. (2021) ()), and obtains textual prompt embeddings Pr NR. through a text backbone (e.g.. BERT Devlin et al. (2019)). Subsequently, X₃ and Pr are fused and enhanced within the encoder, whose layers consist of deformable attention module (for enhancing Xy). a self-attention module (for enhancing Pr), and a bidirectional attention module (for fusing X; and Pr.). as shown in Equation (2). Similar modules are also present the decoder. Given K user-specified normalized boxes by (Ziz Uý. W, hy ). j 21,2,.., K)." T-Rex( Jiang et al. (2023) ) first extracts multi-scale features X using visual backbone and encoder. Each box is transformed into a positional embedding via a sine-cosine encoding and projected to a uniform dimension to obtain box embeddings B. A learnable content embedding C' is replicated K times and concatenated with B, together with a global box B' = {0.5. 0.5, 1, and a class token C' for aggregating visual prompt features. The resulting embeddings are linearly mapped to form the visual prompt queries Q, which are then fed into a multi-scale deformable cross-attention module to extract visual prompts Py from X,. The overall process is summarized in Equation (3).
3

---

Published as a conference paper at ICLR 2026
fus Spus
FIN
Cross Alli
Middality
Image Fonee
f
Laver
CreodAin
Selectio
IBV Con
Visua Prorpes
Self-Atn
Feature Eshaneer
Q.K.V Query
D
Pronpe
Epsss
Person
Haandiag
Integration
Prospt
p
Beckibone
Encoder
Pronuet
Penson.
Arerge Stock
Pronge
Viaud
Emberskings
Text
Text Reheirn
Knowledas
Viszal Rdaiio
Figure 2: The overview of DETR-ViL ViP. DETR-VIL builds on Grounding DINO by incorporating visual prompt encoder for prompted detection. It improves prompt semantics via global prompt Integration and visual-textual prompt relation distillation, and refines the fusion module to stabilize image-prompt interactions, thereby enhancing detection robustness.
X MSDeformSelfAtmm(X.
B Linear Linear(PE((), ...... bx))
Pr = SelfAun(Ps X,, PT BiAtt(X,,
Q Linear(CAT((CC: C*), [B; B())
(3)
Pv MSDeformAttn(2 (Q. X1)
They both adopt the same detection loss DINO Zhang et al. (2022a), which consists of classifi- cation loss Lets and a regression loss composed of an L1 loss L1 and a GloU loss LawoU, as well as denoising loss Lan- T-Rex2 further employs image-text contrastive loss LAiign We also adopt these losses, which we denote as Lrase:
Drsse =
Adthels + X1 + AGIOU + AanCan + >Align CAlign
Here, X denotes the weighting coefficients for each loss term.
3.2 DETR-ViP
We develop the baseline VIS-GDINO from Grounding DINO by inserting the visual prompt encoder, as defined in Equation 3 between the backbone and the encoder, and removing the fusion modules in the encoder and decoder represented in Equation 2 On top of this architecture, we introduce the global prompt integration, visual-textun prompt relation distillation loss, and selective fusion strategy enhance visual prompt detection, thereby upgrading VIS- DETR- ViP, as shown in Figure]
Global Prompt Integration. DETR-like models typically optimize confidence scores using Focal Loss Lin et al. (017)) to endow the model with classification capability. In prompt- based detection, given proposal q, suppose it is matched to the category represented by the prompt pit, while denotes the prompts from other categories. The classification loss contributed by this proposal is
formulated follows:
q log p+ + - log(1 p-)
(5)
positive itom
negative items
The positive term attracts proposal features toward positive prompt embeddings, while the negative term repels them, akin to contrastive learning. Prior work (Chen et al. (2020); [e et al. (2020) shows that abundant negatives are essential for learning globally optimal representations. However, training samples in each iteration typically contain only small number of categories, and under the "current image prompt, current image detect" strategy in T-Rex2 where prompts are sampled exclusively from the GT boxes of the current training image the classification problem degenerates into very small N- W-way classification task. As a result, the limited category diversity directly constrains the model's 's global discriminability.
For textual prompts, this can be easily achieved by padding them with additional category phrases to fixed length. The extension of visual prompts is more challenging, as the extraction of visual
4

---

Published as a conference paper at ICLR 2026
prompts relies on the images they originate from. Sampling additional batch of images during training to extract negative examples would significantly reduce training efficiency. Therefore, we adopt a strategy that aggregates prompts from all samples and integrates them into a unified classifier. Specifically, we collect all visual prompts from the images within the same batch and group them by category. For each category, we compute the mean of all its prompts to obtain a class prototype. The prototypes of all categories are then concatenated and shared across all images in the current batch, serving as the weights of the classifier. Unlike the "current image prompt, current image detect" strategy, this strategy not only increases the number of negative examples for visual prompts, thereby stabilizing training, but also implicitly simulates cross-image prompting. This is because the visual prompts for a given category are aggregated not only from the current sample but also from positive examples other samples. This simple strategy substantially improves the performance of detection based on visual prompts, as evidenced by the experiments reported in Section 4.4
Vixuul-Textual Prompt Relation Distillation. Prompt-based detection localizes and classifies targets by selecting proposals with the strongest responses to the prompt. where the prompt functions analogously classifier. The prompt embedding space is expected exhibit intra-class compactness and inter-class separability, with similar concepts clustering together and dissimilar ones remaining apart. Language encoders like CLIP and BERT inherently possess these properties thanks their learning objectives: BERT's masked language modeling, which clusters semantically related words, and CLIP's contrastive pretraining, which aligns and separates concepts across modalities. However, since the appearance of instances can vary significantly due to individual differences, environmental factors, and other conditions, the distribution of visual prompts tends to exhibit high variance and blurred, ambiguous semantic boundaries. Multimodal detectors such as T-Rex2 address this issue by aligning visual prompts with their corresponding textual prompts. such alignment were to be fully achieved, the visual prompt space would naturally inherit the semantic organization of the text embedding space, thereby obtaining strong intra-class compactness and inter-class discrimination. Nevertheless, studies (Liang et al. (2022) |Schrodi et al. (024)) have shown that images and text cannot be perfectly aligned, which fundamentally limits the effectiveness of this indirect alignment approach. An intuitive approach is to apply the Supervised Contrastive Loss (Khosla et al. (2020) to visual prompt embeddings:
log exp(pep5/T) jES+() EKES exp(p.pa/T)
LSCL
Σ
(6)
(i)( iES
Here, (i) denotes the set of prompts that belong to the same category as Ps, while S denotes the set of all prompts. However, such a hard contrastive loss treats all negative examples equally, making it incapable of capturing the correlations between concepts. Moreover, when training on grounding datasets, it may also suffer from the influence of false negatives. For instance, women and person are treated as different categories during training, yet considering their corresponding visual prompt embeddings negatives against each other is clearly unreasonable. Considering that language models provide a strong prior for semantic similarity, we adopt the following visual-textual prompt relation distillation loss:
Ldisill DaitssntroooSSftax((CCCC Softmax(PP"))
(7)
Ng
N₂
eaplacc /T)
exp(pep] /T))
log
(8)
N₂
En
Σn eaplpert /T.))
Here, N, denotes the number of prompts, p and P represent the visual prompt embedding vector and matrix, respectively, e and C denote the corresponding textual feature vector and matrix, and Tr, To are the temperature parameters. Note that both c and p represent 1.2-normalized vectors.
is worth noting that, unlike the alignment loss which enforces constraints over visual-text pairs, the relation distillation loss leverages the relational structure among text-text pairs as prior and directly optimizes the topology of the visual prompt space. This approach avoids the difficulties of cross-modal alignment and instead adjusts the interrelations among visual prompts in more direct manner, enabling the learned visual prompt space to exhibit strong intra-class compactness and inter-class separability. Importantly, the two losses are compatible and complementary: the alignment loss guides visual prompts toward stable semantic anchors defined by text embeddings, while the relation distillation loss focuses refining the structural relationships within the visual prompt space,
5

---

Published as a conference paper at ICLR 2026
leading to more discriminative and semantically consistent representation. Overall, the total loss of DETR-VE ViP is as follow:
Lotall = Drse + Adistill Edistil-
(9)
Selective Fusion. Fusing text prompt embeddings with image features in the early stage a commonly adopted technique in open- veee-oocayulary object detection. Such fusion not only allows the prompt embeddings to capture image-specific information but also makes the regions in the image with high responses to the prompts more semantically salient, and is generally considered an effective means to enhance prompt- based detection performance. However, in practical applications, the provided prompts can be highly flexible. In interactive detection, users may only wish detect objects of a few categories of interest within an image and thus provide only 1-2 prompts. In contrast, in batch annotation scenarios, users may specify large number of prompts in advance (e.g., the 80 categories COCO), while for given image some of these categories may have no corresponding instances. Under such circumstances, becomes necessary to consider whether fusing prompt embeddings of irrelevant categories could have negative effects.
To investigate this, we incorporate the fusion strategy of Grounding DINO into DETR-ViP, as formulated in Equation (2) The prompt-to-iman fusion is defined in Equation (TO), with image-to- prompt fusion obtained by swapping X₁ and Pv.
Q W WxPv, = WvPv, X9 Softmax QKT VV,
(10)
where Wn (O (Q, V)) denote the query, key, and value projection matrices, respectively. X; represents the fused image features. We use the model equipped with this fusion strategy to perform detection with either single prompt all prompts, and visualize the corresponding results. The visualization results are shown in Figure 3 When all 80 COCO prompts are given, the model detects correctly, but performance collapses with only the 'person' prompt, as the global prompt integration causes the model to overfit to many-prompt scenarios.
This phenomenon exposes the fragility of full fusion strategies. A robust fusion mechanism should yield stable detection regardless of the number of prompts. To this end, we propose selective fusion strategy that first determines whether a category exists in the image, and fuses only the relevant prompts while suppressing the others. Concretely, we introduce gating vector G into the fusion layer's attention weights. The gate Jc for a category is intended to satisfy: 9e -> when e is present in the image, and Ge - -o when e is absent. To estimate category presence, an auxiliary classification branch computes the similarity matrix S E gLixNNr between image features The maximum similarity MAX(S, 0) E Nr for each prompt is taken as its confidence score, followed by a threshold activation function 5(-), which outputs 0 if the input exceeds 0 and -o otherwise. The specific details of the selective fusion strategy are as follows:
X 1R and prompt embeddings Pv E
Xi- Softmax QKT + G W,
8MMXXSS -V,,=&0MAXX( dim
(Simm(X PV) (1I)
Vd
By applying the selective fusion strategy in both training and evaluation, the model filters prompts with insufficient responses to the current image, leading to a more stable and robust fusion process.
4 EXPERIMENT
4.1 IMPLEMENTATION DETAILS
We provide two versions of our model based on Swin Transformer backbones: Tiny and Large. Both variants stack six Transformer encoder layers as the visual encoder, three deformable cross-attention layers as the visual prompt encoder, and six deformable aross-attention layers the box decoder. We adopt Adam ( Loshchilov & Hutter (0077)) as the optimizer, with a learning rate x 10-5 for the backbone and x 10 for the other modules. In Equations (4) and (9). Aels: \1 AGIoU, ÄAlign and Auistill are set to 1.0, 5.0, 2.0, 1.0, and 10.0, respectively. In Equation (8), T₁ and Ty are set to 0.07 and
0.1. DETR- ViP is trained on both detection and grounding datasets, including Objects365 (VI) Shao cet al. 2019)) and GoldG( Kamath et al. (202T) (comprising GQA( Hudson & Manning (2019) and Flickr30k( Plummer al. (2015))) with images from COCO( [in al. (014)) excluded.

---

Published as a conference paper at ICLR 2026
(a) Detect with person' prompt
(b) Detect with all prompts
Figure 3: Illustration of Unstable Fusion. (a) With only the 'pesonn' prompt, detection fails. (b) With
all 80 COCO category prompts, detection succeeds.
4.2 EVALUATION PROTOCOL AND METRICS
Following T-Rex2 Diang et al. (2024) ) ) we evaluate our method in the zero-shot setting on COCO Lin
al. (2022a)), and Roboflow 100( Ciaglia et al. (2022)) ), without training on any of these benchmarks. For fair comparison with T-Rex2( Jiang et al. (2024) and YOLOE Cheng et (024 ), evaluation is conducted under the Visual-G protocol. For fair comparison with T-Rex2 ( (Jiang et al. (2024)) and YOLOE ([heng et al. (2024)). evaluation is conducted under two protocols: Visual-G and Visual-I. Under the Visual-G protocol, methods adopt generic visual prompt workflow: for each benchmark, N images per class are sampled from the training set, each containing at least one instance the target class. All GT boxes of that class are used to extract the prompt embeddings, and their average serves as the visual prompt for evaluation. In contrast, under the Visual-I protocol, for a test image containing A categories, only one GT box randomly selected for each category to extract the corresponding visual prompt. We report standard AP on COCO; Fixed AP( Dave et al. (2021). )) AP,, APc. and APr on LVIS, corresponding to frequent, common, and rare categories and average AP across 35 ODinW and 100 Roboflow datasets.
al. 20144), LVIS( Gupta al. (2019), ODinW
To capture the semantic properties of visual prompts, we introduce IIR (Intra-Inter Similarity Ratio) quantify intra-class similarity and inter-class separability (Equation (22). The numerator averages pairwise similarities within each category, while the denominator measures similarities among category-level mean vectors.
Sim(p: P,) / [C[[c|- -
ICI
IISR = CI
Σ GEC NN
Σ R.j 1:23
Σ 1pe2et
Sim (pe. p.)
(12)
Here, denotes the set of categories, Sim(..) refers to the cosine similarity, N, indicates the number of prompts in category and Pe, Pr. represent the mean vectors of all prompts in categories c, l.
4.3 ZERO-SHOP OBJECT DETECTION
Visual-G. We evaluate the zero-shot generic detection capabilities of DETR- ViP COCO, LVIS. ODinW, and Roboflow 100, and report the results in Table [ The results of T-Rex2 series and YOLOE series are quoted from their paper(] Jiang et al. (2024); Cheng al. 20224). Experimental results show that DETR-ViP substantially outperforms YOLOE under the same training data. On LVIS, DETR-ViL ViP-T surpasses YOLOE-v8-LE by 6.9 AP. with gains of 1.9, 8.7, and 6.3 on AP,, AP c. and AP,. respectively. With the same backbone, DETR- -iP also matches or exceeds T-Rex2 despite using far less data, For example, with Swin-T, DETR- -iP improves over T-Rex2 by 4.4 AP on COCO and by 3.7 AP on LVIS, with larger gains on AP, (+5.2) and APc (+9.4). These improvements are most pronounced on common and rare categories, likely due to more efficient optimization visual prompts, while advantages on frequent classes are smaller. Results ODinW and Roboflow 100, which exhibit larger domain shifts, further confirm this trend: DETR- ViP-T outperforms T-Rex2-L by 3.4 AParg ODinW and 5.1 on Roboflow 100. DETR-ViP-L further improves DETR- DETR-ViP-T, surpassing T-Rex2-L by 3.7 AP on COCO and 1.5 AP on LVIS. These results highlight the advantages of DETR-V ViP compared with other visual prompt detection models.
7

---

Published as a conference paper at ICLR 2026
Table Zero-shot generic detection evaluation on COCO, LVIS, ODinW, and Roboflow For training data, O365, OI, HT, and CH indicate Object365( (Shao et al. (2019)) OpenImages( (aaanaggsKKKasinnn Krasin al. al.] (2017)), HierText( [Long et al. (2022)), and CrowdHumann( Shao et al. (2018), respectively.
GoldG( (aamah et al. (2021)) includes GQA( [Hudson & Manning (2019) and Flickr30k( Trckr30k(Plummme PPlummer
al. (015)). Best results with Swin-T are underlined, and those with Swin-L are boldfaced.
Training Data
COCO
LVIS
ODinW
Roboflow
Model
AP
AP
AP.
AP.
AP,
AP Ragg
AParg
O365 ОI,T
T-Rex2-T
38.8
37.4
29.9
33.9
41.8
23.6
17.4
HT. CH
T-Rex2-L
46.5
47.6
45.4
46.0
49.5
27.8
18.5
SA-IB
YOLOE-v8-S
26.2
21.3
27.7
25.7
Y8LME-v8-M
31.0
27.0
31.7
31.1
YOLOE-v8-L
O365
34.2
33.2
34.6
34.1
YOLOE-vI1-S
GoldG
26.3
22.5
27.1
26.4
YOLOB-vII-M
31.4
27.1
31.9
31.7
YOLOB-v11-L
33.7
29.1
34.6
33.8
DETR-ViP-T DETR-ViP-L
0365
43.2 50.2
41.1
35.1
43.3
40.4
31.2
23.6
GoldG
49.1
46.3
50.5
48.4
34.5
24.7
Table Zero-shot interactive detection evaluation on COCO and LVIS.
COCO
LVIS
ODin
Roboflow 100
Model
AP
AP
AP,
APc
AP,
AP avg
APavg
T-Rex2-Swin-T
56.6
59.3
54.6
63.5
64.4
37.7
30.6
T-Rex2-Swin-L
58.5
62.5
57.9
66.1
70.1
39.7
30.2
DETR-ViP-1 65,4
66.1
57.5
73.5
78.4
46.8
40.1
DETR-ViP-L 71.1
71.9
64.2
78.2
83.6
51.2
44.3
Visual-I. We evaluate the zero-shot interactive detection capability of DETR-VE ViP, and the results are presented in Table Since YOLOE (Cheng et al. (2024)) does not report results under this setting, compare only with T-Rex2 (Jiang al. (2024)). Under the Visual-I protocol, categories that do not appear in the current image are excluded, which effectively provides prior knowledge about the image content and makes the task substantially casier than Visual-G detection. As shown in the results, the AP on COCO, LVIS, ODinW, and Roboflow 100 are consistently higher under Visual-I than Visual-G G. Moreover, DETR-VIN ViP also surpasses T-Rex2 across the board under the Visual-I protocol. We attribute this to the presence of more negative examples during training, enabling DETR-Vin ViP to learn large N-way classification task, thereby strengthening its classification ability.
4.4 ABLATION EXPERIMENTS
DETR-VED ViP extends Grounding DINO Liu et al. (2024) by first introducing VIS-GDINO, a de- tector supporting visual prompts. Unlike Grounding DINO. VIS-GDINO removes fusion modules (Equation (2) and inserts visual prompt encoder (Equation 3 between the backbone and encoder, enabling conditional detection based on ussr-provided reference boxes. We then progressively extend VIS-GDINO into DETR-ViP and validate each modification through ablation studies (Table 3) Furthermore, we sample 128 images for each category from the COCO training set and perform t-SNE visualization along with similarity analysis the generated prompts, shown in Figure
VIS-GDINO & Image-Texx Text Align. For the baseline VIS-GDINO, performance is limited, achieving only 21.1 mAP on COCO and 17.2 mAP on LVIS, as sampling a few prompts from the current image for classification training fails to learn discriminative global distribution of visual prompt embeddings. Introducing the image-text contrastive loss substantially improves results to 29.2 mAP on COCO (+8.1) and 23.4 mAP LVIS (+6.2), by distilling rich semantic priors from text features
8

---

Published as a conference paper at ICLR 2026
Table Roadmap to DETR-ViP. The left-aligned '+' indicates cumulative improvement, while the indented '+' marks standalone improvement not accumulated into the subsequent variants.
COcO-val
LVIS-minival
Model
AP
IIRR
AP
AP,
APc
AP,
IISR
VIS-GDINO-T
21.1
0.8797
17.2
22.0
13.0
11.6
0.8954
+Text-Image Alignment
29.2
0,9743
23.4
27.8
19.5
18.4
0.9872
+Global Prompt Integration
35.6
1.0734
33.0
32.8
33.4
31.8
1.0550
+ TTett-Region Distillation
41.5
1,4954
39.5
39.0
41.5
33.4
1.2248
+Encoder Fusion
41.3
1.5001
39.1
38.4
41.3
33.1
1.2174
+ Encoder Selective Fusion
42.2
1.4963
40.6
40.3
42.4
34.2
1.2165
+Decoder Fusion
40.8
1.4976
25.5
22.8
29.0
23.5
1.2172
+ Decoder Selective Fusion
43.2
1,5010
41.1
40.4
43.3
35.1
1.2212
bicycle
motercycle handbag
airplume
truek
backpack
20
-20
-10
-40
-40
-20
20
Inter Simleiity
of Inter Similarity
of Intern Similanity
30000
50000
30000
40000
40000
30000
500ce
10000
Similarity Range
Similarity Range
Similarity Range
(a) +Text-Image Alignment
(b) +Global Prompt Integration
(c) +Relation Distillation
Figure 4: Visual prompt analysis for different model variants. (Top) t-SNE visualization of the
visual prompts. (Bottom) Distribution of intra- and inter-class pairwise similarities.
into visual prompts. Analysis of visual prompts confirms this effect: in Figure aa), VIS-GDINO's prompts are scattered, whereas Figure top) shows emerging cluster structures. Similarly, pairwise similarity distributions overlap heavily between intra- and inter-class prompts for VIS-GDINO (Figure (b)) but this overlap is greatly reduced after alignment (Figure 4a, bottom)). In addtion, IISR increases by 0,0946 COCO and 0.0918 on LVIS. These results indicate that the alignment enhances the semantic consistency of visual prompts, which in turn improves detection performance.
Global Prompt Integration. With global prompt integration, the intermediate DETR-ViL ViP achieves 35.6 AP on COCO (+6.4) and 33.0 on LVIS (49.6). This mechanism improves training efficiency by aggregating prompts of the same class across samples, which encourages intra-class clustering, while integrating prompts of different classes enlarges the negative set, sharpening inter-class boundaries. and common categories
On LVIS, AP, and AP, increase markedly (+13.4 and +13.9), contribute to the whole updates rather than only the current sample. As shown Figure(41b), visual prompts form clearer clusters with reduced intra-/inter-class overlap, confirming the effectiveness of global prompt integration in structuring the embedding space.
Visual-Textuna Prompt Relation Distillation. As shown in Figure 4cc., introducing the visual- textual prompt relation distillation not only sharpens the inter-class boundaries but also reduces the intra-class variance of prompts. Although some categories exhibit closely clustered prompts, these

---

Published as a conference paper at ICLR 2026
correspond to semantically related concepts such as truck and car, bench and chair, or backpack and handbag. This indicates that the prompt embedding space demonstrates semantic coherence, where semantically related concepts mapped closer together and unrelated ones farther apart. The overlap between intra- and inter-class similarity distributions is further reduced, while the inter-class similarity distribution becomes more skewed toward 1.0, suggesting that intra-class prompts nearly identical. On COCO and LVIS, IISR increased by 0.4220 and 0.1698, respectively. Benefiting from this, the intermediate variant of DETR- ViP achieved an additional AP gain of 5.9 on COCO, reaching 41.5, and 6.5 on LVIS, reaching 39.5. Specifically, APr, APe, and AP, improved by 1.6, 8.1, and 6.2, respectively.
Selective Fusion. The fusion module is designed to enable interactions between prompt embeddings and image features. Its primary role is to adapt the prompts to the characteristics of the current sample. Therefore, after introducing various fusion modules, the IISR of the unfused prompts remains almost unaffected. As shown in Table 3 directly adopting the encoder fusion from Grounding DINO brings almost no gains and even reduces AP on COCO and LVIS.
Practically, this naive strategy is highly sensitive to the number of prompts: detection works when all COCO cat- egories provided but fails with single class prompt
mAP ('person')
60
(Figure 3) As shown in Figure 5 experiments varying the number of prompts Np indicate that AP, increases with more prompts and gradually converges, revealing train-test gap, since global prompt integration dur- ing training typically involves many prompts. The pro- posed selective fusion module first determines whether category is present before interacting with its prompt, thereby suppressing irrelevant information and enabling robust detection. This approach stabilizes performance across different prompt numbers and improves AP by +0.7 COCO and +1.1 on LVIS (Table 3) Applying full fusion to the decoder harms AP, as decoder fusion directly modifies object queries, amplifying the negative effect of irrelevant categories. When selective fusion is applied the decoder as well, these issues are mitigated, yielding further gains of +1.0 AP on COCO and +0.5 on LVIS.
30
15
0
10
20
40
80
number fused categories
mAP ('bicycle')
30
15
0
10
20
40
80
number of fused categories
Full Fusion
Selective Fusion
Figure 5: mAP num. of Prompts
5 CONCLUSION
We propose DETR-ViL an open-vocabulary detection framework that substantially advances the baseline of visual-prompted detection. Building upon Grounding DINO, construct a baseline model, VIS-GDINO, which supports visual prompts. Through this baseline, we identify that the suboptimal performance of visual prompts stems from the lack of semantic coherence. To this end, we introduce Global Prompt Integration and Visual-Texuue Prompt Relation Distillation. Global Prompt Integration enhances the global semantic organization of the prompt space by increasing the number of positive and negative examples. Visual-Textur Prompt Relation Distillation transfers the relational priors encoded in textual prompts into visual prompts, thereby endowing them with structured semantic relationships. With these improvements, visual prompts not only reduce intra- class variance but also acquire sharper semantic boundaries, ultimately leading to enhanced detection performance. In addition, we refine the fusion module of Grounding DINO by introducing Selective Fusion strategy, which enables DETR-ViP to perform stable and robust prompt fusion regardless of the number of user-provided prompts, thereby further improving detection performance. Extensive experiments demonstrating that under the Visual-G protocol, DETR- ViP significantly outperforms existing counterparts, validating its superior effectiveness. Moreover, comprehensive ablation studies confirm the contribution of each proposed component and provide in-depth analyses on how our method enhances the semantic organization of visual prompt distributions.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China No. 62572385, the Fundamental Research Funds for the Central Universities No. xxj032023020, and CAAI-CANN Open Fund, developed on Openl Community.
REFERENCES
Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PmL., 2020.
Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real-time apenbboocayllayy object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16901-16911, 2024.
Floriana Ciaglia, Francesco Saverio Zuppichini, Paul Guerrie, Mark McQuade, and Jacob Solawetz. Roboflow 100: A rich, multi-domain object detection benchmark. arXiv preprint arXiv:2211.3352, 2022.
Achal Dave, Piotr Dollár, Deva Ramanan, Alexander Kirillov, and Ross Girshick. Evaluating large-vocabulary object detectors: The devil in the details. arXiv preprint arXiv:2102.01066. 2021.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for compulational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. In European conference on computer vision, pp. 540-557. Springer, 2022.
Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 1440-1448, 2015.
Xiuye Gu, Tsung-Yi Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104113921 2021.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 5356-5364, 2019.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In 2006 IEEE computer sociery conference on computer vision and pattemn recognition (CVPR '06), volume 2, pp. 1735-1744 IEEE, 2006.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9729-9738, 2020.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchilee Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 000..1888000000 2018.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 6700-6709, 2019.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- laigualang representation learning with noisy text supervision. In International conference on machine learning, pp. 4904-4916, PMLR, 2021.
11

---

Published as a conference paper at ICLR 2026
Qing Jiang, Feng Li, Tianhe Ren, Shilong Liu, Zhaoyang Zeng. Kent Yu, and Lei Zhang. T-rex: Counting by visual prompting. arXiv preprint arXIv:2371.13596, 2023.
Qing Jiang, Feng Li, Zhaoyang Zeng. Tianhe Ren, Shilong Liu, and Lei Zhang. T-rex2: Towards generic object detection via text-visual prompt synergy. In European Conference Computer Vision, pp. 38-57. Springer, 2024.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1780-1790, 2021.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Sama, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 13:18661-18677 2020.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4015 2023.
Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng, Dhyanesh Narayanan, and Kevin Murphy, Openimages: A public dataset for large-scale multi-label and multi-class image classification. 2017.
Datasel available from https///giihub..o/opppiinn
Chunyuan Li, Haotian Liu, Liunian Li, Pengchuan Zhang, Jyoti Aneja, Jianwei Yang, Ping Jin, Houdong Hu, Zicheng Liu, Yong Jae Lee, et al. Elevater: A benchmark and toolkit for evaluating language-an augmentte visual models. Advances in Neural Information Processing Systems, 35: 9287-9301, 2022a.
Feng Li, Qing Jiang, Hao Zhang, Tianhe Ren, Shilong Liu, Xueyan Zou, Huaizhe Xu, Hongyang Li, the Jianwei Yang, Chunyuan Li, et Visual in-context prompting. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12861-12871, 2024.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven
Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694-9705, 2021.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre- training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10965-10975, 2022b.
Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y. Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. ArXiv, abs/2203.02053. 2022. URL https: //api. semanticscholas org/CorpusID; 247244904
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision, pp. 38-55. Springer, 2024.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEFF/CVF international conference computer vision, pp. 10012-10022, 2021.
12

---

Published as a conference paper at ICLR 2026
Shangbang Long. Siyang Qin, Dmitry Panteleev, Alessandro Bissacco, Yasuhisa Fujii, and Michalis Raptis. Towards end-to-end unified scene text detection and layout analysis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1049-1059, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Amab, Mostafa Dehghani, Zhuoran Shen, et al. Simple open-vocabulary object detection. In European conference computer vision, pp. 728-7778777 -55.. Springer, 2022.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1rX7vv/807.0333 2018.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pp. 2641-2649, 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. $748-8763 PmLR, 2021.
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408. 00714, 14, 2024.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788, 2016.
Simon Schrodi, David T. Hoffmann, Max Argus, Volker Fischer, and Thomas Brox. Two effects, one trigger: On the modality gap. object bias, and information imbalance in contrastive vision- language models. In International Conference on Learning Representations, 2024. URL https //api. www.ssteaaanncccccooooooo
ortg/CorpussD::22223333
Shuai Shao, Zijian Zhao, Boxun Li, Tete Xiao, Gang Yu, Xiangyu Zhang, and Jian Sun. Crowdhuman: A benchmark for detecting human in crowd. arXiv preprint wrxiv:/805.00// 2018.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8430 8439, 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European
conference on computer vision, pp. 776-794. Springer, 2020.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018
Hu Xu, Gargi Ghosh, Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2109. 14084, 2021.
Yifan Xu, Mengdan Zhang, Chaoyou Fu, Peixian Chen, Xiaoshan Yang, Ke Li, and Changsheng Xu. Multi-modal queried object detection in the wild. Advances in Neural Information Processing Systems, 36:4452 4469, 2023.
Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detelip: Dictionary-enrrched visual-concept paralleled pre- training for open- world detection. Advances in Neural Information Processing Systems, 35:9125-9135 2022.
13

---

Published as a conference paper at ICLR 2026
Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary detr with conditional matching. In European conference on computer vision, pp. 102-122. Springer, 2022.
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung-Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.03605, 2022a.
Hao Zhang. Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianwei Yang, and Lei Zhang. A simple framework for open-vocabulary segmentation and detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1020-1031, 2023.
Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Con- trastive learning of medical visual representations from paired images and text. In Machine learning for healthcare conference, pp. 2-25. PMLR, 2022b.
Xiangyu Zhao, Yicheng Chen, Shilin Xu, Xiangtai Li, Xinjiang Wang, Yining Li, and Haian Huang. An open and comprehensive pipeline for unified object grounding and detection. arXiv preprint arXiv:2401.02361, 2024.
Yiwu Zhong, Jianwei Yang. Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-base ianguage-image pre- training. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 16793-16803, 2022.
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twenty- thousand classes using image-level supervision. In European conference on computer vision, pp. 350-368. Springer, 2022.
Chengxu Zhuang, Alex Liin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. Proceedings of the IEEE/CVF international conference on computer vision, pp. 6002-6012, 2019.
Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang. Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. Advances in neural information processing systems, 36: 19789, 2023.
14

---

Published as a conference paper at ICLR 2026
A STATEMENTS
Use of Large Language Models (LLMs). During the preparation of this paper, employed LLM-based tools to assist with writing and polishing. Specifically, we used the following instructions:
Pleasse act as a native English speaker and Al expert, and translate the following passage into English."
"Please help refine the following passage to make it more concise while preserving its meaning, and ensure it adheres to the style of Al conference papers."
The LLM was used solely for language assistance. All ideas, analyses, and experimental results are original contributions of the authors.
This work does not involve human subjects, animals, or sensitive personal data. The study does raise any ethical concerns regarding data collection, experimentation, or potential societal impacts.
Ethics Statement.
Reprodicibility Statement. To ensure reproducibility of our results, commit publicly re- leasing the code, pre-trained model weights, and training/evaluation logs necessary to reproduce the main findings reported in this paper. All resources will be provided with sufficient documentation to facilitate replication and verification by the research community.
B DETAILED MOTIVATION AND LIMITATION
Motivation. The motivation behind this work that visual prompts possess advantages not offered by text prompts, yet their potential remains underexplored. Compared with textual prompts, visual prompts are more effective for detecting rare categories. This conclusion has been validated in T-Rex2 (Jiang et al. (2024)): shown in Table 4 the AP of visual prompts rare categories in LVIS, as well as on ODin W and Roboflow 100, is generally higher than that of textual prompts.
Table 4: Comparison between visual and text prompts of T-Rex2 on rare categories. The results are taken from Table 1 of T-Rex2 (Jiang et al. 20244). In the LVIS column, the results on the left and right of "/" correspond to minival and val, respectively.
Model
Prompt Type
LVIS(AP,)
ODinW(AP
Roboflow 100(AP...) ivg)
T-Rex2-Swin-T
Text
37.4/29.0
18.0
8.2
T-Rex2-Swin-T
Visual-G
29.9( (32..7)/332.A4a... 322.4(13.4)
23.6(13.6)
17.4
T-Rex2-Swin-L
Text
49.2/42.7 42.7
22.0
10.5
T-Rex2-Swin-L
Visual-G
45.4 43.8
27.8(15.8)
18.5
Jiang al. (0244) also conducted a per-category accuracy comparison between visual and textual prompts on LVIS, as shown in Figure 4 of its original main paper. For frequent categories, text prompts outperform visual prompts in the majority of cases (Text:Visual 254:151). However, for rare categories, the reverse is more common, with visual prompts achieving higher accuracy (Text:Visuall Visual 84:253). This is because text prompts require aligning category "concepts" across the text and vision domains, which depends on large-scale vision- lisioanlanguage pre-training: in contrast, visual prompts are directly sampled from the vision domain, and thus are inherently compatible and similar to the target regions. These observations highlight the necessity of studying visual prompts.
Although visual prompts perform well on rare categories, their performance on more common categories is significantly lower than that of text prompts, as shown in Table 5 We argue that this gap does not reflect the upper bound of visual prompts, but rather that their potential has not yet been fully explored. In this work, aim to investigate the underlying reasons for the suboptimal performance visual prompts and fully exploit their potential, thereby advancing the development of prsmpt-based detection.
15

---

Published as a conference paper at ICLR 2026
Table 5: Comparison between visual and text prompts of T-Rex2 on frequent and common categories. The results are taken from Table of T-Rex2 Jiang et al. (024). In the LVIS column, the results on the left and right of "/" correspond to minival and val, respectively.
COCO AP
LVIS AP;
Model
Prompt Type
AP
AP.
T-Rex2-Swin-T
Text
45.8
42.8/34.8
46.5/41.2
39.7 39.7/31.5 /3.5
T-Rex2-Swin-T
Visual-G
38.8
37.4(un.4) /34.9110.))
(1.8( (q1.)
33.9 /30.s.s./33....... (3..)
T-Rex2-Swin-L
Text
52.2
54.9/45.8
56.1/50.2
54.8/43.2
T-Rex2-Swin-l.
Visual-G
46.5(45.7)
47.6(17.3)
(1..)
49.5 (9.5
46.0(18.8) (22.0
Limitation. While this work emphasizes the importance of visual prompts and primarily focuses on exploring their potential, it does not imply that we overlook the value of textual prompts. We fully recognize that textual prompts can more explicitly describe object attributes as color (e.g.. "a woman in a red shirt") or spatial position (e.g.. "a car on the left side of the image")-which are advantages that visual prompts inherently lack. Although this paper does not address textual prompts, we argue that combining more discriminative visual prompts with guided natural language prompts represents a promising direction toward more general open-world detection.
C MORE RELATED WORK: CONTRASTIVE LEARNING
Contrastive losses( Hadsell et al. 2006)) enable direct optimization of similarity within the represen- tation space, eliminating the need match inputs to fixed targets. This idea forms the foundation of
many unsupervised representation learning methods( Wu et al. (2018); Oord et al. (0118); ;) Hjelm et al. (2018; Chen et al. (0200): Zhuang al. (2019) Tian et al. 2020); He al. (2020). InfoNCE( Oord et al. (201 8)), one of the most widely used contrastive loss formulations, where decreasing the loss encourages the query q to be similar to its positive key k and dissimilar to negative keys Chen al. (2020) introduces self-supervised pretraining framework that treats two augmented views of the same image as a positive pair and different images as negative pairs, and learns representations using InfoNCE. He et al. 2020) emphasize the importance of large numbers of negatives and propose a memory bank mechanism to expand the negative pool. Although these approaches achieve remark- able progress in self-supervisee visual pretraining. their inability to exploit the label information in annotated datasets limits their effectiveness on downstream tasks. To address this, Khosla et al. (2020) extend batch contrastive learning the fully supervised setting, enabling the model to fully leverage category labels for contrastive representation learning.
Multimodal contrastive representation learning( [Zhang et al. (2022b); Xu al, (2021) Li et al. (002
Jia et al. 2221)) aims to map inputs from different modalities into a shared embedding space, giving rise to powerful form of weakly supervised pretraining. (Raffodd al. (202T), trained on 400 million image-text pairs from WebImageTess learns highly transferable visual representations and demonstrates strong zero-shot performance across diverse downstream tasks, making a promising direction for open-set recognition. However, [Liang et al. (2022) first identified the existence of modality gap. Specifically, the image and text embeddings do not form a unified space but rather reside in two narrow cones separated by gap, and the contrastive loss tends to preserve this geometric discrepancy. [Schrodi et al. (2024) further observe that only few embedding dimensions predominantly account for this modality difference, and there is clear evidence that larger gap correlates with improved downstream performance. Moreover, although simple post-hoc methods shrink the gap geometrically, they generally fail produce meaningful performance gains.
D DETAILS OF ARCHITECTURE
D.1 BASELINE MODEL: VIS-GDINO
VIS- GDINO is baseline model we built to support visual prompting. It can be regarded as inserting a visual prompt encoder between the backbone and the encoder of DINO, while replacing the classification head with the classifier shown in Equation (right).. Similarly, it can also be
16

---

Published as a conference paper at ICLR 2026
interpreted as inserting visual prompt encoder between the backbone and encoder of Grounding DINO, with the fusion modules in both the encoder and decoder removed.
As shown in Figure 6 given input image and K user-specified boxes {6. VIS-GDINO first extracts image features X through the backbone network. The visual prompt encoder then generates visual prompts Pv from X based on {b;}. leveraging deformable attention mechanism to effectively aggregate features within each box. The features are further encoded by a deformable encoder without any fusion modules to produce set of proposals O, Scores are computed using the contrastive classification head shown on the right side of Equation (1) (i.e. Score o (OPP + b)), and the top-k proposals are selected to initialize the decoder queries Q. Within the decoder, Q passes through multiple layers composed of self atff-tttention and deformable cross- cross-attention modules, where the self-attention module models interactions among object queries and the cross-attention module extracts information from the image features. After decoding, Q is multiplied with Pv to compute the scores, and the predicted boxes are obtained via the box regression head.
Defermable Decder
Sakeriinn (laph)
Box
Object
Prompt
Encoder
Figure 6: A simplified illustration of VIS-GDINO. Compared Grounding DINO Liu al. (2024)), VIS-GDINO inserts a visual prompt encoder between the backbone and the encoder, and removes the fusion modules in both the encoder and the decoder.
D.2 TEXT ENCODER
Unlike Grounding DINO, we use CLIP Radford et al. (2021)) the text encoder. We construct the input to the text encoder using the template "This is an image of ..." and take the pooled output as text feature. To improve training efficiency, pre-extract the text features with CLIP and store them for later use.
D.3 FEATURE ENHIANCER
The Feature Enhancer in DETR-ViC ViP is built upon that of Grounding DINO, but with two key modifications: (i) the fusion module is replaced with the proposed selective fusion, and (ii) the self-attention layer over prompt embeddings is removed. This is because, unlike Grounding DINO which uses full sentences text prompts and requires self-attention extract features for each positive phrase. DETR- ViP represents each category with a single token. Specifically, for multiple box prompts of the same category within an image, the global token in the visual prompt encoder aggregates their information, while for global prompts across images, the average vector is used as the category-level prompt embedding.
D.4 GLOBAL PROMPT INTEGRATION
This Global Prompt Integration operation is illustrated by the simple code snippet below.
prompts gather (promptss) N_iid => (N_1+N_26 2+__2+...
labels gather (labels) $ N_i -> (N (N_14N_2+...
gather_prompte - []
for 1 in labels. unique
gather tiwterr_rrmpps..ee append (prompts [labels==1] (um (label==1)
()
prompts stack (gather (gather_pronpts, 0)
labels labels. unique
To intuitively illustrate how global prompt integration works, we provide an example: the prompt
categories in samplel C₁ 2,32,37 5) and those in sample2 Ca (0, 1, 4,5,), then
17

---

Published as a conference paper at ICLR 2026
classifier covering the combined category set C 2,3, when performing classification across all samples. This approach not only increases the number of negative examples but also indirectly trains cross-image prompt detection, the prompts for classes 0 and 5 from sample are also used in the classification of sample
For Visual Grounding (VG) datasets, the situation is typically more complicated. On the hand, their annotations are generally of lower quality compared with object detection (OD) datasets such COCO Object365. On the other hand, the box descriptions VG datasets are usually short phrases extracted from image captions, leading to highly variable formats. For example, a bounding box for yellow scarf may be described as "blades to help board cut through water", which mixes action nouns with other nouns. Likewise, two boxes corresponding to two dogs may be described as short and white dog" and "two dogs". For such grounding data, integrating these phrases into prompts is challenging. If two descriptions are considered to belong to the same class only when their texts perfectly match, a large number of false negatives would be introduced, which in turn harms model training. To address this issue, we adopt two strategies. First, we decouple the training of OD and VG data. In each iteration, the entire batch is either from OD datasets or from VG datasets, thereby preventing noisy VG data from interfering with high-quality OD supervision. Second, for VG data, we extract the head noun using extract head noun(phrase) and obtain the lemma using get (ehamaaarrrrse) Both operations such as "a short and white dog" or "two dogs" are normalized to the lemma "dog" before determining whether they belong to the same class. Although conceptually simple, this approach significantly reduces the occurrence of false negatives.
implemented with the spaCy library. In this manner, phrases
D.5 CLASSIFICATION ALIGNMENT, AND DISTILLATION LOSSES
For the classification loss, we follow T-Rex2 and Grounding DINO by computing confidence scores via matrix multiplication between proposal features and prompt embeddings, followed by sigmoid activation, as defined Equation (T). It's noted that both O and P are kept unnormalized, allowing their dot-product results span the entire real domain (-oo, +o0), which ensures that the sigmoid outputs are distributed within (0, 1). For both the alignment and distillation losses, vectors involved in the computation are L2-normalized
E MORE EXPERIMENTS
Downstream Transferring. We conduct a downstream transfer experiment on COCO using the variant of DETR-ViP without the fusion module. Specifically, the class branch is replaced with a linear layer while all other modules are frozen, and the model is trained for 12 epochs. This setup is equivalent to fitting classification hyperplane in the pre-learned feature space, thus serving as measure of semantic consistency. As shown in Table 6 progressively adding image-text alignment, global prompt integration, and visual-textual distillation to VIS-GDINO steadily improves downstream AP, reflecting enhanced visual representations. However, the gains are smaller than those in Table indicating that the main performance improvements arise from optimizing the visual prompt embedding space.
Table 6: Downstream Transfer on COCO
Model
AP
AP50
AP75
APs
APM M
AP
VIS-GDINO-SwinTT
48.2
64.4
53.0
32.7
51.7
61.8
+Text-Image Alignment
48.5
65.3
53.3
32.3
52.0
62.6
+Global Prompt Prompts
48.8
65.8
53.5
33.8
52.4
62.7
+ +Text- Region Distillation
49.2
65.9
54.3
34.6
52.6
62.6
Visual-Text Prompt Relatcion Distillation rs. Supervised Contrastive Loss. The supervised contrastive loss shown in Equation (6) is another option for enhancing the discriminability of visual features. In Table we compare models trained with visual-textual prompt relation distillation versus those trained with supervised contrastive loss. can be observed that the variant using supervised contrastive loss achieves AP scores that are 0.6 and 0.7 lower than DETR-VE ViP COCO and LVIS,
18

---

Published as a conference paper at ICLR 2026
respectively. We attribute this to the fact that supervised contrastive loss treats all negatives equally, which prevents visual prompts from establishing semantic coherence. Moreover, grounding datasets contain many synonyms and hierarchical relationships- for example, "human" and "person," or "human" " and -whose corresponding instances are visually from the same category. Pulling apart the embeddings of such semantically related prompts under supervised contrastive loss can therefore harm the optimization of the embedding space.
Table 7: Comparison between Relation Distillation Loss and Supervised Contrastive Loss.
COCO-val
LVIS-
Model
AP
AP
AP,
APc
AP,
Disill
43.2
41.1
40.4
43.3
35.1
LSCr.
42.6
40.4
40.4
42.3
31.4
Hyperparameter. To ensure fair comparison with other methods, we adopt the parameter settings of existing approaches for modules that are not our core contributions. Specifically, our A ÅG. AGIoU- and Aan. are kept consistent with DINO, while AAlign follows the setting in T-Rex2. The focal loss hyperparameters y and a are set to and 0.25, respectively, as commonly used in previous works. In this subsection, perform ablation studies the visual-textual relation distillation loss parameters Adistill Tp. and Ty, as well as the selective fusion threshold 0.
First, conduct an ablation study A. Table a) reports the AP, AP, APc. and AP, of DETR-ViP trained with different À values on LVIS. We experiment with = 1.0, 10.0, and 20.0, and observe that the best performance
achieved with À 10.0.
For Ta and Te. we conduct ablations using three values: 0.05, 0.07, and 0.1, and compare the AP on LVIS. The results in Table b)) show that variations in T, and 7. do not lead significant performance fluctuations. The best performance is obtained with 7, = 0.1 and 7. = 0.07. We also note that cases where Tr < To generally yield better results, which is reasonable. Typically, we want the teacher distribution to be sharper, enabling the student to learn a less uniform distribution. However, if T. is small, the teacher distribution approaches one-hot, which undermines the effect of soft labels and causes the relation distillation loss to degenerate into a supervised contrastive loss.
Table 8: Ablation Study of hyperparameters
(b) Ablation Study of Te, To
Ablation Study
(a) Ablation Study of Adiaim
9
AP.
AP.
AP,
Assmn
AP
AP,
AP.
AP;
T
0.05
0.07
0.1
0.05
33.3
41.1
37.7
1.0
40.5
30.1
42,2
41.0
0.1
41.1
35.1
43.3
40.4
43.3
0.05
40.0
39.8
39.0
0.3
40.3
34,0
41.9
39.8
20.0
39.8
34.2
41.2
39.8
0.07
40.6
40.2
39.3
0.5
39.2
33.0
41.4
38.6
0.1
40.8
41.1
40.3
For the selective fusion strategy, we evaluate DETR-ViP under threshold values 0 E (0.05, 0.3, 0.5). Intuitively, smaller 0 allows visual prompts to participate in fusion. In this case, false negatives (FN)-categorie that are present in the image but whose prompts are excluded from fusion-become less frequent. However, false positives #FPP>cattgorinn that are absent from the image but still participate in opposite trend arises: FN increases while FP decreases. At 0.05, DETR- ViP already achieves an AP of 38.7 on LVIS. Further analysis shows that, under 0 = 0.05, the number of fused prompts is reduced by nearly 75%, which effectively suppresses interference from irrelevant categories. How- ever, when 0 = 0.5, AP drops significantly. We attribute this degradation to the exclusion of many in-image categories whose confidence scores are relatively low and therefore filtered out. Based these observations, irrelevant categories may still be fused, it ensures that the vast majority of categories present in the image remain included, avoiding critical FN cases while maintaining robustness.
to increase. When becomes larger, the
adopt 0 0.1. This choice strikes favorable balance: although a few
19

---

Published as a conference paper at ICLR 2026
F MORE ANALYSIS
F.1 ANALYSIS OF THE FUSION MODULE
Figure illustrates that the fusion strategy in Grounding DINO leads unstable outputs when the number of prompts Np varies, while the proposed selective fusion strategy ensures robustness to prompt quantity. To demonstrate that this is not an isolated case, we additionally provide mAP-N, As shown in the figure, the selective fusion strategy also leads to some mAP fluctuations with varying numbers of prompts (e.g., for categories airplane and dog), but the magnitude is substantially smaller than that of the standard fusion strategy.
curves for several other categories, as shown in Figure 7
mAP ('car")
50
mAP ("airplane")
70
mAp ('dog'
70
30
50
30
10
10
10
0
20
40
80
20
40
80
0
20
40
80
number fused categories
number of fused categories
number of fused categories
mAP ('boat')
30
mAP ('tv")
60
mAP ('scissions")
30
15
40
15
20
0
20
40
80
0
20
40
80
20
40
80
number of fused categories
number of fused categories
number of fused categories
Full Fusion
Selective Fusion
Figure 7: mAP vs N₂
Grounding DINO also sensitive to the number of prompts. We evaluate this using the MMDetection implementations of Grounding DINO Liu et al. (2024)) and MM Grounding DINO Zhao et al. (2024)), which involve critical chunked size parameter (Lecunneed). This parameter splits prompts into chunks for separate processing. As shown in Table 9 the performance of both models drops sharply when Lehunked increases from 76 to 151, with a precise transition occurring between 80 and 95, Such instability with respect to the number of prompts not only introduces additional hyperparameter, but also requires multiple runs under long text prompts, thereby increasing the testing time. With the selective fusion strategy, DETR- EEE integrates only the prompts corresponding to categories likely to appear in the input image, regardless of the number of user-specified prompts. This design makes it more robust and eliminates the need for chunking.
Table 9: AP w.r.t. L on LVIS for Grounding DINO and MM Grounding DINO
Lehunked
19
38
76
80
85
90
95
151
301
602
1203
Grounding DINO
29.3
29.0
28.9
29.7
29.5
17.9
6.80
0.26
0.55
0.35
0.17
MM Grounding DINO
41.2
43.6
41.3
40.9
41.0
25.3
9.34
0.22
0.57
0.37
0.18
F.2
VISUAL PROMPTS IN YOLOE
As shown in Figure I a)) the visual prompt representations in VIS-GDING exhibit blurred and indistinct semantic boundaries. We attribute this phenomenon to the training objective used in prompt-based detection. Without contrastive image-text alignment or other auxiliary constraints to regularize the prompt feature space, visual prompts are optimized solely under the supervision of the classification loss. Moreover, under the "current image prompt, current image detect" paradigm adopted by T-Rex2 (Jiang et al. (2024)), visual prompts are sampled only from the GT boxes of the current training image. As a result, the model observes only limited and highly local set of object instances at each iteration, preventing the prompts from being globally optimized throughout training.
20

---

Published as a conference paper at ICLR 2026
40
30
20
20
pance
inderodi
erpice
-20
Land
brchooc
30
-40
-40
-60
40
20
-20
40
(a) YOLOE-JT
(b) YOLOE-JT-Aligee
Figure 8: Visual prompt analysis for different YOLOE-JT variants. YOLOE-JT refers to the YOLOE model obtained through joint visual-text prompt training, while YOLOE-JT-Align builds upon YOLOE-JT by incorporating image-text prompt alignment loss.
Classification loss
Classification loss
'lext'
"text'
Semantic transfer
Semantic transfer
Dec layer.1
Dec layer.2
backbone
backbone
Prompt Encoder
Prompt Encoder
Clussification loss
Classification loss
(a) The single-layer loss in
The multi-layer losses in the DINO-series models.
YOLOE.
Figure 9: Classification loss and semantic transfer YOLOE and DINO.
To further verify this, training paradigm with that of T-Rex2 [Tiang et al. (2024)), where visual- prommpep detection and text-prompted detection are alternated during training. For rapid validation, we conduct experiments on YOLOE-v8s. For convenience, we denote the YOLOE model trained under this joint training scheme as YOLOE-JT. YOLOE-JT achieves 13.5 AP, 19.2 AP on rare categories, 13.7 on common categories, and 12.3 on frequent categories on LVIS minivaln respectively. The t-SNE visualization of its visual prompts is shown in Figure a). We observe that the visual prompt distribution of YOLOE-JT appears even more scattered compared with Figure [a). We suspect this is because models in the DINO family adopt multi-layer losses, i.e., constraints are applied to output of every decoder layer. As illustrated Figure 9b), text prompts directly optimize the multi-layer decoder outputs through multiple classification losses, and turn, these outputs can further optimize the visual prompts. In contrast, YOLOE applies the classification loss only at the final stage of the model as shown in Figure aa).
the publicly available YOLOE (Cheng et al. (2024) code and align its
a). which results lower efficiency in semantic propagation.
We further introduce image-ter text alignment loss and denote this variant YOLOE-JT-align As shown in Figure b), YOLOE-JT--liig produces visual prompts that form more distinct category- specific clusters in the embedding space compared with YOLOE-JT. With this enhancement, YOLOE- Ja-alggn achieves 21.5 AP, 27.8 AP,. 22.2 APc. and 19.7 AP, on LVIS. These results confirm that merely adopting joint visual-text prompt training insufficient obtain visual prompts with coherent global semantic organization. The experiments with contrastive alignment further validate that strengthening the organization of the visual prompt space essential for improving visual- essual-prompted detection performance.
21

---

Published as a conference paper at ICLR 2026
F.3 VISUALIZATION
To demonstrate the effectiveness of DETR-ViP, we present extensive visualization results.
Zero-shot Inference on COCO Figure 10 presents visualizations of DETR-VI ViP on COCO under the Visual-G protocol. For each of the categories in COCO, we sampled 16 instances extract visual prompts, and used the average of all prompts within a category as its prompt embedding.
DOWI
appa:
AODIIT
person:
DRDIO S1M
person: 32.1
Uning Gablo 7Z.
bacyO 01
Figure 10: Visualizations on COCO Dataset (Visual-G).
Additionally, we provide visualizations under the Visual-I protocol Figure where bounding boxes with semi-transparen masks denote the provided visual prompts.
Figure 11: Visualizations on COCO Dataset (Visual-I).
22

---

Published as a conference paper at ICLR 2026
Zero-shot Inference LVIS and Visual-I protocols in Figures 12 and [3] respectively.
We showcase visualizations on the LVIS dataset under the Visual-G
fireplug: 75
pole:
Hosetall
bowl: 47.5
Figure 12: Visualizations on LVIS Dataset (Visual-G).
Figure 13: Visualizations LVIS Dataset (Visual-D).
23


Reviews:
[{"review_id": "tlUmntDYgW", "reviewer": "Reviewer_ZMx9", "summary": "This paper proposes DETR-ViP, a visual-prompt-based open-vocabulary detector built upon Grounding-DINO and T-Rex2. Recent studies suggest that visual prompts can outperform text prompts for rare categories, though they still underperform on common and frequent categories. Motivated by this trade-off, the method focuses on the visual-prompt regime and introduces three components to improve semantic structure and robustness: (1) Global Prompt Integration to aggregate prompts across a batch, (2) Visual–Text Relation Distillation to transfer semantic relations from language to visual prompts, and (3) Selective Fusion to suppress irrelevant prompts during feature fusion. Empirical results show competitive performance across COCO, LVIS, ODinW, and RoboFlow100 in zero-shot detection, with consistent improvements over baselines across rare, common, and frequent category groups.", "questions": "1. The “Global Prompt Integration” section remains very vague and the code snippet is not at the standard of top-tied venue. For example: “aggregates prompts from all samples and integrates them into a unified classifier”. What exactly are “all samples”? What does “unified classifier” refer to? My understanding is that DETR-ViP pools visual prompts from all images $\\underline{\\text{in the batch}}$, groups them by category, and averages them to form one prototype per class present in that batch. These batch-shared prototypes then serve as classifier weights, increasing negative diversity and stabilizing training. If this understanding is correct, I recommend describing the mechanism in this more explicit way in the paper.\n\n2. The paper uses the $L_{dn}$ loss that appears to be the denoising loss (as in DINO). It should be explicitly stated for completeness. \n\n3. The classification loss description does not specify whether a sigmoid is applied to similarity scores before focal loss, as done in T-Rex2. Also, are embeddings L2-normalized prior to similarity computation? Could the authors clarify the exact pipeline?\n\n4. If visual prompts are already aligned with their corresponding text embeddings through the visual-text alignment loss as in T-Rex2, why do they not inherit from the textual semantic structure?  Can the authors comment on why is a relation-distillation loss needed in addition? Clarifying the conceptual motivation for combining both would help and strengthen the paper beyond good empirical results.\n\n5. From my understanding, the selective fusion gating mechanism is applied during both training and inference to handle variable prompt availability and suppress irrelevant prompts. Is this correct? If so, please make it explicit in the method section.\n\n6. Since the selective fusion strategy is motivated by interactive scenarios with variable numbers of prompts, could the authors include interactive detection and few-shot counting evaluations, similar to Tables 2–3 in T-Rex2?\n\n7. In Table 1, T-Rex2 is trained on Object365, OpenImages, HierText, CrowdHuman and SA-1B, while DETR-ViP uses Object365 + GoldG. T-Rex2 notes SA-1B improves interactive ability but can weaken generic capability. Could the authors provide a comparison under the same training data regime? This would strengthen the paper and isolate architectural contributions.\n\n8. I understand that the core of this paper is to improve the visual-prompt capabilities for open-vocabulary detection. But why remove text-prompt capability? Is there a practical issue in conserving this textual-prompt capability while improving the visual prompt capability? Could the authors explain why this capability was not retained?\n\n9. Grounding DINO already includes cross-modality (image-text) feature fusion in its encoder. The ablation includes a line “+Encoder Fusion”. Is this not already present in VIS-GDINO? Please clarify what exactly is disabled/enabled.\n\n10. In Eq. (10), the output appears to be prompt-space features (V = Wᵥ Pᵀ), but the text refers to fused image features. Could the authors adjust the description in lines 265–266 for consistency?\n\n11. Could the authors provide sensitivity results for λ (fusion gate), α/β/γ (focal loss), τₜ/τᵥ (temperatures), and λ_distill?\n\n12. Am I correct that the main difference from T-Rex2 in negative sampling is the global prompt integration mechanism, whereas T-Rex2 samples visual prompts within each image to avoid cross-image label inconsistencies? A clearer contrast in the \" global prompt integration\" section would help highlight the contribution.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- **Relevant and timely problem**\n\n    The paper addresses an emerging direction in open-vocabulary detection, focusing on strengthening visual prompts. This is particularly relevant given recent findings that visual prompts can better support rare categories compared to textual prompts, even though they still trail on common and frequent ones. Framing the work around this trade-off makes the study well-motivated and impactful.\n\n- **Novelty and contributions**\n\n    The paper proposes clear and meaningful contributions toward improving the effectiveness of visual prompts for open-vocabulary detection. The introduced mechanisms are sensible, well-motivated, and grounded in practical challenges of visual-prompt learning. While the ideas are not radically complex, they are thoughtfully designed, address concrete limitations of existing approaches, and together offer an interesting and coherent advancement in visual-prompt-based detection.\n\n- **Good empirical results**\n\n    The evaluation covers a broad set of benchmarks (COCO, LVIS, ODinW, RoboFlow100), demonstrating consistent improvements across datasets. Importantly, gains are observed across rare, common, and frequent category subsets, strengthening the motivation for exploring visual prompts in open-vocabulary detection and supporting the practical value of the proposed design.", "weaknesses": "- **Method description lacks clarity in several key areas**\n\n    Several core components are not described with sufficient precision in the main text, which makes the method harder to fully understand without forcing the reader to infer implementation behavior. In particular: the mechanics of global prompt integration remain poorly described, or the fact that selective fusion is used during training and/or inference only. Clarifying these points would greatly improve readability and transparency.\n- **Fairness of comparison to T-Rex2**\n\n    Given the architectural similarities to T-Rex2, a cleaner evaluation would require matching training conditions. However, the training data differs substantially (T-Rex2 leverages SA-1B and additional datasets, whereas DETR-ViP does not), which complicates attribution of performance gains to the proposed design alone. This point is further relevant because the T-Rex2 paper itself notes that including the SA-1B dataset “lightly weakens its generic capability.” As a result, it remains unclear to what extent differences in training data rather than architectural changes contribute to the performance gap.\n\n- **Lack of sensitivity analysis for key hyperparameters**\n\n    Several important hyperparameters are introduced throughout the method, including those for fusion $\\lambda$, distillation strength ($λ_{distill}$), temperature parameters (τₜ, τᵥ), and focal loss weighting (α, β, γ). However, the paper does not provide sensitivity studies or robustness analysis for these choices. Since these components play a central role in shaping the behavior of the model, it would be valuable to understand how performance varies with different settings and whether the method remains stable across reasonable ranges. Including such experiments would help substantiate the robustness of the approach and clarify the extent to which reported gains depend on careful hyperparameter tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "98rEL8lzhf", "reviewer": "Reviewer_dHTA", "summary": "This paper suggests the visual prompts suffer from a lack of sufficient semantic discriminability. The reseaerch question motivates the authors to propose the detection model, DETR-ViP, by enhancing the baseline model, VIS-GDINO, with more robust and discriminative visual prompts. Extensive experiments are conducted.", "questions": "Please see weaknesses above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper enhances Grounding-DINO with text and visual prompts. Furthermore, three strategies are utilized to improve the visual prompts.\n\nThe experiments show that the proposed method outperforms existing detection models in the task of zero-shot generic detection.", "weaknesses": "In the introduction, the visualizaiton for semantic discriminability in visual prompts looks similar to VIS-GDINO, so the motivation is similar? From Figure 5 in T-Rex2, I do not find the same phenomenon in tSNE visualization. Would be nice if the authors evaluate some publicly used models to clarify the motivation.\n\nSome parts of is a bit unclear to me. For instance, in preliminary, the $\\mathcal{L}_{dn}$ in Equation 4 is not defined. Also, I couldn't find detailed description for the proposed VIS-DINO. Since VIS-DINO is also a new proposed model, the details of it can be explained?\n\n In the second paragraph of Introduction, the authors state \"Nevertheless, visual prompts still underperform text prompts overall, which limits their practical applicability.\" Is there any reference or evidence for this phenomenon? \n\nIt seems the proposed method can also be used for other detection models, so would you please clarify the reason behind choosing Grounding-DINO as backbone? From Table 1 and 2, it seems Grounding-DINO can not perform better than YOLOv11 on LVIS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FmPx4eAEKj", "reviewer": "Reviewer_pcqE", "summary": "This paper introduces a openset detection baseline model called DETR-ViP, The model aims to address a long-standing problem in open-vocabulary object detection: how to use visual prompts more effectively.", "questions": "[1] Can the Q-selection mechanism be effective in other transformer-based visual prompting architectures, and is the module plug-and-play?\n[2] How does the method handle ambiguous targets, such as conflicts between category names and action names?\n[3] Table 2 introduces a new metric, but it should clearly explain how its variation reflects the model’s performance.\n[4] The zero-shot experiment only covers 10 common categories, which is insufficient to demonstrate the method’s advantage in discriminative ability.\n[5] What is the difference between the distillation method in the paper and the contrastive learning used in CLIP?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "[1The paper is easy to follow.\n[2] The ablation experiments are thorough and demonstrate the effectiveness of each module.", "weaknesses": "[1] The comparison experiments do not include enough existing methods for a comprehensive evaluation like dinov.\n[2]  The method mainly relies on prompt ensembling and supervised contrastive learning, which are not particularly novel, and the related work section lacks corresponding discussions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)