Learning Domain-Aware Detection Head with Prompt Tuning
Authors: Haochen Li
Rui Zhang
Hantao Yao
Xinkai Song
Yifan Hao
Yongwei Zhao
Ling Li
Yunji Chen
Venue: NeurIPS 2023 poster (2023)
Keywords: domain adaptation
object detection
prompt tuning

Abstract: Domain adaptive object detection (DAOD) aims to generalize detectors trained on an annotated source domain to an unlabelled target domain.
  However, existing methods focus on reducing the domain bias of the detection backbone by inferring a discriminative visual encoder, while ignoring the domain bias in the detection head.
  Inspired by the high generalization of vision-language models (VLMs), applying a VLM as the robust detection backbone following a domain-aware detection head is a reasonable way to learn the discriminative detector for each domain, rather than reducing the domain bias in traditional methods.
  To achieve the above issue, we thus propose a novel DAOD framework named Domain-Aware detection head with Prompt tuning (DA-Pro), which applies the learnable domain-adaptive prompt to generate the dynamic detection head for each domain. 
  Formally, the domain-adaptive prompt consists of the domain-invariant tokens, domain-specific tokens, and the domain-related textual description along with the class label. 
  Furthermore, two constraints between the source and target domains are applied to ensure that the domain-adaptive prompt can capture the domains-shared and domain-specific knowledge.
  A prompt ensemble strategy is also proposed to reduce the effect of prompt disturbance. 
  Comprehensive experiments over multiple cross-domain adaptation tasks demonstrate that using the domain-adaptive prompt can produce an effectively domain-related detection head for boosting domain-adaptive object detection.
  Our code is available at https://github.com/Therock90421/DA-Pro.

Learning Domain-Aware Detection Head with Prompt Tuning
Haochen Li4,4 Rui Zhang?* Hantao Yao
Xinkai Song²
Yifan Hao² Yongwei Zhao? Ling Li Yunji Chen
'Ittelligent Software Research Center, Institute of Software, CAS, Beijing, China State Key Lab of Processors, Institute of Computing Technology, CAS, Beijing, China
3 State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, CAS,
Beijing, China
University of Chinese Academy of Sciences, Beijing, China
haochen2021 Biscasicccac... ac en, zhangruiGict.es ac...i..ct.................... cn, haotao, yaotnal...ol.rr..... aar.iaa.cc..mm en, {eongxinkai, haoyifan, zhaoyongwei ac. lilingeiscas cn, cyj@ict.ac.cr
Abstract
Domain adaptive object detection (DAOD) aims to generalize detectors trained on an annotated source domain to an unlabelled target domain. However, existing methods focus reducing the domain bias of the detection backbone by inferring discriminative visual encoder, while ignoring the domain bias the detection head. Inspired by the high generalization of vision-language models (VLMs), applying VLM as the robust detection backbone following a domain-aware detection head is reasonable way to learn the discriminative detector for each domain, rather than reducing the domain bias in traditional methods. To achieve the above issue, thus propose novel DAOD framework named Nomain-AAart detection head with Prompt tuning (DA-Pro), which applies the learnable domain-adaptive prompt generate the dynamic detection head for each domain. Formally, the domain- adaptive prompt consists of the domain-invariant tokens, domain-specific tokens, and the domain-related textual description along with the class label. Furthermore, two constraints between the source and target domains are applied to ensure that the domain-adaptive prompt can capture the domains-shared and domain-specific knowledge. A prompt ensemble strategy is also proposed to reduce the effect of prompt disturbance. Comprehensive experiments over multiple cross-domain adaptation tasks demonstrate that using the domain- effectively domain-related detection head for boosting domain-adaptive object detection. Our code is available at http:/hhwwh..oo////ee/////1/1000
prompt can produce
1 Introduction
The essence of object detection lies in training a detection backbone extract visual features from images and a detection head to recognize and locate objects based on the visual features. Object detectors whose backbone are based on Convolutional neural networks (CNNs) and Visual- Transformer (ViT) have achieved encouraging performance with annotated data [31 30 25 5
However, may suffer intolerable performance degradation when applied to an unlabelled domain due to domain bias. In this respect, domain adaptive object detection (DAOD) is explored generalize detectors trained on an annotated source domain to an unlabelled target domain.
Current research on DAOD focuses on inferring a discriminative visual encoder as the detection backbone. They encourage the visual encoder to generate domain inviriiinn features by aligning
"Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Testual Enceder
[V] TC" [C"[[[Css]] [Cles]]"Cllar"]
"Clear"]
Source Desection Head
[Class]
[Bbax]
Lebeiled
Visual Encoder
Detection
[Class] [box]
Labeled
Visual Encoder
Target Detection Head
[Class] [Abox]
Unisbelled Target data
Ualabelled Target data
Target prompt "["""[[[sss]
Enceder
(a) Existing Method
(b) DA-Pro
Figure 1: (a) Existing methods focus reducing the domain bias of the detection backbone by inferring discriminative visual encoder across domains, ignoring the domain bias in the detection head. (b) The proposed DA-Pro consists of a VLMM-bssed backbone and domain-awart detection head obtained by learning domain-adaptive prompt.
These works strive to reduce the domain bias of the detection backbone. However, they ignore the domain bias in the detection head. As shown in Fig. (fa). they apply the same detection head on both domains, inevitably leading performance degradation on the target domain [3 33]. Recently. vision-language models (VLMs) show remarkably high generalization for downstream tasks on different domains, such as CLIP 29, GLIP (00 and ALIGN [6] The ability of VLMs to generate highly generalized features makes it possible to be robust visual encoder for DAOD. On this basis, applying VLM as the detection backbone following a domain- aware detection head is a reasonable way directly learn discriminative detector for target domains, rather than reducing the domain bias in traditional methods. Furthermore, inspired by prompt tuning such as CoOp using domain-relatee prompt can produce domain detection head.
source and target domains in feature space 3
In this work, we propose novel Dowaen-Aware detection head with Prompt tuning (DA-Pro) which is new DAOD framework with a VLM-based backbone and a domain- aware detection head, as shown in Fig. [b). To the best of our knowledge, we are the first apply prompt tuning in DAOD. Based on the highly generalized features generated from a ILM-based backbone and the domain-aware detection head, there is no need to focus on reducing the domain bias in the detection head. In DA-Pro, the aomain-aware detection head is obtained by learning the domain- prompt. To achieve high discrimination, domain-adape addptiive prompt exploits both domain-invarian and domain-specifi, knowledge, consisting of domain-invarian tokens, domain-specific tokens and domain-related textual descriptions, along with class textual descriptions. Doomain-invarian tokens, which are shared across domains, Iearn the Nomain-invarian knowledge. Domain-speca specific tokens and the domain- textual description are different across domains, aiming to capture the domain-specifee knowledge of the corresponding domain by both learning and hand-craft.
To optimize the domain adaative prompt so that can capture the domains-shared and domain-specnie speciicc knowledge, we further propose two unique constraints. Firstly, to learn the shared knowledge, we constrain the detection heads generated by the source-domain prompt and target- domain prompt to recognize the images as accurately as possible. Secondly, given images of one domain, we constrain the detection head generated by corresponding domain prompt to output higher confidence of its own domain than of the other domain. Moreover, we introduce a historical prompts ensemble strategy to reduce the disturbance caused by data mutation the mini-batch batch. Concretely, maintain prompt buffer, which is updated by EMA in each iteration of training, and use for inference.
We conduct extensive experiments for the proposed DA-Pro on three mainstream benchmarks: Cross-Wii Weather (Cityscapes Foggy Cityscapes), Cross-Fov (KITTI - Cityscapes), and Sim-to- Real (SIM10K + Cityscapes). The experimental results show that our method brings noticeable improvement and achieves state-of -att--of-the-at performance. Concretely, DA-Pro improves the mAP by 1.9% ~ 3.3% on synthetic and real datasets over the strong Baseline RegionCLIIP. In the best case, we achieve 55.9% mAP on the widely accepted benchmark of Cross Weather, showing remarkable effectiveness in applying the domain adaptive detection head.

---

2
Related Works
Domain Adaptive Object Detection (DAOD) Domain Adaptive Object Detection (DAOD) aims to generalize the object detector trained on the labelled source domain to the unlabelled target domain. The key concept is inferring discriminative visual encoder as the detection backbone by aligning source and target domains in feature space. Previous works for domain adaption 126 42 27 40
33
19 42) have explored minimizing various distance metrics to reduce feature discrepancy. Inspired by them, DA-Faster [] first introduces domain discriminator to extract domain-invarianl features [3] adversarially at the image level. To avoid sub-optimization by directly aligning domain- specific features, DSS (37 suppresses the relevant gradient during backpropagation. TIA [44] divides the detection task into two sub-tasks, localization and classification, and employs task-specific discriminators to optimize them separately. CaCo [1] proposes that categories-agnostic alignment ignores class-specific knowledge and may lead to negative adaptation. Then it explicitly models the intra-class compactness and inter-class separability and aligns features on category hierarchy. SIGMA 22 further models class-conditional distributions of two domains with cross- cross-image graphs and minimizes graph distances to bridge the domain gap.
9
28
Despite the considerable performance of domain alignment, existing methods ignore the domain bias in the detection head. They share the detection head on both domains, inevitably leading to worse discrimination on the target domain. In this work, we introduce visual-languagg models to DAOD, to obtain the domain-aware detection head via the domain-related textual description and the text encoder.
Visual-Ianguage models Recent advances in visual-language representation learning have shown that learning directly from image-text pairs is a promising alternative that leverages a much broader source of supervision. The model inferred by aligning the representation of the image-text pairs is defined as Visual-Language Model (VLM). A representative work is CLIP, which trains a visual encoder and a text encoder using the contrastive loss based on 400 million image-text pairs, demonstrating good generability for the unseen classes. When inference, classifies images using textual representations generated from natural language descriptions, i.e. prompt. Some research [1 adapt VLM into the object detection framework to encode robust knowledge from pre- pre-trained VLM into two-stage detector via aligning the region embeddings of proposals to CLIP'' output. RegionCLIPP 45 further extends VLM to directly learn region-level visual representations by generating region-text pairs as supervision. Some others 8 35 introduce semantic domain concepts via textual prompts to infer robust detectors. In this work, we apply a VLM as the robust detection backbone and utilize the text encoder to build the domain-awart detection head. We adapt [45] with a domain classifier (3] as the Baseline.
and textual features. ViLD [ distills the
Prompt Tuning Prompt tuning has been explored to transfer the pre-trained VLM to its downstream tasks, which utilizes a learnable textual prompt to embed task-relevant information for prediction. A proper prompt could boost the performance significantly with prompt engineering, however, it requires domain expertise and takes an amount of time for words tuning. Inspired by prompt tuning in language tasks, Context Optimization (CoOp) (47) and Conditional Context Optimization (CoCoOp) [46] replaces the hand-crafted prompts with the learnable continuous tokens to automate prompt engineering in end-to-end manner. Furthermore, KgCoOp [4]] explores the forgetting of general textual knowledge during prompt tuning and alleviates it by reducing the discrepancy between the learnable and hand-crafted prompt. DetPro [6] extends CoOP to object detection with context grading scheme to separate proposals in the image foreground for tailored prompt training. However, these methods only develop highly generalizable and discriminative prompt on training domain (single domain). Ignoring the cross-domain difference, their domain agnostic prompts can only capture domain knowledge on the training set. Due to domain bias, they have limited performance on target domain. To enable the prompt to learn cross-domane information, we introduce novel aodain-adaptive prompt and infer robust detection heads on each domain.
3 Methodology
DAOD aims to generalize the detector trained on the annotated source domain to the unlabelled target domain. In this work, we propose a novel Domain-Aware detection head with Prompt tuning (DA-Pro) for DAOD, which employs prompt tuning to generate the domain-aware detection head for
3

---

[P']
Dernabb-levarierr tibeen
Frence tritaing
Seare proppt PT TTrFFrFaarr
Enbeekigg
+
Label
[P*)
Updand wabh gradiom
Classtfer
[Cliss]
Choss
[Class]
[Cker ('rsas 7
Updated with ENA
[Bbox]
Training
Viuua Encoder
Bbox C Head
Labdted Searce thta
Inference
Viveal Enender
Blox
[Class]
Bead
Bbos
Intandee Tarpet
Target Clavitte
Terget presp! (PP) ["IP""c "Fago"
Enooder
Testual
Indeiknng
Porade Label
Letanale Target data
(Class
Tirget Choother
Textial
Prappi bulker "haad
Pronpe buller O'p""CaassPPeea
plocee (Class"
Uncader
Clas Enbooding
Tland-erhhad prosgt
Figure Overview the proposed DA-Pre framework on the Weess-W adaptation scenario.
capturing the domain-spectie and domain-invariant knowledge, shown in Figure [2] Therefore, first give a brief introduction prompt tuning and then introduce the proposed method.
3.1 Preliminaries
Inspired by the effectiveness of CLIP for visual-language learning, we apply CLIP to embed the image and text description. Formally, CLIP consists of the visual encoder f and the text encoder g. For a classification task with K categories, CLIP uses the classes' name generate the textual embedding space with a hand-crafted prompt, a text encoder g(.). and a text tokenizer e(-). where e(-) map the text description into vectors. By denoting the class name of the i-th category "class-i", its corresponding fixed prompt t; is established by using the template "A photo of a [class-i]" to generate the word vector tokens, e.g., = [e(" A photo of [class-i]")], After that, the text encoder further project the fixed prompt into textual embedding space, defined as z, = =ttt).
Given an input image 7 and its class label the visual encoder f firstly extracts the visual embedding f()). Then, the predicted probability PÚū = y|4) of input image on the class y is computed with the distance between visual embedding and text embedding:
p(O bl1) = (p)(s(f()),g(tt)))
E exp(p(f(())).)..... (xt)
where s(-) is the cosine similarity, and a learnable temperature parameter.
Especially, CLIP applies the hand-crafted prompts to generate the class textual embedding, having less discriminative ability for downstream tasks. Recently researchers, such CoOp, show that using the continuous learnable prompt can effectively capture domain-related knowledge. Formally, the hand-crafted prompt is replaced by M learnable tokens V = [v1]] [ After that, the learnable prompt t; is defined as the concatenation of v and the tokenized vector C; e("class-i") of i-th class:
t. [v.][va]..[vvv]I]c)
(2)
Recently, lot of CLIP-based detectors proposed to mine textual knowledge of categories effectively. For example, RegionCLIP (45) learns region-level visual representations by adapting CLIP with the hand-crafted prompt vanilla detector. Furthermore, DetPro (6) further applies the learnable prompt(Eq.) (2) to generate the domain-agnostic representations. However, both the proposed prompts in RegionCLIP 45) and DetPro (6) cannot model the domain knowledge, which is critical for DAOD.
Domain-Adaptive Prompt
Since the existing DAOD methods only consider the visual information, they aim to infer an unbiased visual encoder to generate domain-invariant features and use a unique detection head for recognition. For example, DA-Faster (3) introduces domain discriminator to train a discriminative detection backbone 31 adversarially. However, they ignore the domain bias in the detection head. From cognitive perspective, using shared visual encoder and domain-related detection head can also

---

reduce the domain bias. Furthermore, thanks to the increasing scale of models and data, the pre-trained lasuualan model (VLM) has a high generalization for the downstream task. Consequently, treating the pre-trained model as a visual encoder to extract the highly generalized features and using the domain-sper -ppaai- classifier (detection head) for recognization is a reasonable way for domain- adaptive object detection rather than reducing the domain bias in traditional methods. Therefore, how to propose a domain- -aaaaivvd detection head is the rest problem.
Inspired by prompt tuning such as CoOp, we can generate the dynamic textual class embedding by feeding different prompts, i.e., using the source (target)-related prompt can produce the source (target)-related detection heads for the source (target) domain. Based on the CLIP, the domain- adaptive detection head can be obtained by feeding a domain- adaativee prompt into the text encoder of CLIP. To ensure the Nomain-adaptivy prompt has a high discrimination and generalization ability, an ideal domain- aaaative prompt should satisfy the following conditions: 1) can model the shared knowledge between the source and target domains; 2) it can model the specific knowledge of each domain; 3) contains information that distinguishes or describes each domain; 4) The class information should be embedded for discriminative representation learning.
To achieve the above issue, we design an effective domain-adaptive prompt,
[vilvv] (v.. III.
(3) are
In Eq 3 the domain- domain-adaptive adaptive prompt consists of four components. Firstly, V v== [viVv the domain-invariant tokens with M learnable vectors tokens to learn the common knowledge across two domains. Then, V -v])vv))--[[V are the domain-specifie tokens with N learnable tokens, which are independent across domains to exploit specific knowledge of each domain, i.e., V = V..IvV and y [vi][v]]... [v.[vv]] represent the domain-specific prompt of the source and target domains. Similar to CLIP, c; is the tokenized vector of the i-th class, which is the critical component for boosting the representation discrimination. To constrain the domain-adaptive prompt can fully capture the domain-related knowledge, the domain labels such as the domain-related textual descriptions are used to generate the domain-related vector tokens. Given the domain-related textual description ('domain-d"], ]. the Nomain-related prompts dl, = e("domain-d") is the vector of the hand-crafted textual description "domain-d" for domain Cityscape for example, the domain labels ['domain-d'] for the source domain( CCityscapes') ) and the target domain ('Foggy Cityscape') ) are 'clear' and 'foggy", respectively. Note that the domain- related prompts [dl,] are the significant prompt to control the proposed prompt for inferring the dowain-aware knowledge.
{s,t]. As the Cityscapes to Foggy
By replace the domain-specific prompt V in Eq. 3 with the source-specific prompt V and the target-specific prompt Vr, the final source and target prompts are:
t = [v=]][v --....-. MIlv" i][v [V...[v>>[[e]]]]
(4)
t: = [[v M [[ [[v]=]_..[v [vV]][[]]]]
(5)
3.3 Domain-Adaptive Prompt for object detection
Following 45
we use class-agnostic RPN 43 to propose image regions and extract visual embedding via the visual encoder. Then, the class textual embeddings are generated by feeding the domain-adaptive prompts into the text encoder. As the generated class textual embeddings can be treated as a classifier for computing the contrastive loss with the input visual feature, which can be further used to optimize the domain-adaptio prompt with the given labels.
Given an input image I, the RPN r(-) proposes N, image region boxes R = (r,))=m N. After that, the corresponding visual embedding F = {f, 1m= inferred by the visual encoder f. where f, = ((1,) is a fixed-size feature patch. Then the domain -adativee detection head predicts probability p(ù y|r d, D) of the proposal ry belongs to the class y and domain d:
exp(s(((r,)) g (( \/T) ΣVED ΣX eap(s (
p(0 vlr,, D)
(6)
where s(.) is the cosine similarity. D E {(s), (t), {s.t)) indicates probability is calculated with the class embedding generated by the source prompt {t [ [t U [t/)e.
11 the target prompt tt K 1=
both prompts
5

---

The goal is to make the domain-invarisnn invariant tokens learn the shared knowledge across domains and the domain-specific tokens learn the specific knowledge of each domain. The following two con- straints are proposed to achieve the above goal. Firstly, the domain-invariant prompt is learnable by constraining that the detection head generated by the source-domain prompt and target-domain prompt should both classify the input images as accurately possible. Secondly, the detection head generated by one domain should output higher confidence on its own domain than the other domain when predicting images of its own domain, so as to learn domain-specifie knowledge.
Formally, given an image I E X from domain d along with the generated regions و Nr and its class label {bī ( N- cross-entropy loss as:
1==1- Following the prediction probability calculated by Eq. 6 we denote the
N- = -x= N
p = výlo d, d,)
where D
E {{s), (t) {8,0) indicates probability is calculated with the class embedding generated
by the source prompt {t; [1;);=1 the target [t or both prompts {t;)/= U [t) To learn domain-invariant knowledge, both classifiers should predict ry to the ground truth u7:
L Ldt(o) + Ldttt)
(8)
To learnable domain-specific knowledge, constrain that the classifier of domain d should be more confident on r by minimizing the cross-entropy over the source and target prompts:
6 Ld.
(9)
Therefore, the objective function of the source data X is:
C
iine Cape
(10)
Due to the lack of annotations, the above optimization objective cannot be directly performed on the target domain. We thus generate the pseudo labels for unlabeled target images with considering the powerful zero-shot capability of CLIP. Formally, we apply the hand-crafted template "A photo of [class]" to produce the class having the highest probability as pseudo labels, In particular, the pseudo labels y for the image regions r/J is generated with Eq. 11
v arg max p(ù v/rr))
(11)
where the probability is computed via Eq
As untrusted pseudo would harm the learning of the source classifier, we use the pseudo labels whose probabilities are higher than 7. Similar to Eq. 7 the cross-entropy loss computed with target data is denoted as:
N Σ 1(pū-= (1) logr(ū =
LID -xD=Ex+
(16)
N
where the I(-)) is an indicator function. Meanwhile, we minimize the information entropy of the logits achieve high-confident classification.
Lent = Ex
1 N₂
Σ I(plū uf -)(( vflr,, (t)) log t (f))
(17)
Considering that inaccurate pseudo labels may be detrimental to the training of the source classifier, only them to optimize the target classifier. Overall, the objective on the target data X is:
L: Lu,4 + Lefts,) + Lent
(12)
The overall training objective is:
L= + XLe.
(13)
where À is used to balance the effect of L, in L.

---

3.4 Prompt Ensemble
During prompt tuning, all the visual and text encoder parameters are frozen. In each iteration, only the learnable prompts are updated via backward gradients. However, data mutation across mini-batches may cause prompt perturbations
the early stage of training, harming the prompt tuning. To alleviate this phenomenon, a prompts ensemble strategy is proposed to improve the stability of prompt tuning by averaging all the history prompts. Concretely, we maintain prompt buffer B = [bf. b ..., by bj. b D2,..., bjk about the learnable prompt in DA-Pro. In each iteration iter of training, buffer Byter can not be optimized with gradients, and can only be updated by Exponential Moving Average (EMA) according to the weights of itself and the learnable prompt titer Titer = [ti. tg. ..., tg t1,
}
(14) where is the ensemble ratio of history prompts. Considering that the buffer needs to quickly forget the inaccurate parameters learned in the early stage, we set dynamic updating ratio min - Mer+I 0.99]. As EMA reduces the variance of the model, the prompt ensembled strategy infers more robust prompt After training, B is saved and used for inference.
Biter B (1 T)Trttr
4
Experiment
This section evaluates our DA-Pro on mainstream DAOD scenarios, including the Weashe, Cross-Fov, and Sim-to-Real. Further ablation studies are conducted validate the effectiveness of the proposed domain-adaptive prompt, training scheme, and prompt ensemble strategy.
4.1 Dataset
Cross-Weather Weather Cityscapes 4 is a large-scale dataset that contains diverse images recorded in street scenes. It divided into 2,975 training and 500 validation images, annotated with classes. Foggy Cityscapes [32] is a synthetic foggy dataset that simulates fog with three distinct densities on Cityscapes, containing 8,925 training images and 1,500 validation images. We take the training set of Cityscapes the source domain and the training set of foggy Cityscapes the target domain, evaluating Cross-Wess Weather adaptation performance on the 1500-sized validation set in all 8 categories.
Cross-Fov KITTI 10 is a vital dataset in self-driving, containing 7,481 images with car annotations. Images are captured by driving in rural areas and on highways, yielding the Field of View (FoV) disparity with Cityscapes. For the Cross-Fov adaptation scenario, we migrate KITTI to Cityscapes on the car category and validate performance on Cityscapes.
Sim-to-Real SIM1OK 18 is a synthetic dataset containing 10,000 images with car annotations rendered from the video game Grand Theft Auto V. We perform simulated environment to real worldworld adaptation on SIM10K and Cityscapes datasets.
4.2 Implementation Details
We adapt RegionCLIP with a domain classifier [3 as bascline, and initialize the default prompt with "A photo of a [comas][d We use ResNet-50 [2] as the visual encoder and Transformer (+ as the text encoder, and initialize with the pre- pre-trained CLIP model. Configurations of the detector and image pre-pros pre-processing follow the default settings in [3 Following B. we first pre-train the baseline model with classification loss, regression loss, and adversarial loss. Then for adaptation, one batch of source images with ground truth and one batch of target domain images are forwarded calculate the supervising and self-training loss. We fix
to the proposed DA-Pro in each iteration the length of learnable tokens M, N to 8, 8, respectively. The hyperparameter À is set to 1.0. We set the batch size of each domain 2 and use the SGD optimizer with warm-up learning rate for training. We take the mean Average Precision (mAP) with a threshold of 0.5 as the evaluation metric. All experiments are deployed on Tesla V100 GPU.
4.3 Comparison with existing methods
We introduce various state-of-the-art DAOD approaches for comparison, including DA-Faster +
VDD (38. DSS (37), MeGA (36), SCAN 21, TIA [44), SIGMA (22), and AT (24..
7

---

Table I: Comparison with existing methods three adaptation tasks, for Weash-r adaptation Sityscapes->Fog Cityscapes (C-F)), Cross-Fov adaptation Kiyst-Cssyscaps (K+CC) and Sim- to- to-Real adaptation SIM10K HIM10K-CCtyscaaa Citysccapes (S SCC mAP: mean Average Precision (%).
C-
K C
S-+ C
Methods
Person
Rider
Car
Truck
Bus
Train
Motor
Bicycle
mAP
MAP
mAP
DA-Faster 3
29.2
40.4
43.4
19.7
38.3
28.5
23.7
32.7
32.0
41.9
38.2
VDD B8
33.4
44.0
51.7
33.9
52.0
34.7
34.2
36.8
40.0
DSS 37
42.9
51.2
53.6
33.6
49.2
18.9
36.2
41.8
40.9
42.7
44.5
MeGA
37.7
49.0
52.4
25.4
49.2
46.9
34.5
39.0
41.8
43.0
44.8
SCAN [21
41.7
43.9
57.3
28.7
48.6
48.7
31.0
37.3
42.1
45.8
52.6
TIA [44
52.1
38.1
49.7
37.7
34.8
46.3
48.6
31.1
42.3
44.0
SIGMA 22
44.0
43.9
60.3
31.6
50.4
51.5
31.7
40.6
44,2
45.8
53.7
AT 24
56.3
51.9
64.2
38.5
45.5
55.1
54.3
35.0
50.9
Baseline
51.8
59.0
67.4
36.8
59.5
50.6
39.7
55.9
52.6
59.5
60.8
DA-Pro 55.4
62.9
70.9
40.3
63.4
54.0
42.3
58.0
55.9
61.4
62.9
Table Ablation studies (%) on Weather adaptation scenario Cityscapes->> Cocapnes-Foogee -Fggy Cityscapes. AP50 evaluates mAP on detection boxes with IoU 2 0.5, and 2 0.75 for AP75. AP averages AP50 AP95 with step
Prompt Design
M N
Prompt Ensemble
AP
AP50
AP75
A photo of [class][domain]
28.5
52.6
28.7
[vslv [vrr][c.]
0
28.9
53.0
28.5
[v [
[c [dl.]
16
0
29.2
53.8
29.3
[v
[dla]
0
16
28.9
53.1
28.7
[
[.....v [v
I [dld
8
8
31.2
55.5
30.5
[vil[ [
[c.][dl
8
8
V
31.9
55.9
32.0
vvil
S
[dl
4
4
31.1
55.0
30.2
vills
[c [ca]dda
12
12
31.4
55.4
31.3
[c. [dla]
[da]
16
16
31.3
55.3
30.6
Cross- Weather Adaptation Scenario As shown in Table 1 (C-+F), the proposed DA-Pro achieves the highest mAP over 8 classes of 55.9%, outperforming SOTA AT [24] by a remarkable margin of 5.0%. Compared with existing methods, our method significantly improves 6 categories (i.e. rider, car, truck, bus, train and bicycle) ranging from 1.8% 16.2%. The improvement is particularly significant on the previously poor categories, e.g. 11.0% on Rider and 16.2 16.2% on Bicycle, which may have harder semantics for adaptation. Though promising mAP of 52.6%, DA-Pro still improves Baseline by 3.3%. Concretely, DA-Pro promotes 2.1 ~ 3.9% on all 8 categories via learning domain- adaptive prompt. The superior performance demonstrates the proposed DA-Pro can improve the generalization ability of the detector the target domain.
Cross-Fov Adaptation Scenario T also shows that the DA-Pro improves hand-crafted prompt Baseline by 1.9%. And the proposed method reaches the best remarkable 61.4% mAP, outperforming SOTA methods in the Cross-Fov detection task. According (37), K-+C adaptation faces more complicated shape confusion than C-F, which requests higher discriminability of the model. The considerable performance improvement confirms that the proposed method can efficiently generate domain- dowain-aware aware detectors with high discrimination.
Sim-to-Real Adaptation Scenario Unlike the Cross- Weather and Cross-For adaptation, the Sim-to- Real task has a wider domain gap at semantic level. As presented in Table (S-C), the proposed method gains 2.1% 1 on Baseline and peaks at 62.9%, surpassing SOTA by 9.2%. It further demon- strates that our strategy is robust in not only appearance but also harder semantics adaptation tasks.
4.4 Ablation Studies
We report detailed ablation studies (Table conducted on the Cross-Weathee Weather adaptation scenario, to validate the effectiveness of DA-Pro under different configurations.
8

---

Table The influence (%) of loss design on Cross- Weather adaptation scenario Cityscapes Cityscapes. Total number of tokens is set to 16. stands for the prompt is not compatible with the loss. Without the historical prompt ensemble strategy.
A photo of a [class][domain] L.,(s) + Le,11 C2,(0) Ca.lad) + Le, Lene Le,(o) mAP
Prompt Design
52.6
[v.]
53.0
[vilvv]] [r..
[[c][[[a]]]
53.5
vil
Virlee
53.8
vil .- Vir c. dl.
53,8
Villvá
53.7
villv ...
52.9
vi]l
53.1
v vi
52.7
52.9
[ ...
[v
[c]
54.4
v vis
[c.]
dL.
54.8
v [v
...
vi
vi
[c. dld
55.2
Mi
vv
vill
[c.] dla
55.5
M
[
[[a]
53.4
Comparison on Prompt Design In this section, we analyze the performance of different prompts. As shown in line 1 ~ 5 of Table we compare the performances on the following four prompt designs: the pre-defined prompt "A photo of a [class][domain]]. the CoOp-style learnable prompt, and the tokens or domain-specific tokens with domain-related textual description. The CoOp-style prompt with 16 learnable domain-invariant tokens improves 0.4% on AP and AP50 over the hand-crafted prompt. Introducing related textual description further boosts 0.3 0.8% on three metrics. This exhibits that learning domain-invariant tokens improves the discrimination of the target detection head. When directly applying domain-spectie tokens, it only gains limited improvement over the pre-defined prompt. This shows that the source domain information cannot be transferred the target domain only using domain-specific tokens. Diving the 16 tokens into domain invariant and 8 Noomain-specifie tokens further gains extra 0.3 ~ 1.0% improvements. We suppose that based on shared knowledge across domains, learning Nomain-speeiff knowledge leads to more discriminative detection head on each domain. These results reveal that compared with the fixed or the domain- aggnosicc prompt, the domain- prompt has higher ability to embed domain information for prediction. Moreover, both domain-invariant tokens and domain-specific tokens bring improvements.
Comparison on Loss Function Table 3 shows that each term of loss function contributes to im- provement in the adaption performance. To reduce the influence of other factors, we do use the historical prompt ensemble strategy. With L.,(s) + LL.(t). four types of learnable prompts have improved 0.3 ~ 1.8% on the hand-crafted prompt. Combined with L..(t)- which commands the target prompt to classify the source image accurately, further gains 0.2 ~ 0.4% on three prompts distinguished across domains. This reveals that by constraining the detection heads in the source and target domains to classify the input image as accurately as possible, the prompt can learn more domain-invariant invariant knowledge. Further, though introduce [..(..)) + (s.(s.t) has few effects on prompt with only
tokens, it improves 0.4% on the Nomain-adaptiv prompt. tokens
We conclude that in two respect. On one hand, just using cannot model shared knowledge between domains and Nomain-specifet knowledge at the same time. On the other hand, constraining the detection head generated by one domain to output higher confi- dence in its own domain than the other domain contributes to learning domain-specific knowledge. Moreover, adding Lent brings extra 0.3% improvement. However, introducing L. (=) suffers 2. 2.1% degradation, showing that directly optimizing the source classifier with pseudo target labels could do harm to the learning process.
invarian//dmmain-speee
Comparison on Prompt Ensemble To verify the effectiveness of the proposed historical prompt ensemble strategy, we introduce to the domain-adaptive prompt with 8 and 8 domain-specific tokens. Line of Table 2 shows that it gains improvements of 0.7%, 0.4% and
tokens

---

Ground Truth
(c.1)
(c)
F [e
mAP: 63.9
(b) "A photo of [chass[demmiii" mAP: 54.6
(b.1)
(d.1)
(6) [ He][].
MAP:
Figure 3: Detection comparison of different prompts on the Cross- wess-er adaptation scenario. We visualize the ground truth (a) and the detection results of three prompts: (b) "A photo of [class][domain]" (c) [V][ [d][]]]]] (d) [V[[]] d[a]. (a.1) (b.1) (c.1) (d.1) are roomed from the same region of the image (a) (b) (c) (d). mAP: mean Average Precision on the example image
1.5% AP, AP50 and AP75. This reveals ensembling the history prompt can reduce the effect of prompt disturbance and infer a robust prompt, thus improving detection performance.
Comparison on Prompt Length Further, we explore the impact of prompt length, where M for domain-invariant tokens and N for domain-specinf tokens on DA-Pre with full configuration line 6 9 of Table 2 We vary the length from (4, 4) to (8, 8) to (12, 12) to (16, 16). DetPro[6] has shown that a longer prompt may cause over-fitting to base categories in open-set object detection. Similarly, we conclude that a shorter prompt is less discriminative, while a too-long prompt is prone to overfitting to the source domain in DAOD. Thus, we set the token length as (8, 8) DA-Pro.
4.5 Visualization
In Fig. 3. we present the comparison on the target domain among the ground truth boxes (a) and the detection boxes using different prompts. (a. (b. Iddc.))dd.) are zoomed from the same region of images (a)() for better view. Fig. afa.)) indicates four objects in the cropped region: a car and bicycle on the left, and two overlapping persons on the right. Both hand-crafted and learnable prompt models detect the car correctly. However, it is difficult for the hand-crafted prompt to describe domain information, like weather conditions, accurately. Thus, the hand-crafted prompt misses some objects the right side of Fig. b)), and the bicycle and one of the persons in Fig. 3 b.11. ). In Fig. (). the domain-invariin prompt improves 9.3% mAP compared with the hand-crafted prompt. But still suffers worse discrimination due to insufficient domain representation learning. In Fig. c.. the domain-invariant prompt misclassifies the bicycle as a rider even confident with 93%, and also ignores one person. The proposed domain-adaptive prompt correctly detects all objects in the cropped region Fig d.).). 1). By learning domain information, the domain-specific tokens enable the model to perform confident predictions on the bicycle (79%) and persons (100%, 89%). These comparison results reveal the effectiveness of the proposed domain domain-adaptive prompt in DA-Pro.
5 Conclusion
In this paper, we propose a novel DAOD framework named Domain Aware detection head with Prompt tuning (DA-Pro). Serving a pre-trained visual-language model as the robust detection backbone, applies the learnable comminnadaptie prompt generate the diseriminative detection head for each domain. As the learnable part is designed to capture the domain-invarie specific knowledge, we proposed two constraints over the source and target domains to guide the optimization. Moreover, prompt ensemble strategy is also proposed to reduce the effect of prompt disturbance. Comprehensive experiments over multiple cross-domain adaptation tasks demonstrate that using the domain-adaptiv prompt can produce domain-aware detection head with more discrimination for domain-adaptive object detection.
and domain-
Acknowledgments and Disclosure of Funding
This work is partially supported by the NSF of China (under Grants 92364202, 61925208, 62102399, 62102398), Beijing Natural Science Foundation (4222039), CAS Project for Young Scientists in Basic Research(YSBR-O0 and Xplore Prize.
10

---

References
[1] Chaoqi Chen, Jiongcheng Li, Zebiao Zheng, Yue Huang, Xinghao Ding, and Yizhou Yu. Dual bipartite graph learning: A general approach for domain adaptive object detection. In ICCV, pages 2703-2712, 2021.
[2] Chaoqi Chen, Jiongcheng Li, Hong- Yu Zhou, Xiaoguang Han, Yue Huang, Xinghao Ding, and Yizhou Yu. Relation matters: foreground-awanl graph-based relational reasoning for domain adaptive object detection. TPAMI, 45(03):3677-33))
2023.
[3] Yuhua Chen, Wen Li, Christos Sakaridis, Dengxin Dai, and Lue Van Gool. Domain adaptive faster for object detection in the wild. CVPR, pages 3339-3348, 2018.
[4] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bert Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
[5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenhom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakoh Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[6] Yu Du Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open- open-vocabulary object detection with vision-language model. In CVPR, pages 14084-14093, 2022.
[7] Mark Everingham, SM Ali Eslami, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. IJC,, 111:98-136, 2015.
Mohammad Fahes, Hunn-Hug Vu, Andrei Bursue, Patrick Pérez, and Raoul de Charette. P (10) da: Prompt-driven zero-shot domain adaptation. arXiv preprint arXiv:2212.03221, 2022.
Yaroslav Ganin, Eygeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-advvrsanr adversarial training neural networks. JMLR, 17(1): 2096-2030, 2016.
[10] Andreas Geiger, Philip Lenz, and Raquel Urtasun, Are we ready for autonomous driving? kitti vision benchmark suite. In CVPR, 2012.
[11] Xiuye Gu, Tsung-Yi Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In ICLR, 2022.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[13] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision leameers. In CVPR, pages 16000-16009, 2022.
[14] Jiaxing Huang, Dayan Guan, Aoran Xiao, Shijian Lu, and Ling Shao, Category contrast for unsupervised domain adaptation in visual tasks. In CVPR, pages 1203 1214, 2022.
[15] Naoto Inouc, Ryosuke Furuta, Toshihiko Yamasaki, and Kiyoharu Aizawa. Cross-domain weakly- supervised object detection through progressive domain adapiation. In CVPR, pages 5001-5009, 2018.
[16] Chao Jia, Yinfei Yang. Ye Xia. Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le. Yun- Yun-Hsuan Sung. Zhen Li, and Tom Duerig, Scaling visual and vision languae representation learning with noisy text supervision. In ICML, pages 4904-4916 PMLR, 2021.
[17] Yifan Jiao, Hantao Yao, and Changsheng Xu. Dual instance-consistent network for cross-domain object detection. IEEE Transactions Puttern Analysis and Machine Intelligence, 2022.
[18] Matthew Robbeson,,, Charles Barto, Rounak Mehta, Sharath Nittur Sridhar, Karl Rosaen, and Ram Vasudevan. Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks? In ICRA 20/7, pages 746 753. IEEE, 2017.
[19] Congcong Li, Dawci Du, Libo Zhang, Longyin Wen, Tiejian Luo, Yanjun Wu, and Pengfei Zhu. Spatial attention pyramid network for unsupervised domain adaptation. In ECCV, pages 481-497. Springer. 2020.
[20] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang. Jeng-Neng Hwang, et al. Grounded language-image pre- pre-training. In CVPR, pages 11965-10977 2022.
11

---

[21] Wuyang Li, Xinyu Liu, Xiwen Yao. and Yixuan Yuan. Scan: Cross domain object detection with semantic conditioned adaptation. In AAAI, volume page 2022.
[22] Wuyang Li, Xinyu Liu, and Yixuan Yuan, Sigma: Semantic-complete graph matching for domain adaptive object detection, In CVPR, pages 5291-5300, 2022.
[23] Wuyang Li, Xinyu Liu, and Yixuan Yuan. Sigma++: Improved semantic-complete graph matching for domain adaptive object detection. TPAMI, 45007))9022-9440, 2023.
[24] Yu-Jhe Li. Xiaoliang Dai, Chih-Yat Ma, Yen-Cheng Liu, Kan Chen, Bichen Wu, Zijian He, Kris Kitani, and Peter Vajda. Cross-domain adaptive teacher for object detection. In CVPR, pages 7581-7590, 2022.
[25] Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongic. Feature pyramid networks for object detection. In CVPR, pages 2117-2125 2017.
[26] Mingsheng Long. Yue Cao, Jiammin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In ICML, pages 97-105. PMLR, 2015.
[27] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Unsupervised domain adaptation with residual transfer networks. NeurIPS, 29, 2016.
[28] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. Deep transfer lcarning with joint adaptation networks. In ICML, pages 2208-2217. PMLR, 2017.
[29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In /CMI,, pages 8748-8763. PMLR, 2021.
[30] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR, pages 779-788, 2016.
[31] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. NewrIPS, 28, 2015.
[32] Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Semantic foggy scene understanding with synthetic data, NJV, 126(9):973-99) 992, Sep 2018.
[33] Rui Shao, Xiangyuan Lan, and Pong C Yuen. Feature constrained by pixel: Hicrarchical adversarial deep domain adaptation. In ACMMM, pages 220-228, 2018
[34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurlPS, 30, 2017.
[35] Vidit Vidit, Martin Engilberge, and Mathicu Salzmann. Clip the gap: A single domain generalization approach for object detection. arXiv preprint arXiv:2301.05449 2023.
[36] Vibashan Vs. Vikram Gupta, Poojan Oza, Vishwanath A Sindagi, and Vishal M Patel. Mega-cda: Memory guided attention for calegory-awan unsupervised domain adaptive object detection. In CVPR, pages 4516-4526, 2021.
[37] Yu Wang. Rui Zhang, Shuo Zhang, Miao Li, Yangyang Xia, XiShan Zhang, and ShaoLi Liu. Domain- specific suppression for adaptive object detection. In CVPR, pages 9603 9612,961.222 2021.
[38] Aming Wu, Rui Liu, Yahong Han, Linchao Zhu, and Yi Yang. Vector-decomposed disentanglement for domain-invariant object detection. In ICCV, pages 9342-9351, 2021.
[39] Minghao Xu, Hang Wang, Binghing Ni, Qi Tian, and Wenjun Zhang. Cross-domain detection via graph-induced prototype alignment. In CVPR, pages 12355-12364, 2020.
[40] Hongliang Yan, Yukang Ding, Peihua Li, Qilong Wang, Yong Xu, and Wangmeng Zuo. Mind the class weight bias: Weighted maximum mean discrepancy for unsupervised domain adaptation. In CVPR, pages 2272-2281, 2017.
[41] Hantao Yao, Rui Zhang, and Changsheng Xu. Visual-language prompt tuning with knowledge-guidet context optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition, pages 6757-6767, 2023.
[42] Xu Zhang, Felix Xinnan Yu, Shih-Fu Chang, and Shengjin Wang. Deep transfer network: Unsupervised domain adaptation. arXiv preprint 150.00503.00511, 2015.
12

---

[43] Yixin Zhang. Zilei Wang. and Yushi Mao. Rpn prototype alignment for domain adaptive object detector. In CVPR. pages 12425-12434, 2021.
[44] Liang Zhao and Limin Wang. Task-specific inconsistency alignment for domain adaptive object detection. In CVPR. pages 14217-14226, 2022.
[45] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, al. Regionclip: Region-based language-image pretraining. In CVPR, pages 16793 16803, 2022.
[46] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. CVPR, pages 16816-16825, 2022.
[47] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning prompt for vision-language models. IJCV, 130(9):2337-2344 2022.
[48] Qianyu Zhou, Qigi Gu, Jiangmiao Pang. Zhengyang Feng, Guangliang Cheng, Xuequan Lu, Jianping Shi, and Lizhuang Ma. Self-adversarial disentangling for specific domain adaptation. TPAMI, 45(7):8954-8868, 2023.
13

---

Appendix
5.1 Additional Experimental Results
In order further verify the effectiveness of DA-Pro, we evaluate performance on three harder benchmarks: Pascal to Clipart, Pascal to Watercolor and Pascal to Comic, Pascal VOC [ is a large- scale real-world dataset annotated with 20 classes, which contains 2007 and 2012 subsets. Clipart [T5] is collected from the website with 1000 comical images, providing bounding box annotations with the same 20 classes as Pascal VOC. Watercolor and Comic [TS] both contain 1000 training images and 1000 test images in art style, sharing 6 categories with Pascal VOC. Three benchmarks enable the method to be evaluated under more challenging domain shifts and multi-class problem scenarios. The comparison is shown in Table 4 Our DA-Pro surpasses the SOTA method (SIGMA+ with ResNet- 101) with weak backbone (ResNet-50) on all three additional benchmarks, showing the effectiveness of DA-Pro.
Table 4: mAP(%) comparison with existing methods on Pascal Watellolor,, Pascal->Clipan and Pascal Comic tasks.
Methods
Pascal + Watercolor
Paspal-<Clipart
Pascal->Comic
DBGL[1 (ResNet-101)
53.8
41.6
29.7
Baseline (ResNet-50)
54.8
43.4
40.6
55.7 FGRR 2 (ResNet-101)
43.3
32.7
SIGMA+
23 (ResNet-101)
57.1
46.7
37.1
DA-Pro (ResNet-50)
58.1
46.9
44.6
5.2 Hyperparameter Search
To select hyprraarameters for our loss functions, we perform experiments of different choices of the weight values X(C,). We conduct the experiment DA-Pro both Messcapes-Foogeee -Foggyottyscapyypp Cityscapes and Sim10K Ciitssppps adaptation scenarios. Table Tablee5shooe shows that our DA-Pro is robust with different settings of Å. Considering that the best result is achieved under 1.0, take 1.0 the default.
Table 5: The influence (%) of for the loss with target data Lt.
Cityscapes + Foggy DityyCityssapes
0.25
0.5
1.0
5.0
10.0
mAP
55.6
55.8
55.9
55.7
55.4
Sim Siml0K-a Cityscapes
À
0.25
0.5
1.0
5.0
10.0
mAP
62.5
62.9
62.9
62.4
62.8
5.3 Options of the pseudo labels
We optimize the Nomain-adaptio prompt with annotated source data and further distill CLIP's remarkable zero-shot classification ability to the detection head of the target domain. A naive way is generate classification probabilities, i.e.soft labels, with the fixed prompt "A photo of [class]. However, aligning the model's predictions with these probabilities will result in the learnable prompt converging the hand-crafted prompt. Unlike probabilities, generating pseudo labels with threshold, hard labels, does not demand learning the relative distances to each category provided by hand-crafted prompts. Instead, they require the prompt to be as close to the correct category and as far from the incorrect categories as possible, thereby learning a more discriminative prompt. To verify this, we conduct experiments on the three benchmarks (C-+ F, K- C, S-+ C). As shown Table 6 replacing pscudo labels with probability supervision suffers 0.6 1.6% degradation performance.
14

---

Table 6: The comparison (%) of different options of pesudo labels on adaptation scenario Cityscapes -oggy Cityscapes.
Options
Cityscapes-Ale >Foggy Cityscapes
KITTI HITTI->Cityscper
Sim10K -iityscppesss
probabilities
54.3
60.8
62.1
pseudo label (ours)
55.9
61.4
62,9
5.4 Effect of the domain-related textual tokens dla
The domain-related textual tokens aim to leverage textual descriptions of domains and introduce hand-crafted prior information to facilitate more efficient learning of domain-specifie tokens. The hand-crafted token offers solid initialization which is discriminative. On this foundation, domain- specific tokens further learn the bias between the two domains, further enhancing the prompt's discrimination. The combination both hand-crafted and learnable tokens yields superior results. We have conducted additional experiments three different scenarios. The results are shown in [[v][v]] [v&]....&v&cc
Table 7 Without dly token, using prompt [vi]vv]]... I mAP. Experimental results demonstrate that dI2 assists in the convergence of domain-speciff tokens and enhances the discrimination of the prompt.
suffering 0.7 1.7%
The influence (%) of the domain-rrlatedd textual tokens dla on Cityscapes-Fogge Cityscapes, -ittI--Cttyscpps and Sim10K Baal0KK--ityycappee
Table 7:
Prompt Design
C-F
K-+C
S->C
[vfllvs)...vi -v.
54.3
60.8
62.1
[v]][v [v [[c
55,9
61.4
62.9
5.5 Effect of the Historical Prompt Ensemble Strategy
To verify the effect of the proposed historical prompt ensemble strategy, calculate the mAP of whether apply prompt ensemble under different loss function. We conduct the experiment on DA-Pro Cityscapes=FoogyCiyyy Cityssappss scenarios. Table g shows the effect of our historical prompt ensemble strategy. It improves the performance all forms of learnable prompts without any inference overhead. In particular, the best case of improvement is 0.4% for the domain-adaptive prompt.
Table 8: The influence (%) of the historical prompt ensemble on Meesher adaptation scenario Cityscapes- Cityscapes. Total number of tokens is set 16.
Historical Prompt Ensemble
C.,(,) Li,(
C.,(2.)) + C(,10,))
Lent
54.4
54.8
55.2
55.5
54.7
55.0
55.6
55.9
Gain
0.3
0.2
0.4
0.4
15


Reviews:
[{"review_id": "jlEJPD8Iu2", "reviewer": "Reviewer_x91n", "summary": "This paper introduces VLM to domain adaptive object detection. To be specific, this paper uses highly generalized VLM as detection backbone, and adapts detection head instead. To learn the domain-invariant and  domain-specific knowledge, this paper extends the prompt to domain-invariant and  domain-specific ones, which are optimized with corresponding loss. Besides, to adapt domain-adaptive prompt for unsupervised object detection, this paper uses the CLIP to get pseudo labels and Prompt Ensemble to stabilise the training. Experiments on DAOD benchmark show better performance.", "questions": "- Source and Target classifier in Figure 2 means similarity calculation, and drawn entity is misleading\n- What is the motivation of prompt dl_d. I mean it seems to play the same role with domain specific prompts.\n- An important comparison method [1] is missing, which as far as I know, is the SoTA of DAOD, which pushes the K2C performance to a very high level.\n- CLIP initialized visual encoder is an image level extractor, I am afraid if it is suitable for detection.\n- I am curious if the learned prompts are independently semantic. For example, K2C and S2C both learned domain specific prompts of domain C, and what performance will get if exchanging the corresponding prompts.\n", "limitations": "n/a", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper shows that using highly generalized VLM seems as a promising direction with relatively high performance", "weaknesses": "- The comparison method and evaluation could be improved. See questions for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Comment：\n\nWe sincerely thank you for the valuable comments. We are encouraged to see that our work is recognized as a promising direction. We will explain your concerns point by point.\n\n**Q1: Source and Target classifier in Figure 2 means similarity calculation, and drawn entity is misleading.**\n\nA1: Thanks for your nice suggestion. The classifier in Figure 2 does indeed correspond to the similarity calculation from image embeddings to text embeddings. We intend to revise this entity in the manuscript, changing it to a bar graph to dipict the class similarity.\n\n**Q2: What is the motivation of prompt dl_d. I mean it seems to play the same role with domain specific prompts.**\n\nA2: Thanks for raising an important point. \n\nOur motivation is to leverage textual descriptions of domains and introduce hand-crafted prior information to facilitate more efficient learning of domain-specific tokens. In order to learn highly discriminative prompts, CoOp adopts a prompt design of [learnable prompt, [class]], tuning the learnable prompt based on the class text description provided by humans. Inspired by this, we introduce hand-crafted \"dl_d\" to incorporate domain-related textual descriptions. The hand-crafted token offers a solid initialization which is discriminative. On this foundation, domain-specific tokens further learn the bias between the two domains, further enhancing the prompt's discrimination. The combination of both hand-crafted and learnable tokens yields superior results. We have conducted additional experiments on three different scenarios. Without dl_d token，using prompt t^d_i=[v^c_1][v^c_2]…[v^c_M][v^s_1][v^s_2]…[v^s_N][c_i] suffering 0.7~1.7% mAP. Experimental results demonstrate that dl_d assists in the convergence of domain-specific tokens and enhances the discrimination of the prompt.\n\n| Prompt Design\\Benchmark | C→F | K→C | S→C |\n| --- | --- | --- | --- |\n| [v^c_1][v^c_2]…[v^c_M][v^s_1][v^s_2]…[v^s_N][c_i] | 54.9 | 60.7 | 61.2 |\n| [v^c_1][v^c_2]…[v^c_M][v^s_1][v^s_2]…[v^s_N][c_i][d_i] | **55.9** | **61.4** | **62.9** |\n\n**Q3: An important comparison method [1] is missing, which as far as I know, is the SoTA of DAOD, which pushes the K2C performance to a very high level.**\n\nA3: Thanks for the suggestion. It seems like you forgot to specify what method [1] refers to. To this end, we supplement a series of SOTA methods employing the same detection framework (Faster R-CNN) as our approach. Among these, the method PT (ICML 2022) indeed elevates the K2C performance significantly. However, in comparison with PT, our approach achieves even better performance.\n\n|Method\\Benchmark  | C→F | K→C | S→C |\n| --- | --- | --- | --- |\n| MGA | 44.3 | 45.2 | 49.8 |\n| TDD  | 43.1 | 47.4 | 53.4 |\n| PT | 47.1 | 60.2 | 55.1 |\n| DA-Pro | **55.9** | **61.4** | **62.9** |\n\n*MGA: Zhou W, Du D, Zhang L, et al. Multi-granularity alignment domain adaptation for object detection[C]. In CVPR, 2022.*\n\n*TDD: He M, Wang Y, Wu J, et al. Cross domain object detection by target-perceived dual branch distillation[C]. In CVPR, 2022.*\n\n*PT: Chen M, Chen W, Yang S, et al. Learning Domain Adaptive Object Detection with Probabilistic Teacher[C]. In ICML, 2022.*\n\n**Q4: CLIP initialized visual encoder is an image level extractor, I am afraid if it is suitable for detection.**\n\nA4: Thanks for your concern. In our work, we apply RegionCLIP as the visual encoder rather than CLIP. Due to CLIP's focus on learning image-text pairs, it lacks the capability to localize regions within images indeed. Therefore directly employing CLIP as the visual encoder in a detection framework would lead to unacceptable performance degradation. To address this challenge, RegionCLIP establishes region-text pairs and achieves alignment in the local feature space, effectively integrating CLIP into the object detection framework. \n\n**Q5: I am curious if the learned prompts are independently semantic. For example, K2C and S2C both learned domain specific prompts of domain C, and what performance will get if exchanging the corresponding prompts.** \n\nA5: Thank you for raising this intriguing question. \n\nSemantic information is jointly represented by domain-invariant and domain-specific tokens for a given domain. The domain-specific token is learned by capturing the differences between domains based on the domain-invariant token. And the domain-invariant token captures the domain-shared knowledge, which is determined simultaneously by both the source and target domains. As a result, for the same target domain, if the source domain varies, the learned domain-invariant tokens also differ, leading to discrepancies in learned domain-specific tokens. We exchanged the learned domain-specific tokens between K2C and S2C and observed a severe performance drop in inference results. In fact, K2C and S2C possess distinct domain-shared and domain-specific knowledge. In this scenario, the learned domain-invariant and domain-specific tokens are distinct in K2C and S2C. Hence, directly swapping learned tokens leads to a significant performance decrease.\n\n| Benchmark\\Inference prompt | Default(DA-Pro) | exchange domain-specific token |\n| --- | --- | --- |\n| K→C | 61.4 | 17.2 |\n| S→C | 58.7 | 12.9 |"}, {"review_id": "ku3xbuox7g", "reviewer": "Reviewer_eDFv", "summary": "This paper designs a novel Domain-Aware Detection Head with Prompt Tuning (DA-Pro) framework for domain adaptive object detection. The motivation is learning the discriminative detector for each domain instead of reducing the domain bias as in the traditional DAOD methods. Specifically, the authors leverage the vision-language model (VLMs) to build domain-aware detection head. The domain adaptive prompt consists of the domain-invariant tokens, domain-specific tokens, and the domain-related textual description along with the class label. The experimental results seem to be effective on serval DAOD benchmarks.", "questions": "1.\tIn line 260, the baseline is RegionCLIP and DetPro with a domain classifier, so are the baseline results reported in the Tables based on RegionCLIP, DetPro, or both? It is recommended to clarify the setting of the baseline model.\n2.\tWhat is the initialization for the detection model (e.g., the visual encoder and Bbox Head in Figure 2)? Which datasets are used to pre-trained the detection model? How about the backbone only being pre-trained in ImageNet as previous DAOD methods (e.g., DA-Faster RCNN, SIGMA, AT, etc.)?\n", "limitations": "The ClipRegion and DetPro have the ability to handle open-vocabulary object detection, this work degrades them to close-set object detection and limits the real-world application of ClipRegion and DetPro.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tThis paper addresses an important research problem and the prompt tuning in vision is a popular direction.\n2.\tThis paper is well-written and the proposed method is easy to comprehend.\n3.\tThe experimental results appear to be effective compared to previous approaches.\n4.\tThe motivation for designing a domain-aware detection head for domain adaptive object detection is reasonable and the technical implementation is easy to follow.\n", "weaknesses": "1.\tOne major concern of this paper is the differences between the proposed prompts and those in COOP and DetPro. They utilize similar learnable prompts for the text prompts. In line 145, this paper claim “both the proposed prompts in RegionCLIP [39] and DetPro [4] cannot model the domain-specific knowledge”. In my opinion, RegionCLIP just fills concepts into prompts so that it does not model domain-specific knowledge. However, DetPro learns the prompt in a certain domain and could indeed learn domain-specific knowledge. \n2.\tThe experimental results are not adequate. There are two benchmarks (KITTI to Cityscapes and SIM10K to Cityscapes) that are only evaluated on the ‘car’ category and do not adequately show the effectiveness of the proposed method.\n3.\tWhy the bounding box head is shared for both domains? The proposed domain-aware detection head may be further improved for the bounding box head.\n4.\tThe ClipRegion and DetPro have the ability to handle open-vocabulary object detection, this work degrades them to close-set object detection. Why not directly study the domain adaptive open vocabulary object detection that handles the domain shift and knowledge shift (recognizing new concepts) such as [A1]?\n[A1] Rethinking Open-World Object Detection in Autonomous Driving Scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Comment: \nWe sincerely thank you for your comprehensive comments and constructive advice. We are pleased to see our work being regarded as reasonable and addressing a crucial problem. We will explain your concerns point by point.\n\n**Q1: One major concern of this paper is the differences between the proposed prompts and those in COOP and DetPro. They utilize similar learnable prompts ... domain-specific knowledge.**\n\nA1: Thanks for your valuable concern. In the context of DAOD tasks, the shared knowledge between source and target domains is denoted as domain-invariant knowledge, and knowledge unique to a specific domain is denoted as domain-specific knowledge. \n\nThe motivation of CoOp and DetPro is learning highly generalizable and discriminative prompt (shown in Eq.2) on training domain (single domain). Ignoring the cross-domain difference, their prompts can only capture domain-specific knowledge on the training set. Due to domain bias, they have limited performance on target domain.\n\nTo enable prompt to learn cross-domain information, we introduce a novel domain-adaptive prompt, consisting of a domain-invariant token shared across domains and domain-specific token unique in each domain. With the domain-invariant and domain-specific tokens to capture domain-shared and domain-specific knowledge respectively, our DA-Pro gains better performance on the unlabelled target domain. Evaluation on C2F task shows that prompts of CoOp\\DetPro achieve an mAP of 53.0, while our domain-adaptive prompt achieved 55.9, demonstrating better cross-domain performance.\n\n**Q2: The experimental results are not adequate. K2C and S2C are only evaluated on the ‘car’ category and do not adequately show the effectiveness of the proposed method.**\n\nA2: Thanks for raising this concern. K2C and S2C are two of the mainstream benchmarks in the field of DAOD. Previous methods are evaluated on them with single class 'car'. We only test on 'car' to fairly compare with other methods. \n\nTo further evaluate the effectiveness of DA-Pro, we added three extra benchmarks: Pascal to Clipart, watercolor and comic. They enable the method to be evaluated under more challenging domain shifts and in multi-class problem scenarios. Our DA-Pro surpasses the SOTA method (SIGMA++ with ResNet-101) with a weak backbone (ResNet-50) on all three additional benchmarks, showing effectiveness of DA-Pro.\n||Pascal to Watercolor|Pascal to Clipart|Pascal to Comic|\n|---|---|---|---|\n|DBGL*|53.8|41.6|29.7|\n|Baseline|54.8|43.4|40.6|\n|FGRR*|55.7|43.3|32.7|\n|SIGMA++*|57.1|46.7|37.1|\n|DA-Pro(ResNet-50)|**58.1**|**46.9**|**44.6**|\n\\* denote backbone is ResNet-101\n\n**Q3: Why the bounding box head is shared for both domains? The ... head.**\n\nA3: Thanks for the constructive suggestion. In DAOD, domain bias primarily manifests as semantic variations. Box regression is semantic-agnostic. Hence a bbox head trained on the source domain is robust to unknown target domains. However, classification is semantic-relevant and is more affected by domain bias. Consequently, when a detector trained on the source domain is applied to the target domain, errors primarily stem from misclassifications. To explore this further, we randomly selected 100 images from the Cityscapes dataset and calculated the recall of GT bbox. The results indicated a successful localization rate of 90.7% (IoU > 0.5), whereas the correct classification rate was only 70.8%. Most of the GT can be localized by bbox head, while many are misclassified. As a result, we opt to share the bbox head and tune the classifier in DA-Pro. Our future research will explore more efficient strategies for tuning the bbox head, such as class-aware and domain-aware bbox heads.\n\n**Q4: The ClipRegion and DetPro have the ability to handle open-vocabulary object detection, this work degrades them to close-set object detection. Why ... such as [A1]?**\n\nA4: Thanks for the valuable suggestion. The DAOD task setting is closed-set object detection. To maintain consistency with other methods, we design DA-Pro to work within the closed-set detection. However, DA-Pro can also be modified as an open-vocabulary detection approach, with minimal additional overhead. For the source and target domain data, DA-Pro leverages them to tune the learnable prompt, enhancing the detection capability for classes present in both domains. For newly introduced classes, hand-crafted prompts can still be employed for detection, as the dictionary mentioned in [A1]. This work focuses on DAOD, and our future work could explore open-vocabulary domain adaptation, delving into the adaptation of new classes.\n\n**Q5: In line 260, the baseline is ..., or both? It is recommended to clarify the setting of the baseline model.**\n\nA5: Thanks for this point. The baseline adapts RegionClip, where the prompt is the hand-crafted \"A photo of [class][domain],\" as shown in the first row of Table 2. We will correct the typo on line 260 and update the baseline settings in the manuscript.\n\n**Q6: What is the initialization for the detection model? Which datasets are used to pre-trained the detection model? How about the backbone only being pre-trained in ImageNet as previous DAOD methods?**\n\nA6: Thanks for your question. In RegionCLIP, the visual and text encoder is initialized from CLIP and finetuned on CC3M(3M image-caption pairs without human annotations). Our work initializes the whole detection model with the weight of RegionCLIP. \n\nAfter initialization, we pre-train the detection model on the annotated source and the unlabelled target domain with a domain classifier for each benchmark. \n\nImageNet-pretrained weights lack alignment with textual information. Previous methods address DAOD using visual alignment without considering interactions with text. Therefore they could initialize with ImageNet-pretrained weights. However, image embeddings must align with text embeddings in the CLIP-based framework. To this end, ImageNet pretraining weights are unsuitable for our work."}, {"review_id": "qoKDWbTOv3", "reviewer": "Reviewer_85mu", "summary": "Most existing methods incorporate a visual encoder (detection backbone) to mitigate the shift across the domain. This paper leverages domain adaptive prompts comprises of domain invariant tokens, domain-specific tokens, and domain-related textual description with class label. These domain adaptive prompts introduce detection heads across the domain having a backbone of vision language models, exploring its generalizability for adaptation tasks. Experiments include three domain adaptation scenarios for evaluation.", "questions": "Please see the weakness section for relevant questions", "limitations": "Limitations are not mentioned explicitly. Authors are encouraged to add limitations", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ This paper is well-written and easy to follow\n+ An interesting application in domain adaptation space, utilizing the vision language models and analyzing the importance of the detection head instead of the backbone (visual encoder)\n+ Figures are helpful to understand the overall proposed method", "weaknesses": "- Authors need to consider Pascal to Clipart, watercolor, and comic experiments. That allows the method to be evaluated in more challenging domain shifts and multi-class problem settings.\n- Impact of uncertainty on pseudo labels? It would be interesting to see that in this problem setting, how uncertainty is helpful for a selection of pseudo labels instead of probabilities are taken into account above certain thresholds. Also how accurate are the pseudo labels?\n- Evaluation set 1500 in Cityscapes to Foggy Cityscapes? Authors need to be consistent in the evaluation e.g. in the works [see SIGMA, TIA, etc], the evaluation set is 500 images with the highest level of density in Foggy Cityscapes.\n- It would be interesting to see error bar plots on multiple runs", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Comment:\n\nWe appreciate the reviewer for the valuable comments. We are pleased to see our idea being regarded as interesting. We will explain your concerns point by point.\n\n**Q1: Authors need to consider Pascal to Clipart, watercolor, and comic experiments. That allows the method to be evaluated in more challenging domain shifts and multi-class problem settings.**\n\nA1: Thanks for your constructive suggestion. We included these three additional benchmark sets: Pascal to Clipart, watercolor, and comic. Our proposed method surpasses the SOTA method (SIGMA++ with ResNet-101) with a weak backbone (ResNet-50) on all three additional benchmarks, showing effectiveness of DA-Pro.\n\n|  | Pascal to Watercolor | Pascal to Clipart | Pascal to Comic |\n| --- | --- | --- | --- |\n| DBGL(ResNet-101) | 53.8 | 41.6 | 29.7 |\n| Baseline | 54.8 | 43.4 | 40.6 |\n| FGRR(ResNet-101) | 55.7 | 43.3 | 32.7 |\n| SIGMA++(ResNet-101) | 57.1 | 46.7 | 37.1 |\n| DA-Pro(ResNet-50) | **58.1** | **46.9** | **44.6** |\n\n*DBGL: Chen C, Li J, Zheng Z, et al. Dual bipartite graph learning: A general approach for domain adaptive object detection[C]. In CVPR, 2021.*\n\n*FGRR: Chen C, Li J, Zhou H Y, et al. Relation matters: foreground-aware graph-based relational reasoning for domain adaptive object detection[J]. In TPAMI, 2022.*\n\n*SIGMA++: Li W, Liu X, Yuan Y. SIGMA++: Improved Semantic-complete Graph Matching for Domain Adaptive Object Detection[J]. In TPAMI, 2023.*\n\n**Q2: Impact of uncertainty on pseudo labels? It would be interesting to see that in this problem setting, how uncertainty is helpful for a selection of pseudo labels instead of probabilities are taken into account above certain thresholds. Also how accurate are the pseudo labels?**\n\nA2: Thanks for raising this point. Considering that the probabilities are generated using a fixed prompt \"A photo of [class],\" aligning the model's predictions with these probabilities will result in the learnable prompt converging to the hand-crafted prompt. Unlike probabilities, pseudo-labels do not demand learning the relative distances to each category provided by hand-crafted prompts. Instead, they require the prompt to be as close to the correct category and as far from the incorrect categories as possible, thereby learning a more discriminative prompt. We conduct experiments on the three benchmarks.  Replacing pseudo-labels with probability supervision suffers 0.6~1.6% degradation on performance.\n\n| Supervision\\Benchmark | c→f | k→c | s→c |\n| --- | --- | --- | --- |\n| probabilities  | 54.3 | 60.8 | 62.1 |\n| pseudo-label(ours) | **55.9**  | **61.4** | **62.9** |\n\nIn C2F scenario, the generated pseudo-labels achieve an accuracy of 91.6%.\n\n**Q3: Evaluation set 1500 in Cityscapes to Foggy Cityscapes? Authors need to be consistent in the evaluation e.g. in the works [see SIGMA, TIA, etc], the evaluation set is 500 images with the highest level of density in Foggy Cityscapes.**\n\nA3: Thanks for pointing this out. We evaluate on 1500 set with three level of fog density to be consistent with [DA-Faster, VDD, DSS, SCAN, AT]. For [SIGMA, TIA, MeGA], we conduct additional evaluation on 500 images with the highest level, achieving 51.2% mAP over 8 class on C2F adaptation task.\n\n| 1500 test set | mAP |\n| --- | --- |\n| DA-Faster | 32.0 |\n| VDD | 40.0 |\n| DSS | 40.9 |\n| SCAN | 42.1 |\n| AT | 50.9 |\n| DA-Pro | **55.9** |\n\n| 500 test set | mAP |\n| --- | --- |\n| MeGA | 41.8 |\n| TIA | 42.3 |\n| SIGMA | 44.2 |\n| DA-Pro | **51.2** |\n\n**Q4: It would be interesting to see error bar plots on multiple runs**\n\nA4：Thanks for the advice. We conducted 5 runs in the C2F scenario. The results indicate that the mean mAP is $55.88 \\pm 0.1$. We will include experiments on other benchmarks."}, {"review_id": "KtqpsADD7R", "reviewer": "Reviewer_P2GS", "summary": "This paper proposes a new domain adaptive object detection (DAOD) method named DA-Pro. Unlike previous methods, which ignore the domain bias in the detection head, DA-Pro applies the learnable domain-adaptive prompt to generate the dynamic detection head for each domain. To do so, the prompt is designed to be composed of domain-invariant tokens, domain-specific tokens, domain descriptions, and class labels. The proposed method is evaluated in three scenarios and shows favorable performance compared to existing methods (non vision-language-model (VLM) based) and a baseline method (VLM based).", "questions": "Please discuss the points that I raised in the weaknesses section.", "limitations": "There is no discussion on the limitation of the work. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1. The proposed method is reasonably designed.\n\nS2. The proposed method shows favorable performance compared to existing non VLM-based DAOD methods and a VLM-based baseline method.\n\nS3. The ablation study reveals the effectiveness of each proposed component.\n\nS4. The paper generally reads well and easy to follow.\n", "weaknesses": "W1. The novelty of the paper is a little weak. I do acknowledge that the paper has certain novelty in a sense that it reasonably integrates [4] into the DAOD task and the authors showed not only the proposed method achieved better performance than existing non VLM-based DAOD methods and the VLM-based baseline (table 1), but also the newly introduced components contributed to the performance gain (table 2). However, the contribution is rather straightforward extension given the previous work of [4] and the DAOD task.\n\nW2. The proposed method is evaluated on rather limited datasets. All the datasets used are related to cityscape. In addition, K->C and S->C scenarios only deal with car class. The evaluation on wider variety of datasets such as PASCAL VOC, clipart, and Watercolor2k would make the paper more convincing.\n\nW3. The details of the baseline method, which is very important because the proposed method can be fairly compared only to this baseline method due to the usage of strong VLM, is a bit unclear. I suggest to show the architecture of the baseline method as in Fig 2 and explicitly show the difference with the proposed method. I believe this would more clearly reveal which component in the proposed method is important. I wonder if the pseudo labels are used in the baseline method.\n\nMinor:  \nW4. It is nice to have the ablation on the hand-crafted prompt. The paper says “clear” and “foggy” is used for Cityscape and Foggy Cityscape, respectively. I wonder how the performance changes if alternative words are used. Which words are used for the other scenarios?\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Comment:\n\nWe appreciate the reviewer for the valuable comments. Our response to the reviewer’s questions is as follows.\n\n**Q1: The novelty of the paper is a little weak. I do acknowledge that the paper has certain novelty in a sense that it reasonably integrates [4] into the DAOD task and the authors showed not only the proposed method achieved better performance than existing non VLM-based DAOD methods and the VLM-based baseline (table 1), but also the newly introduced components contributed to the performance gain (table 2). However, the contribution is rather straightforward extension given the previous work of [4] and the DAOD task.**\n\nA1: Thanks for your valuable concern. As an object detection method, the motivation of [4] is learning highly generalizable and discriminative prompt (shown in Eq.2) on training domain (single domain). When applied to DAOD task, it can only capture knowledge on the training domain and ignore the vital cross-domain information for DAOD. Due to domain bias, they achieve limited performance on target domain.\n\nOur work overcomes this limitation in DAOD by enabling prompt to learn cross-domain information, which is considered as a interesting (Review 85mu), reasonable (Reviewer A2TR), promising direction (Reviewer x91n) to address a significant research problem (Reviewer eDFv). We introduce a novel concept of a domain-adaptive prompt, consisting of a domain-invariant token shared between both domains and domain-specific tokens unique to each domain. With the domain-invariant and domain-specific tokens to capture domain-shared and domain-specific knowledge respectively, our DA-Pro achieves better performance on the unlabelled target domain. Evaluation on C2F task shows that learnable prompt with the form of [4] achieves an mAP of 53.0, while our domain-adaptive prompt achieves 55.9, demonstrating better cross-domain performance.\n\n**Q2:  The proposed method is evaluated on rather limited datasets. All the datasets used are related to cityscape. In addition, K->C and S->C scenarios only deal with car class. The evaluation on wider variety of datasets such as PASCAL VOC, clipart, and Watercolor2k would make the paper more convincing.**\n\nA2: Thanks for your constructive suggestion. We included three additional benchmark sets: Pascal to Clipart, watercolor, and comic. This expansion enables the method to be evaluated under more challenging domain shifts and in multi-class problem scenarios. Our proposed method surpasses the SOTA method (SIGMA++ with ResNet-101) with a weak backbone (ResNet-50) on all three additional benchmarks, showing effectiveness of DA-Pro.\n\n|  | Pascal to Watercolor | Pascal to Clipart | Pascal to Comic |\n| --- | --- | --- | --- |\n| DBGL(ResNet-101) | 53.8 | 41.6 | 29.7 |\n| Baseline | 54.8 | 43.4 | 40.6 |\n| FGRR(ResNet-101) | 55.7 | 43.3 | 32.7 |\n| SIGMA++(ResNet-101) | 57.1 | 46.7 | 37.1 |\n| DA-Pro(ResNet-50) | **58.1** | **46.9** | **44.6** |\n\n*DBGL: Chen C, Li J, Zheng Z, et al. Dual bipartite graph learning: A general approach for domain adaptive object detection[C]. In CVPR, 2021.*\n\n*FGRR: Chen C, Li J, Zhou H Y, et al. Relation matters: foreground-aware graph-based relational reasoning for domain adaptive object detection[J]. In TPAMI, 2022.*\n\n*SIGMA++: Li W, Liu X, Yuan Y. SIGMA++: Improved Semantic-complete Graph Matching for Domain Adaptive Object Detection[J]. In TPAMI, 2023.*\n\n**Q3: The details of the baseline method, which is very important because the proposed method can be fairly compared only to this baseline method due to the usage of strong VLM, is a bit unclear. I suggest to show the architecture of the baseline method as in Fig 2 and explicitly show the difference with the proposed method. I believe this would more clearly reveal which component in the proposed method is important. I wonder if the pseudo labels are used in the baseline method.**\n\nA3: Thanks for your advise. The baseline adapts the detection framework of RegionClip with a domain classifier, where the prompt is generated from a hand-crafted template \"A photo of [class][domain]”. The primary distinction between the baseline and DA-Pro lies in the design of the prompt within the detection head, as well as the utilization of two sets of constraint losses for tuning the prompt. And we will include a figure in the appendix to show the architecture of the baseline. Both the baseline and the proposed method are trained on the annotated source and the unlabelled target domain with classification, regression, and adversarial loss. After that, the backbone is frozen and we tune the proposed domain-adaptive prompt with two sets of constraints to learn domain-shared and domain-specific knowledge. We extensively discuss the impact of each proposed component in the table 3,4,5 of the appendix. \n\nSince pseudo-labels are only utilized in the prompt tuning process, they are not employed in the baseline method.\n\n**Q4: It is nice to have the ablation on the hand-crafted prompt. The paper says “clear” and “foggy” is used for Cityscape and Foggy Cityscape, respectively. I wonder how the performance changes if alternative words are used. Which words are used for the other scenarios?**\n\nA4: Thanks for raising this point. For hand-crafted prompts, even subtle differences can lead to variations in performance, and precise descriptions often result in better performance. This phenomenon is also reported in CoOp. We evaluated the performance of various alternative words in the C2F scenario. In other scenarios, for instance, we use “game” and “real” in SIM10K to Cityscapes, “real” and “watercolor” in Pascal to Watercolor.\n| Source domain | Target domain | mAP |\n| --- | --- | --- |\n| clear | foggy | **55.9** |\n| cityscapes | foggycityscapes | 55.1 |\n| clear | fuzzy | 54.7 |\n| - | - | 53.4 |"}, {"review_id": "SuYQunwzqU", "reviewer": "Reviewer_A2TR", "summary": "1. Proposed a novel framework for domain-aware object detection with \na) a vision-language model-based backbone to extract highly generalized features\nb) a domain-aware detection head by prompt tuning \n2. Design the prompt includes domain-invariant tokens, specific tokens, the token for class, domain-related textual description\n3. The domain-adaptive prompt tuning maintains a prompt buffer with an ensembled strategy, and the buffer is saved and used for inference.", "questions": "see the \"Weaknesses\"", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of this work is evident, reasonable, and well-expressed.\n2. The generalized semantic knowledge makes practical help with prompt tuning. \n3. The proposed method is evaluated on several benchmarks and obtains significant improvements compared with related works.\n", "weaknesses": "1. Some details about the methodology are not clear:\n- I see that the box head is frozen when tuning the prompts, so the box head is trained with the baseline detector only using source data, right? Thus, the domain-aware head is only trained on the classification branch. Won’t the box regression accuracy be influenced when changing the domain? If I understood correctly, there seems to be no adaptation process to deal with the regression.\n- The image regions rj is obtained from RPN with RoIAlign, so is each rj in fj=f(rj) already a fixed-sized feature patch? Since f is the frozen visual encoder, where are the features R={rj} extracted from?\n\n2. How much extra inference time will be increased by introducing the textual encoder?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Comment:\n\nWe sincerely appreciate the reviewer for the constructive feedback. We are encouraged that the reviewer finds our idea is evident and reasonable. We will explain your concerns point by point.\n\n**Q1: I see that the box head is frozen when tuning the prompts, so the box head is trained with the baseline detector only using source data, right? Thus, the domain-aware head is only trained on the classification branch. Won’t the box regression accuracy be influenced when changing the domain? If I understood correctly, there seems to be no adaptation process to deal with the regression.**\n\nA1: Thanks for raising a concern about training bbox head. The bbox head is trained with only source data. We observed that domain bias often impacts classification more than localization. In DAOD, domain bias primarily manifests as semantic variations. Box regression is semantic-agnostic. Hence a bbox head trained on the source domain is robust to unknown target domains. However, classification is semantic-relevant and is more affected by domain bias. Consequently, when a detector trained on the source domain is applied to the target domain, errors primarily stem from misclassifications. To explore this further, we randomly selected 100 images from the Cityscapes dataset and calculated the recall of GT bbox. The results indicated a successful localization rate of 90.7% (IoU > 0.5), whereas the correct classification rate was only 70.8%. Most of the GT can be localized by bbox head, while many are misclassified. As a result, we opt to share the bbox head and tune the classifier in DA-Pro. Our future research will explore more efficient strategies for tuning the bbox head, such as class-aware and domain-aware bbox heads.\n\n**Q2: The image regions rj is obtained from RPN with RoIAlign, so is each rj in fj=f(rj) already a fixed-sized feature patch? Since f is the frozen visual encoder, where are the features R={rj} extracted from?**\n\nA2: Thanks for pointing this issue out. r_j is the region proposal (box) obtained from RPN, and f_j = f(r_j) is a fixed-size feature patch inferred by the visual encoder f with RoIAlign. We will make the adjustments in the manuscript to prevent any potential ambiguity.\n\n**Q3: How much extra inference time will be increased by introducing the textual encoder?**\n\nA3: Thanks for your concern. After training, the learned prompts and their corresponding text embeddings will be stored. During inference, the textual encoder will not be invoked, and there is no additional inference time introduced."}]

Author rebuttal:
**Comment:**

We thank all the reviewers for their insightful and valuable comments! Overall, we are encouraged that they find that:

1. The idea of learning domain-aware detection head is **reasonable** (Reviewer eDFv, Reviewer p2gs, Reviewer A2TR), **evident** and **interesting** (Reviewer 85mu).
2. This paper **addresses an important research problem,** the prompt tuning in vision is **a popular direction**. (Reviewer eDFv) and using VLM seems as **a promising direction** (Reviewer x91n).
3. The proposed method obtains **high performance** and **significant improvements** and is **easy to follow** (Reviewer eDFv, Reviewer x91n, Reviewer 85mu, Reviewer P2GS).

We have revised the manuscript according to the reviewers' comments. The main changes we made include:

1. We add experiments of three additional mainstream benchmarks.
2. We add experiments of prompt design, exploring effectiveness over pseudo-labels and hand-crafted domain-related textual tokens.
3. We add more details about the setting of the baseline model.
4. We revise the details in Figure 2 and add the architecture of the baseline.

Next, we address each reviewer's detailed concerns point by point. We hope we have addressed all of your concerns. Thank you!

Meta-review: At first, there were concerns on novelty, evaluation on limited datasets, limitation on closed-vocabulary object detection, among others. Authors have carefully clarified these points and provided additional experimental results in making the paper stronger. The paper finally received all accept recommendations. AC agrees with this recommendation and therefore is happy to accept the paper. Authors are required to address reviewer comments and incorporate the rebuttal and discussion material to the camera-ready version of the paper.
Decision: Accept (poster)