SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haochen Li $^{1,4}$  Rui Zhang $^{2*}$  Hantao Yao $^{3}$
Xinkai Song² Yifan Hao² Yongwei Zhao² Ling Li<sup>1,4*</sup> Yunji Chen<sup>2,4</sup>
$^{1}$ Intelligent Software Research Center, Institute of Software, CAS, Beijing, China
$^{2}$ State Key Lab of Processors, Institute of Computing Technology, CAS, Beijing, China
<sup>3</sup> State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, CAS, Beijing, China
<sup>4</sup> University of Chinese Academy of Sciences, Beijing, China
haochen2021@iscas.ac.cn, zhangrui@ict.ac.cn, haotao.yao@nlpr.ia.ac.cn, {songxinkai,
haoyifan, zhaoyongwei}@ict.ac.cn, liling@iscas.ac.cn, cyj@ict.ac.cn
Domain adaptive object detection (DAOD) aims to generalize detectors trained on an annotated source domain to an unlabelled target domain. However, existing methods focus on reducing the domain bias of the detection backbone by inferring a discriminative visual encoder, while ignoring the domain bias in the detection head. Inspired by the high generalization of vision-language models (VLMs), applying a VLM as the robust detection backbone following a domain-aware detection head is a reasonable way to learn the discriminative detector for each domain, rather than reducing the domain bias in traditional methods. To achieve the above issue, we thus propose a novel DAOD framework named Domain-Aware detection head with Prompt tuning (DA-Pro), which applies the learnable domain-adaptive prompt to generate the dynamic detection head for each domain. Formally, the domain-adaptive prompt consists of the domain-invariant tokens, domain-specific tokens, and the domain-related textual description along with the class label. Furthermore, two constraints between the source and target domains are applied to ensure that the domain-adaptive prompt can capture the domains-shared and domain-specific knowledge. A prompt ensemble strategy is also proposed to reduce the effect of prompt disturbance. Comprehensive experiments over multiple cross-domain adaptation tasks demonstrate that using the domain-adaptive prompt can produce an effectively domain-related detection head for boosting domain-adaptive object detection. Our code is available at https://github.com/Therock90421/DA-Pro.
The essence of object detection lies in training a detection backbone to extract visual features from images and a detection head to recognize and locate objects based on the visual features. Object detectors whose backbone are based on Convolutional neural networks (CNNs) and VisualTransformer (ViT) have achieved encouraging performance with annotated data [31, 30, 25, 5]. However, it may suffer intolerable performance degradation when applied to an unlabelled domain due to domain bias. In this respect, domain adaptive object detection (DAOD) is explored to generalize detectors trained on an annotated source domain to an unlabelled target domain.
Current research on DAOD focuses on inferring a discriminative visual encoder as the detection backbone. They encourage the visual encoder to generate domain-invariant features by aligning
Figure 1: (a) Existing methods focus on reducing the domain bias of the detection backbone by inferring a discriminative visual encoder across domains, ignoring the domain bias in the detection head. (b) The proposed DA-Pro consists of a VLM-based backbone and a domain-aware detection head obtained by learning domain-adaptive prompt.
source and target domains in feature space [3, 48, 37, 44, 22, 39, 14, 17]. These works strive to reduce the domain bias of the detection backbone. However, they ignore the domain bias in the detection head. As shown in Fig. 1(a), they apply the same detection head on both domains, inevitably leading to performance degradation on the target domain [13, 8, 35]. Recently, vision-language models (VLMs) show remarkably high generalization for downstream tasks on different domains, such as CLIP [29], GLIP [20] and ALIGN [16]. The ability of VLMs to generate highly generalized features makes it possible to be a robust visual encoder for DAOD. On this basis, applying a VLM as the detection backbone following a domain-aware detection head is a reasonable way to directly learn a discriminative detector for target domains, rather than reducing the domain bias in traditional methods. Furthermore, inspired by prompt tuning such as CoOp [47], using domain-related prompt can produce domain-related detection head.
In this work, we propose a novel Domain-Aware detection head with Prompt tuning (DA-Pro) which is a new DAOD framework with a VLM-based backbone and a domain-aware detection head, as shown in Fig. 1(b). To the best of our knowledge, we are the first to apply prompt tuning in DAOD. Based on the highly generalized features generated from a VLM-based backbone and the domain-aware detection head, there is no need to focus on reducing the domain bias in the detection head. In DA-Pro, the domain-aware detection head is obtained by learning the domain-adaptive prompt. To achieve high discrimination, domain-adaptive prompt exploits both domain-invariant and domain-specific knowledge, consisting of domain-invariant tokens, domain-specific tokens and domain-related textual descriptions, along with the class textual descriptions. Domain-invariant tokens, which are shared across domains, learn the domain-invariant knowledge. Domain-specific tokens and the domain-related textual description are different across domains, aiming to capture the domain-specific knowledge of the corresponding domain by both learning and hand-craft.
To optimize the domain-adaptive prompt so that it can capture the domains-shared and domain-specific knowledge, we further propose two unique constraints. Firstly, to learn the shared domain-invariant knowledge, we constrain the detection heads generated by the source-domain prompt and target-domain prompt to recognize the images as accurately as possible. Secondly, given images of one domain, we constrain the detection head generated by the corresponding domain prompt to output higher confidence of its own domain than of the other domain. Moreover, we introduce a historical prompts ensemble strategy to reduce the disturbance caused by data mutation in the mini-batch. Concretely, we maintain a prompt buffer, which is updated by EMA in each iteration of training, and use it for inference.
We conduct extensive experiments for the proposed DA-Pro on three mainstream benchmarks: Cross-Weather (Cityscapes  $\rightarrow$  Foggy Cityscapes), Cross-Fov (KITTI  $\rightarrow$  Cityscapes), and Sim-to-Real (SIM10K  $\rightarrow$  Cityscapes). The experimental results show that our method brings noticeable improvement and achieves state-of-the-art performance. Concretely, DA-Pro improves the mAP by  $1.9\% \sim 3.3\%$  on synthetic and real datasets over the strong Baseline RegionCLIP. In the best case, we achieve  $55.9\%$  mAP on the widely accepted benchmark of Cross-Weather, showing remarkable effectiveness in applying the domain adaptive detection head.
Domain Adaptive Object Detection (DAOD) Domain Adaptive Object Detection (DAOD) aims to generalize the object detector trained on the labelled source domain to the unlabelled target domain. The key concept is inferring a discriminative visual encoder as the detection backbone by aligning source and target domains in feature space. Previous works for domain adaption [26, 42, 27, 40, 33, 9, 28, 19, 42] have explored minimizing various distance metrics to reduce feature discrepancy. Inspired by them, DA-Faster [3] first introduces a domain discriminator to extract domain-invariant features [31] adversarially at the image level. To avoid sub-optimization by directly aligning domain-specific features, DSS [37] suppresses the relevant gradient during backpropagation. TIA [44] divides the detection task into two sub-tasks, localization and classification, and employs task-specific discriminators to optimize them separately. CaCo [14] proposes that categories-agnostic alignment ignores class-specific knowledge and may lead to negative adaptation. Then it explicitly models the intra-class compactness and inter-class separability and aligns features on category hierarchy. SIGMA [22] further models class-conditional distributions of two domains with cross-image graphs and minimizes graph distances to bridge the domain gap.
Despite the considerable performance of domain alignment, existing methods ignore the domain bias in the detection head. They share the detection head on both domains, inevitably leading to worse discrimination on the target domain. In this work, we introduce visual-language models to DAOD, to obtain the domain-aware detection head via the domain-related textual description and the text encoder.
Visual-language models Recent advances in visual-language representation learning have shown that learning directly from image-text pairs is a promising alternative that leverages a much broader source of supervision. The model inferred by aligning the representation of the image-text pairs is defined as Visual-Language Model (VLM). A representative work is CLIP, which trains a visual encoder and a text encoder using the contrastive loss based on 400 million image-text pairs, demonstrating good generability for the unseen classes. When inference, it classifies images using textual representations generated from natural language descriptions, i.e. prompt. Some research [11, 45] adapt VLM into the object detection framework to encode robust visual and textual features. ViLD [11] distills the knowledge from a pre-trained VLM into a two-stage detector via aligning the region embeddings of proposals to CLIP's output. RegionCLIP [45] further extends VLM to directly learn region-level visual representations by generating region-text pairs as supervision. Some others [8, 35] introduce semantic domain concepts via textual prompts to infer robust detectors. In this work, we apply a VLM as the robust detection backbone and utilize the text encoder to build the domain-aware detection head. We adapt [45] with a domain classifier [3] as the Baseline.
Prompt Tuning Prompt tuning has been explored to transfer the pre-trained VLM to its downstream tasks, which utilizes a learnable textual prompt to embed task-relevant information for prediction. A proper prompt could boost the performance significantly with prompt engineering, however, it requires domain expertise and takes an amount of time for words tuning. Inspired by prompt tuning in language tasks, Context Optimization (CoOp) [47] and Conditional Context Optimization (CoCoOp) [46] replaces the hand-crafted prompts with the learnable continuous tokens to automate prompt engineering in an end-to-end manner. Furthermore, KgCoOp [41] explores the forgetting of general textual knowledge during prompt tuning and alleviates it by reducing the discrepancy between the learnable and hand-crafted prompt. DetPro [6] extends CoOP to object detection with a context grading scheme to separate proposals in the image foreground for tailored prompt training. However, these methods only develop highly generalizable and discriminative prompt on training domain (single domain). Ignoring the cross-domain difference, their domain-agnostic prompts can only capture domain knowledge on the training set. Due to domain bias, they have limited performance on target domain. To enable the prompt to learn cross-domain information, we introduce a novel domain-adaptive prompt and infer robust detection heads on each domain.
DAOD aims to generalize the detector trained on the annotated source domain to the unlabelled target domain. In this work, we propose a novel Domain-Aware detection head with Prompt tuning (DA-Pro) for DAOD, which employs prompt tuning to generate the domain-aware detection head for
Figure 2: Overview of the proposed DA-Pro framework on the Cross-Weather adaptation scenario.
capturing the domain-specific and domain-invariant knowledge, as shown in Figure 2. Therefore, we first give a brief introduction of prompt tuning and then introduce the proposed method.
Inspired by the effectiveness of CLIP for visual-language learning, we apply CLIP to embed the image and text description. Formally, CLIP consists of the visual encoder  $f$  and the text encoder  $g$ . For a classification task with  $K$  categories, CLIP uses the classes' name to generate the textual embedding space with a hand-crafted prompt, a text encoder  $g(\cdot)$ , and a text tokenizer  $e(\cdot)$ , where  $e(\cdot)$  map the text description into vectors. By denoting the class name of the  $i$ -th category as "class-i", its corresponding fixed prompt  $\mathbf{t}_i$  is established by using the template "A photo of a [class-i]" to generate the word vector tokens, e.g.,  $\mathbf{t}_i = [e("A photo of a [class-i]")]$ . After that, the text encoder further project the fixed prompt into textual embedding space, defined as  $\mathbf{z}_i = g(\mathbf{t}_i)$ .
Given an input image  $I$  and its class label  $y$ , the visual encoder  $f$  firstly extracts the visual embedding  $f(I)$ . Then, the predicted probability  $p(\hat{y} = y|I)$  of input image  $I$  on the class  $y$  is computed with the distance between visual embedding and text embedding:
$$
p (\hat {y} = y \mid I) = \frac {\exp (s (f (I) , g \left(\mathbf {t} _ {y}\right)) / \tau)}{\sum_ {i = 1} ^ {K} \exp (s (f (I) , g \left(\mathbf {t} _ {i}\right)) / \tau)}, \tag {1}
$$
where  $s(\cdot)$  is the cosine similarity, and  $\tau$  is a learnable temperature parameter.
Especially, CLIP applies the hand-crafted prompts to generate the class textual embedding, having a less discriminative ability for downstream tasks. Recently researchers, such as CoOp, show that using the continuous learnable prompt can effectively capture domain-related knowledge. Formally, the hand-crafted prompt is replaced by  $M$  learnable tokens  $\mathbb{V} = [\mathbf{v}_1][\mathbf{v}_2]\dots[\mathbf{v}_M]$ . After that, the learnable prompt  $\mathbf{t}_i$  is defined as the concatenation of  $\mathbf{v}$  and the tokenized vector  $\mathbf{c}_i = e("class-i")$  of  $i$ -th class:
$$
\mathbf {t} _ {i} = \left[ \mathrm {v} _ {1} \right] \left[ \mathrm {v} _ {2} \right] \dots \left[ \mathrm {v} _ {M} \right] \left[ \mathbf {c} _ {i} \right]. \tag {2}
$$
Recently, a lot of CLIP-based detectors are proposed to mine textual knowledge of categories effectively. For example, RegionCLIP [45] learns region-level visual representations by adapting CLIP with the hand-crafted prompt to a vanilla detector. Furthermore, DetPro [6] further applies the learnable prompt(Eq. 2) to generate the domain-agnostic representations. However, both the proposed prompts in RegionCLIP [45] and DetPro [6] cannot model the domain-specific knowledge, which is critical for DAOD.
3.2 Domain-Adaptive Prompt
Since the existing DAOD methods only consider the visual information, they aim to infer an unbiased visual encoder to generate domain-invariant features and use a unique detection head for recognition. For example, DA-Faster [3] introduces a domain discriminator to train a discriminative detection backbone [31] adversarially. However, they all ignore the domain bias in the detection head. From a cognitive perspective, using a shared visual encoder and a domain-related detection head can also
reduce the domain bias. Furthermore, thanks to the increasing scale of models and data, the pre-trained visual-language model (VLM) has a high generalization for the downstream task. Consequently, treating the pre-trained model as a visual encoder to extract the highly generalized features and using the domain-specific classifier (detection head) for recognition is a reasonable way for domain-adaptive object detection rather than reducing the domain bias in traditional methods. Therefore, how to propose a domain-adaptive detection head is the rest problem.
Inspired by prompt tuning such as CoOp, we can generate the dynamic textual class embedding by feeding different prompts, i.e., using the source (target)-related prompt can produce the source (target)-related detection heads for the source (target) domain. Based on the CLIP, the domain-adaptive detection head can be obtained by feeding a domain-adaptive prompt into the text encoder of CLIP. To ensure the domain-adaptive prompt has a high discrimination and generalization ability, an ideal domain-adaptive prompt should satisfy the following conditions: 1) it can model the shared knowledge between the source and target domains; 2) it can model the specific knowledge of each domain; 3) it contains information that distinguishes or describes each domain; 4) The class information should be embedded for discriminative representation learning.
To achieve the above issue, we design an effective domain-adaptive prompt,
$$
\mathbf {t} _ {i} ^ {d} = \left[ \mathbf {v} _ {1} ^ {c} \right] \left[ \mathbf {v} _ {2} ^ {c} \right] \dots \left[ \mathbf {v} _ {M} ^ {c} \right] \left[ \mathbf {v} _ {1} ^ {d} \right] \left[ \mathbf {v} _ {2} ^ {d} \right] \dots \left[ \mathbf {v} _ {N} ^ {d} \right] \left[ \mathbf {c} _ {i} \right] \left[ \mathbf {d l} _ {d} \right] \tag {3}
$$
In Eq.3, the domain-adaptive prompt consists of four components. Firstly,  $\mathbb{V}^c = [\mathbf{v}_1^c][\mathbf{v}_2^c]...[\mathbf{v}_M^c]$  are the domain-invariant tokens with  $M$  learnable vectors tokens to learn the common knowledge across two domains. Then,  $\mathbb{V} = [\mathbf{v}_1^d][\mathbf{v}_2^d]...[\mathbf{v}_N^d]$  are the domain-specific tokens with  $N$  learnable tokens, which are independent across domains to exploit specific knowledge of each domain, i.e.,  $\mathbb{V}^s = [\mathbf{v}_1^s][\mathbf{v}_2^s]...[\mathbf{v}_N^s]$  and  $\mathbb{V}^t = [\mathbf{v}_1^t][\mathbf{v}_2^t]...[\mathbf{v}_N^t]$  represent the domain-specific prompt of the source and target domains. Similar to CLIP,  $\mathbf{c}_i$  is the tokenized vector of the  $i$ -th class, which is the critical component for boosting the representation discrimination. To constrain the domain-adaptive prompt can fully capture the domain-related knowledge, the domain labels such as the domain-related textual descriptions are used to generate the domain-related vector tokens. Given the domain-related textual description ['domain-d'], the domain-related prompts  $\mathbf{dl}_d = e("domain-d")$  is the vector of the hand-crafted textual description "domain-d" for domain  $d \in \{s,t\}$ . As the Cityscapes to Foggy Cityscape for example, the domain labels ['domain-d'] for the source domain('Cityscapes') and the target domain ('Foggy Cityscape') are 'clear' and 'foggy', respectively. Note that the domain-related prompts  $[\mathbf{dl}_d]$  are the significant prompt to control the proposed prompt for inferring the domain-aware knowledge.
By replace the domain-specific prompt  $\mathbb{V}$  in Eq. 3 with the source-specific prompt  $\mathbb{V}^s$  and the target-specific prompt  $\mathbb{V}^t$ , the final source and target prompts are:
$$
\mathbf {t} _ {i} ^ {s} = \left[ \mathbf {v} ^ {\mathbf {c}} _ {1} \right] \left[ \mathbf {v} ^ {\mathbf {c}} _ {2} \right] \dots \left[ \mathbf {v} ^ {\mathbf {c}} _ {M} \right] \left[ \mathbf {v} ^ {\mathbf {s}} _ {1} \right] \left[ \mathbf {v} ^ {\mathbf {s}} _ {2} \right] \dots \left[ \mathbf {v} ^ {\mathbf {s}} _ {N} \right] \left[ \mathbf {c} _ {i} \right] \left[ \mathbf {d l} _ {s} \right] \tag {4}
$$
$$
\mathbf {t} _ {i} ^ {t} = \left[ \mathbf {v} ^ {\mathbf {c}} _ {1} \right] \left[ \mathbf {v} ^ {\mathbf {c}} _ {2} \right] \dots \left[ \mathbf {v} ^ {\mathbf {c}} _ {M} \right] \left[ \mathbf {v} ^ {t} _ {1} \right] \left[ \mathbf {v} ^ {t} _ {2} \right] \dots \left[ \mathbf {v} ^ {t} _ {N} \right] \left[ \mathbf {c} _ {i} \right] \left[ \mathbf {d l} _ {t} \right] \tag {5}
$$
3.3 Domain-Adaptive Prompt for object detection
Following [45], we use a class-agnostic RPN [43] to propose image regions and extract visual embedding via the visual encoder. Then, the class textual embeddings are generated by feeding the domain-adaptive prompts into the text encoder. As the generated class textual embeddings can be treated as a classifier for computing the contrastive loss with the input visual feature, which can be further used to optimize the domain-adaptive prompt with the given labels.
Given an input image  $I$ , the RPN  $r(\cdot)$  proposes  $N_{r}$  image region boxes  $\mathbb{R} = \{\mathbf{r}_j\}_{j=1}^{N_r}$ . After that, the corresponding visual embedding  $\mathbb{F} = \{\mathbf{f}_j\}_{j=1}^{N_r}$  is inferred by the visual encoder  $f$ , where  $\mathbf{f}_j = f(\mathbf{r}_j)$  is a fixed-size feature patch. Then the domain-adaptive detection head predicts probability  $p(\hat{y} = y|\mathbf{r}_j,d,D)$  of the proposal  $\mathbf{r}_j$  belongs to the class  $y$  and domain  $d$ :
$$
p (\hat {y} = y \mid \mathbf {r} _ {j}, d, \mathcal {D}) = \frac {\exp \left(s \left(f \left(\mathbf {r} _ {j}\right) , g \left(\mathbf {t} _ {y} ^ {d}\right)\right) / \tau\right)}{\sum_ {k \in D} \sum_ {i = 1} ^ {K} \exp \left(s \left(f \left(\mathbf {r} _ {j}\right) , g \left(\mathbf {t} _ {i} ^ {k}\right)\right) / \tau\right)}, \tag {6}
$$
where  $s(\cdot)$  is the cosine similarity.  $\mathcal{D} \in \{\{s\}, \{t\}, \{s, t\}\}$  indicates probability is calculated with the class embedding generated by the source prompt  $\{\mathbf{t}_i^s\}_{i=1}^K$ , the target prompt  $\{\mathbf{t}_i^t\}_{i=1}^K$ , or both prompts  $\{\mathbf{t}_i^s\}_{i=1}^K \cup \{\mathbf{t}_i^t\}_{i=1}^K$ .
The goal is to make the domain-invariant tokens learn the shared knowledge across domains and the domain-specific tokens learn the specific knowledge of each domain. The following two constraints are proposed to achieve the above goal. Firstly, the domain-invariant prompt is learnable by constraining that the detection head generated by the source-domain prompt and target-domain prompt should both classify the input images as accurately as possible. Secondly, the detection head generated by one domain should output higher confidence on its own domain than the other domain when predicting images of its own domain, so as to learn domain-specific knowledge.
Formally, given an image  $I \in \mathcal{X}^d$  from domain  $d$  along with the generated regions  $\{\mathbf{r}_j^d\}_{j=1}^{N_r}$  and its class label  $\{y_j^d\}_{j=1}^{N_r}$ . Following the prediction probability calculated by Eq. 6, we denote the cross-entropy loss as:
$$
\mathcal {L} _ {d, \mathcal {D}} = \mathbb {E} _ {\mathcal {X} ^ {s}} \left[ - \frac {1}{N _ {r}} \sum_ {j = 1} ^ {N _ {r}} \log p \left(\hat {y} = y _ {j} ^ {s} \mid \mathbf {r} _ {j} ^ {s}, d, \mathcal {D}\right) \right] \tag {7}
$$
where  $\mathcal{D} \in \{\{s\}, \{t\}, \{s, t\}\}$  indicates probability is calculated with the class embedding generated by the source prompt  $\{\mathbf{t}_i^s\}_{i=1}^K$ , the target prompt  $\{\mathbf{t}_i^t\}_{i=1}^K$ , or both prompts  $\{\mathbf{t}_i^s\}_{i=1}^K \cup \{\mathbf{t}_i^t\}_{i=1}^K$ . To learn domain-invariant knowledge, both classifiers should predict  $\mathbf{r}_j^d$  to the ground truth  $y_j^d$ :
$$
\mathcal {L} _ {d} ^ {i n v} = \mathcal {L} _ {d, \{s \}} + \mathcal {L} _ {d, \{t \}} \tag {8}
$$
To learnable domain-specific knowledge, we constrain that the classifier of domain  $d$  should be more confident on  $\mathbf{r}_j^s$  by minimizing the cross-entropy over the source and target prompts:
$$
\mathcal {L} _ {d} ^ {s p c} = \mathcal {L} _ {d, \{s, t \}} \tag {9}
$$
Therefore, the objective function of the source data  $\mathcal{X}^s$  is:
$$
\mathcal {L} _ {s} = \mathcal {L} _ {s} ^ {\text {i n v}} + \mathcal {L} _ {s} ^ {\text {s p c}}. \tag {10}
$$
Due to the lack of annotations, the above optimization objective cannot be directly performed on the target domain. We thus generate the pseudo labels for unlabeled target images with considering the powerful zero-shot capability of CLIP. Formally, we apply the hand-crafted template "A photo of [class]" to produce the class having the highest probability as pseudo labels, In particular, the pseudo labels  $y_{j}^{t}$  for the image regions  $\mathbf{r}_{j}^{t}$  is generated with Eq. 11:
$$
y _ {j} ^ {t} = \underset {y} {\arg \max } p (\hat {y} = y | \mathbf {r} _ {j} ^ {t}) \tag {11}
$$
where the probability  $p$  is computed via Eq.1.