Title: Cloud Object Detector Adaptation by Integrating Different Source Knowledge
Venue: NeurIPS 2024
Accepted: yes
Keywords: Object detection

Cloud model

Domain adaptation
Abstract: We propose to explore an interesting and promising problem, Cloud Object Detector Adaptation (CODA), where the target domain leverages detections provided by a large cloud model to build a target detector. Despite with powerful generalization capability, the cloud model still cannot achieve error-free detection in a specific target domain. In this work, we present a novel Cloud Object detector adaptation method by Integrating different source kNowledge (COIN). The key idea is to incorporate a public vision-language model (CLIP) to distill positive knowledge while refining negative knowledge for adaptation by self-promotion gradient direction alignment. To that end, knowledge dissemination, separation, and distillation are carried out successively. Knowledge dissemination combines knowledge from cloud detector and CLIP model to initialize a target detector and a CLIP detector in target domain. By matching CLIP detector with the cloud detector, knowledge separation categorizes detections into three parts: consistent, inconsistent and private detections such that divide-and-conquer strategy can be used for knowledge distillation. Consistent and private detections are directly used to train target detector; while inconsistent detections are fused based on a consistent knowledge generation network, which is trained by aligning the gradient direction of inconsistent detections to that of consistent detections, because it provides a direction toward an optimal target detector. Experiment results demonstrate that the proposed COIN method achieves the state-of-the-art performance.

Cloud Object Detector Adaptation by Integrating Different Source Knowledge

Abstract

We propose to explore an interesting and promising problem, Cloud Object Detector Adaptation (CODA), where the target domain leverages detections provided by a large cloud model to build a target detector. Despite with powerful generalization capability, the cloud model still cannot achieve error-free detection in a specific target domain. In this work, we present a novel Cloud Object detector adaptation method by Integrating different source kNowledge ( COIN ). The key idea is to incorporate a public vision-language model (CLIP) to distill positive knowledge while refining negative knowledge for adaptation by self-promotion gradient direction alignment. To that end, knowledge dissemination, separation, and distillation are carried out successively. Knowledge dissemination combines knowledge from cloud detector and CLIP model to initialize a target detector and a CLIP detector in target domain. By matching CLIP detector with the cloud detector, knowledge separation categorizes detections into three parts: consistent, inconsistent and private detections such that divide-and-conquer strategy can be used for knowledge distillation. Consistent and private detections are directly used to train target detector; while inconsistent detections are fused based on a consistent knowledge generation network, which is trained by aligning the gradient direction of inconsistent detections to that of consistent detections, because it provides a direction toward an optimal target detector. Experiment results demonstrate that the proposed COIN method achieves the state-of-the-art performance.

1 Introduction

The emergence of large language models like GPT-4 [45] heralds a future in which cloud model possesses remarkable capabilities tailored to specific tasks. But the performance always will be degraded for special target domain. Naturally, the problem of Cloud Domain Adaptation (CDA) emerges, where the knowledge from cloud model is transferred to the target domain by cloud API requests. This work focuses on Cloud Object Detector Adaptation (CODA), a new problem setting in domain adaptation. This problem is training a detector for any target domain under the conditions that there exists a large cloud detector offering API service while target domain samples have not any labels. As shown in Fig.1(a), compared with previous settings, there are two main advantages: (1) open target scenarios and object categories; (2) without high domain similarity between domains.

There have been some advances in related fields. Source-free Object Detection (SFOD) [35, 33] transfers a pre-trained model from the source domain to the target domain without considering the

Ability

Flexible architecture

Open categories

Open scenarios

person

Current

target model

Optimal target model

Gradient direction

alignment

Gradients of

inconsistent detections

Cloud

CLIP

Gradients of

consistent detections

(c) Knowledge Separation

(d) Knowledge Distillation

CLIP detector

CLIP

car

car

Box

detections

(b) Knowledge Dissemination

Supervision

Cloud

detector

Conditions

UD

AOD

SF

OD

Black-box

DAOD

CO

DA

Source data access

Source model access

High domain similarity

(a) CODA vs other settings

×

×

×

×

×

×

×

model privacy issue. The existing methods usually resort to three routes: pseudo label refinement [35, 9, 62, 71, 38], where more reliable labels are refined for self-training; knowledge distillation [33, 39, 13, 18, 28, 51], where the Mean Teacher framework is used to distill knowledge; domain alignment [33, 53, 10, 61], where distribution alignment methods are used to learn domain-invariant feature. Despite notable success has been achieved, these methods are not suited for CODA because cloud model is not accessible, and any target domain cannot be transferred because of category limitations.

Another related field is Black-box Domain Adaptive Object Detection (Black-box DAOD), which only offers detection results of several predefined categories without accessing source model and data [67]. There exists one work for Black-box DAOD, where three types of memory are used for label calibration [67]. More Black-box DA methods [36, 64, 60, 46, 68, 58] focus on classification and fall into two categories. The first uses knowledge distillation [36, 46, 58] to distill source domain knowledge, while the second uses sample selection [64, 60, 68, 57] to select representative samples for training. They also can not be applied to CODA. Unlike Black-box DAOD, CODA benefits from large training data and language modality, thus eliminating the trouble of finding a tailored source domain containing all target categories and facilitating unrestricted target domain adaptation.

Despite being trained on large data, the powerful cloud model also cannot achieve error-free detection. So leveraging public auxiliary models with enough object categories to correct the error detections is a natural choice. The public vision-language model such as CLIP [47], pre-trained on millions of image-text pairs, is a good model to help adaptation. Due to the lack of detection ability and domain shift, CLIP is first extended as a detector, abbreviated as CLIP detector, to inherit and further adapt the knowledge from CLIP, as shown in Fig.1(b). However, the CLIP detector also has error detections. As shown in Fig.1(c), the detections of CLIP detector and cloud detector can be classified into three categories: consistent, inconsistent and private detections. Now, the problem is how to distill these detections to the target detector. The consistent and private detections can be easily distilled to the target detector as the supervision signals. While for inconsistent detections, one potential route is to use consistent detections to help inconsistent detections, as shown in Fig.1(d).

Based on the above analysis, we propose a novel adaptation method in a divide-and-conquer manner, dubbed as COIN , where knowledge dissemination, separation, and distillation stages are carried out successively. Knowledge dissemination combines the knowledge from CLIP model and cloud model to initialize a CLIP detector and a target detector based on Faster R-CNN [48] architecture. At the same time, prompt learning is performed to align the target domain for CLIP detector. Knowledge separation matches the detections from cloud detector and CLIP detector, categorizing three parts as consistent, inconsistent and private detections. Based on Mean-Teacher framework, knowledge distillation regards CLIP detector and cloud detector as teachers while the target detector as a student. Consistent and private detections are used as supervision; prompt learning is performed again to adapt CLIP model for target detector. A Consistent Knowledge Generation network (CKG) is proposed to fuse inconsistent detections. Since the gradient direction for optimizing target detector based on consistent detections offers an optimization direction, for inconsistent detections, a gradient direction alignment loss is proposed to learn CKG under the situation without supervision labels. The Mean-Teacher framework also updates CLIP detector based on target detector, thereby achieving better knowledge integration.

Our contributions can be summarized as follows. (1) We propose to explore a promising problem CODA suited for real-world scenarios with large cloud models. A novel method COIN is proposed in a divide-and-conquer manner. An open auxiliary model (CLIP) is introduced to help adaptation. By carefully combining different source knowledge, the effect of one plus one being greater than two has been achieved. (2) A novel decision-level fusion strategy is proposed. The gradient direction alignment loss is proposed which fuses the conflicts by using consistence detections in a rational and self-promotion way. (3) Different prompt learning are performed for CLIP detector and target detector. For CLIP detector, the class text embeddings are aligned to CLIP visual feature class prototypes; while for target detector, the prototypes based on consistent detections are used since target detector combines different source knowledge.

2 Related Work

Domain adaptive object detection. Unsupervised Domain Adaptive Object Detection (UDAOD) assumes that source samples are freely accessible and labeled but target samples have no labels. Existing methods can be roughly classified into three categories. The first approach is based on domain alignment [8, 49, 50, 59, 6, 75, 70, 4, 31, 27], where the source and target domains are aligned by adversarial learning, contrastive learning, etc. The second approach is the popular knowledge distillation [12, 7, 3, 22, 14, 4, 20], where the Mean-Teacher framework is used to distill knowledge from teacher to student. The third approach is based on graph learning [5, 34, 41, 15], where graphs are constructed to achieve better adaptation. Source-free Object Detection (SFOD) assumes that only the pre-trained source domain model is accessible. Existing SFOD methods usually resort to three technical routes. The first is pseudo label refinement based method [35, 9, 62, 71, 38], e.g., SED [35] seeks a confident threshold for filtering pseudo labels according to self-entropy descent. The second is knowledge distillation based approach [33, 39, 13, 18, 28, 51]. For example, LODS [33] enhances target domain style and then overlooks target domain style, resulting in an impressive two-way knowledge distillation. The third is based on domain alignment [53, 10, 33, 61]. For instance, IRG [53] uses a contrastive loss to enhance the target representations by exploiting the object relations.

Black-box domain adaptation. Recently, black-box domain adaptation for image classification receives major attention. There are two routes. The first is knowledge distillation [36, 46, 58]. For example, DINE [36] starts by training a model using knowledge distillation and structural regularization, then further refines it for better adaptation; RAIN [46] introduces phase mixup and subnetwork distillation to learn from both regularized data and subnetworks; AEM [58] first proposes to explore CLIP for Black-box DA by introducing an adversarial experts model. The second is based on sample selection [64, 60, 68, 57]. For instance, IterLNL [64] follows learning with noisy labels technique and estimates a noise rate to select confident target samples for training; BETA [60] divides the target domain into two subdomains and leverages synergistic twin networks and subdomain augmentation for robust model learning; RFC [68] introduces selection training to pick samples from minority classes for reviewing forgotten classes, and employs neighborhood clustering for more balanced learning. There exists only one work for Black-box DAOD task, BiMem [67] refines the pseudo labels by constructing sensory, short-term and long-term memories, where a forward memory construction and a backward label calibration are performed iteratively. Despite the great performance achieved, the source model cannot be transferred to arbitrary target domains due to category restrictions and domain similarity, and require customizing the source domain model for a specific target domain.

3 Methodology

Problem statements. Cloud Object Detector Adaptation (CODA) assumes the unlabeled target domain D = { x i } N t i =1 and C = { c i } N c i =1 is a set of classes that need to be detected, where N t is the total number of target images, c i is the i -th class name and N c is the number of classes. There exists a powerful cloud detector F θ cld , and the goal of CODA is to train a target detector by the detection results y i cld via a cloud API. y i cld consists of boxes b i cld and class probabilities p i cld for any target domain image x i , where class probabilities p i cld are derived from class predictions, which can be in the form of class-only, confidence score, or probability, depending on the cloud detector. Moreover, with the powerful cloud detector trained on large-scale image caption datasets, open target scenarios and even categories can be adapted, eliminating the hassle of finding a similar source domain.

(a)  Knowledge Dissemination Stage

Text

Encoder

E

RPN

Backbone

BoxHead

l

b

𝒇

Feature

Per-class

Embeddings

𝒆

◉

‧‧‧

Probability

𝒑

Box

𝒃

‧‧‧

ROI

Pooling

Feature

extractor

G2

Transformation

Trans

ClassHead

l

c

A photo of a

t

1

t

2

t

3

t 4  [CLS].

Template:

Truck

‧‧‧

Person

Car

Classes

Target image

CLIP detector

Target detector

EMA

Consistent Detections

Private Detections

Box matching

Inconsistent Detections

𝓛𝓲𝓷𝓬

𝓛𝒑𝒓𝓲

CKG

Consistent Knowledge Generation

𝓛𝓬𝒌𝒈

(b)  Knowledge Separation Stage

Strong

augmentation

(c) Knowledge Distillation Stage

Similarity

Cloud

detector

A photo of a t 1

t

2

t

3

t

4

car.

‧‧‧

Prompts:

A photo of a t 1

t

2

t

3

t

4

person.

A photo of a t 1

t

2

t

3

t

4

truck.

Overview. The proposed COIN method introduces a vision-language model CLIP [47] to assist in domain adaptation of a freely chosen large-scale pre-trained cloud detector, like GDINO [40]. It consists of three stages, i.e., knowledge dissemination, knowledge separation and knowledge distillation, as illustrated in Fig.2. (a) Knowledge dissemination stage first collects results from cloud detector and CLIP respectively, where the detection boxes from cloud and the classification results from CLIP are used to initialize a CLIP detector and a target detector. Meanwhile, prompt learning is performed for CLIP detector. (b) Knowledge separation stage initially obtains and then matches the detections from cloud and CLIP detectors, resulting in the categorization of consistent, inconsistent and private detections. (c) Knowledge distillation stage updates target detector. The Mean-Teacher framework is employed which regards cloud detector and CLIP detector as two teachers while the target detector is a student. To enhance robustness, the student is fed into a strong augmentation version of target image, and consistent and private detections are directly used as pseudo labels. For inconsistent detections, a Consistent Knowledge Generation network (CKG) is designed to fuse them in decision-level; a gradient direction alignment loss is proposed to optimize CKG; the target detector and CKG are updated mutually. Better knowledge integration is achieved by updating CLIP detector.

3.1 Knowledge Dissemination

In this section, the knowledge from cloud detector and auxiliary model (CLIP) are combined to get two object detectors in target domain, i.e., CLIP detector F θ clip and target detector F θ T . An intuitive idea is to train a Faster R-CNN based detector [48] using the predicted boxes from cloud detector and its corresponding labels from CLIP. However, there are two deficiencies. The first is that the knowledge from the auxiliary model has not been fully utilized; here the auxiliary model CLIP is supposed to be open-source and known. Another is domain shift existed between CLIP and target domain; the auxiliary model CLIP should be aligned with target domain. So the pre-trained CLIP visual encoder is used to build two detectors F θ clip and F θ T ; then domain-specific prompts are learned to align CLIP model to target domain for CLIP detector; in the end, the detections are collected to train CLIP detector. They are detailed as follows.

Detector architecture is based on Faster R-CNN framework as shown in Fig.2(a). The pre-trained CLIP visual encoder G is split into G 1 and G 2 (the last block), which are used as backbone and ROI head feature extractor respectively. Because CLIP is pre-trained for classification task, the region feature of proposal r for a target image x , f r = G 2 ( ROI ( G 1 ( x ) , r )) , can not be used for box regression. So, a transformation network Trans , composed of mean pooling and three dense layers, is used to endow the localization ability. Finally, a linear layer l c and a linear layer l b are used to get the box feature and locations respectively, i.e, f = l c ( Trans ( f r ))) and b = l b ( Trans ( f r )) . The class probability p i for the i -th category is calculated by computing the similarity with the i -th class embedding e i . The background is also considered to be a class. It can be written as

where sim ( · , · ) is the cosine similarity function and τ = 0 . 01 is the fixed temperature following CLIP. The class embedding e i = E ( P i ) is obtained based on the frozen CLIP text encoder E and the prompt P i wrapping the i -th class name into a later introduced prompt template PT .

Both F θ clip and F θ T are based on this architecture and are randomly initialized except that the backbone and ROI head feature extractor are initialized by CLIP visual encoder G 1 and G 2 respectively.

Prompt learning for CLIP detector. Since the simple prompt template, like 'a photo of a [CLS].', has not target domain information, we use a trainable prompt template PT , like 'a photo of a { t i } i M =1 [CLS].', where M is fixed to 4 and t i is a placeholder whose word embedding is randomly initialized. Prompt learning methods [74, 73, 29, 69] adapt the class embeddings e = { e i } N c +1 i =1 with ground truths. To further adapt CLIP model to target domain, we propose to use visual features to align e . Specifically, the visual feature class prototypes are used to learn prompt instead of matching all visual features with the prompt embeddings. The prototypes e p = { e i p } N c +1 i =1 are updated by exponential moving average (EMA) as

where R is the set of proposals for image x , l is the label for the box feature f , and ✶ ( a = b ) an indicator function. η = 0 . 9996 is a fixed hyperparameter and e p are initialized as the original CLIP per-class embeddings e c for accelerating training. The following L 1 loss is used to learn the prompt,

Pre-training CLIP detector. Since CLIP do not predict boxes for target domain images, to train CLIP detector, we need to prepare supervision labels based on CLIP and cloud detector knowledge. For any target image x , the detection boxes b cld are borrowed from cloud detector and the corresponding labels are obtained based on the CLIP model; they can be used as supervision signals. The detection boxes are easy to obtain, while obtaining the labels needs some specific operations. Suppose a box feature f is obtained by ROI pooling on the feature map output by CLIP visual encoder G . To obtain more accurate pseudo labels, as RegionCLIP [72], 81 kinds of prompt templates are used. If the j -th kind of prompt template is 'a [target domain name] style rendering of the [CLS]', the j -th kind of class embedding for the i -th object class is e i,j c through CLIP text encoder E . The final i -th class embedding is e i c = ∑ j e i,j c / 81 . The class probability p c of f is obtained by computing the similarity like Eq.(1) using e c . The boxes predicted as 'background' are removed. After the preparation of supervision signals, the CLIP detector is pre-trained via the following losses:

where L RPN and L ROI are the standard detection losses. λ is a hyperparameter fixed as 10.

Remark. To inherit knowledge for better knowledge dissemination and accelerate training, the visual encoder of the original CLIP model is directly used as the backbone and ROI Head feature extractor. Compared to the previous CLIP based detector F-VLM [30], our backbone and ROI Head feature extractor are trainable with supervisions from CLIP and cloud detector to facilitate knowledge dissemination process. Moreover, in contrast to PromptSRC [29] and CLIP-GAP [52], which align the features to CLIP semantic space, our dynamically updated class prototypes align CLIP semantic space to target domain in an opposite way, thus capturing more target domain-specific attributes.

3.2 Knowledge Separation

Just like flipping two coins at the same time, the detections from cloud detector and CLIP detector exhibit both consistency and conflicts due to different pretraining sources. It is obvious that consistent detection results can be used as ground truths, while inconsistent results pose obstacles to knowledge fusion. To integrate their knowledge into the target detector sensibly, we adopt a divide-and-conquer strategy. Specifically, box matching is utilized to achieve knowledge separation by categorizing them into consistent, inconsistent, and private detections.

Given a target image x , suppose the detections based on cloud detector are y cld = { b cld , p cld } containing R 1 detected boxes and similarly the detections based on CLIP detector after NMS (NonMaximum Suppression) are y clip = { b clip , p clip } containing R 2 detected boxes. To find the matched

boxes, an identification matrix Γ is defined as follows, Γ i,j = 1 if the IoU ≥ κ between the i -th box from cloud detector and the j -th box from CLIP detector, otherwise Γ i,j = 0 . κ is a fixed threshold set to 0.5 according to popular settings. For the i -th box of cloud detector, the label l i cld = arg max c p i cld,c , while l j clip = arg max c p j clip,c is the label of j -th box of CLIP detector. Then, as shown in Fig.2(b) the consistent detection set ˆ P and inconsistent detection set ˜ P are defined as follows,

̸

The unmatched detection set Q , also called private detections, is defined as

Γ i, ∗ means the number of CLIP detector boxes that match the i -th cloud detector box; so does Γ ∗ ,j .

For one pair of matched boxes b i cld and b j clip from the consistent or inconsistent detections, an object is located. Typically, the box with a higher score has more precise localization. Therefore, we merge them in a probability-weighted manner to facilitate subsequent distillation, as the features extracted from two matched boxes exhibit slight inconsistencies. So the fused box b t m is

After box refinements, the consistent detection set ˆ P and inconsistent detection set ˜ P can be denoted as ˆ P = { ˆ y } and ˜ P = { ˜ y } respectively, where ˆ y = ( ˆ b m , ˆ p cld , ˆ p clip , ˆ l m ) and ˜ y = ( ˜ b m , ˜ p cld , ˜ p clip , ˜ l cld , ˜ l clip ) is the inconsistent detections.

Remark. Detection conflicts is a core challenge here. Previous UDAOD method SSAL [44] performs sample selection within the same class, so boxes in the same region that are predicted as different classes may be selected for self-training, resulting in conflicts. While we address this issue by adopting a divide-and-conquer strategy to separate conflicts here and solve them in the following text.

3.3 Knowledge Distillation

Mean-Teacher framework is utilized to distill the above three kinds of detections into target domain detector. The cloud detector and CLIP detector are two teachers while the target detector is student.

Consistent and private detections knowledge distillation. For the consistent detections ˆ P , they are directly used as ground truths to train target domain detector. The consistency distillation loss is defined as L con = L RPN + L ROI . For the private detections Q , because the prediction of private boxes is not accurate, only classification loss is calculated. By regarding the private boxes b q ∈ R |Q|× 4 as proposal boxes and feeding them into ROI Head, we obtain the classification probabilities p stu q for student. Then standard distillation loss is employed to distill all private knowledge from both teachers to the target detector as L pri = L kl ( p stu q , p q ) , where L kl ( · , · ) is the Kullback-Leibler divergence and p q are prediction results from cloud detector or CLIP detector.

By integrating different source knowledge, the class embedding should be different from CLIP detector. Similar as previous prompt learning method, we also align per-class embeddings e stu to visual class prototypes ˆ e p based on consistent detections computed as Eq.(2): L 2 align = || ˆ e p -e stu || 1 .

Inconsistent detections knowledge distillation. As shown in Fig.3, a Consistent Knowledge Generation network (CKG), noted as F θ ckg , is proposed to do decision-level fusion which refines inconsistent detections to consistent ones. Specifically, CKG takes the inconsistent box features ˜ f stu ∈ R | ˜ P|× C from target detector, inconsistent visual feature class prototypes ˜ e cld p and ˜ e clip p , inconsistent probabilities ˜ p cld and ˜ p clip as input. It outputs the consistent probabilities ˜ p ckg as follows,

where w cld = CA 1 ( ˜ f stu , ˜ e cld p ) and w clip = CA 2 ( ˜ f stu , ˜ e clip p ) are two adaptive weights generated by two cross-attention modules [19]. ⊙ represents the element-wise multiplication and δ ( · ) represents the softmax function. The architecture of cross-attention module is represented as CA ( ˜ f stu , X ) =

Student

Feature

Cloud Detector

Class Prototypes

‧‧‧

Q

K

V

CLIP Detector

Class Prototypes

‧‧‧

Q

K

A

A

𝒘𝒄𝒍𝒊𝒑

V

‧‧‧

‧‧‧

‧‧‧

Cloud Detector

Probability

CLIP Detector

Probability

Softmax

Multiply

Multiply

δ ( Q ( ˜ f stu ) ⊗ K ( X ) T ) ⊗ V ( X ) , where Q ( · ) , K ( · ) and V ( · ) are the mapping functions, and ⊗ represents the matrix multiplication. With the cross-attention module, the features ˜ f stu are compared with the class prototypes through query and key, making the weights generation process more reliable.

Since there do not exist labels in the target domain, a gradient direction alignment is proposed to train CKG network in a self-promotion way, which is also our key contribution. The idea is based on the following observation. Since the consistent detections can be regarded as ground truths in the target domain, they provide an optimization direction toward an optimal target detector. So the gradient direction from consistent detections is used as the supervised signal to train CKG network. Specifically, the gradients of consistent detections and inconsistent detections are computed using L 2 loss as follows,

where I ( · ) is the one-hot vector function, ˆ p stu are the predicted probabilities corresponding to the target detector features ˆ f stu on consistent detections; ˜ p stu are the predicted probabilities corresponding to ˜ f stu on inconsistent detections. Then, the CKG network is optimized by aligning ˜ g to ˆ g by cosine similarity. Meanwhile, CKG should also be consistent on consistence detections, i.e., ˆ p ckg = δ ( CA 1 ( ˆ f stu , ˆ e p ) ⊙ ˆ p cld + CA 2 ( ˆ f stu , ˆ e p ) ⊙ ˆ p clip ) , is consistent with the label ˆ l m . So the total loss for training CKG network is

The CKG network and target detector are updated mutually. First, CKG is optimized based on target detector, then the output ˜ p ckg is used in turn to update target detector. In order to avoid the interference of low-confidence predictions, we use a threshold π to filter out those low-confidence predictions, resulting in ˜ p π stu and ˜ p π ckg , so target detector is optimized as follows:

Remark. Traditional decision-level fusion method [55] employs simple averaging to merge knowledge from various sources, where different sources share one RPN network to generate fully matched detections. In contrast, we achieve decision-level fusion for two unrelated models based on a divideand-conquer strategy without ground truth. For inconsistent detections, our method uses a gradient direction alignment to optimize the fusion network in a self-promotion manner.

Overall optimization. In each iteration, the CKG network is first updated via Eq.(10). Then we update target detector via the following objective function:

where γ 1 and γ 2 are two hyperparameters. λ is fixed as 10 as in Eq.(4). The CLIP detector is updated by θ clip = η · θ clip +(1 -η ) · θ T , where η = 0 . 9996 as in Eq.(2), enabling the integrated knowledge in the target detector flows into the CLIP detector gradually, thus achieving better knowledge integration.

Remark. Although CLIP is utilized in COIN, CODA does not impose restrictions on the use of CLIP.

4 Experiments

Datasets. The problem CODA enables versatile target domain adaptation based on cloud detector, so there are no limitation to transfer scenarios, unlike the problem settings of UDAOD, SFOD and

Black-box DAOD. Specifically, we validate the effectiveness of the proposed COIN method on six object detection datasets, e.g., Cityscapes [11], Foggy-Cityscapes [11], Clipart [25], BDD100K [63], KITTI [16] and Sim10K [26].

Implementation details. By default, we use the Swin-B [42] version of GDINO [40] as our cloud detector, where class predictions are provided in probability format. Additionally, in Appendix, we present results using the Swin-L [42] version of GLIP [32] as an optional alternative, which offers class predictions in the form of confidence score. SAM is not chosen here due to its need for weak supervision [65]. For each dataset, both CLIP detector and target detector are based on the same version of CLIP visual encoder. Specifically, for Clipart, to be consistent with the compared methods [49, 33], ResNet101 [21] is used. While for others, ResNet50 [21] is used. The hyperparameters γ 1 , γ 2 and π are set to 0.1, 0.1 and 0.7 by default. The shorter side of the image is resized to 600 during training and testing, and the reported mean average precision (mAP) is based on an IoU threshold of 0.5. For more details about datasets, network architectures, algorithm et al, please refer to Appendix.

4.1 Comparison with State-of-the-arts

Since these do not exist any works on the CODA problem, we compare our method COIN with UDAOD, SFOD and Black-box DAOD methods, since their settings are closest to ours and the target domain is consistent. The performances of CLIP detector and cloud detector are also compared, which shows our method is better than both of them. UDAOD methods are DA-Faster [8], MTOR [3], SCL [50], ICR-CCR[59], SIGMA++ [34], PT [7], MGADA [75], CIGAR [41], TFD [54], MAF [23], ATF [24]. SFOD methods are SED [35], LODS [33], A 2 SFOD [10], IRG [53], PETS [39], LPU [9], BT [13], DRU [28] and WSCoL [61]. Black-box DAOD method is BiMem [67]. CLIP represents the original CLIP predictions with boxes from cloud detector. CLIP detector represents the detector after pre-training. Oracle represents target detector under supervision from ground truth. The results of compared methods in the tables are cited from their papers.

Sensitivityanalysisofπt

Sensitivityanalysisofy1(y2=0.1)

Sensitivityanalysisofy2(y1=0.1)

40

40

V1

40

V2

mAP(%)

mAP(%)

mAP(%)

38

38

38

36

36

36

34

34

34

0.2

0.4

0.6

0.7

0.8

0.9

0.1

0.3

valueofy1

0.5

0.7

0.9

1.0

0.1

0.3

0.5

0.7

0.9

1.0

Quantitative results for GDINO [40] are shown in Table 1-4, and results for GLIP [32] are shown in Appendix. First, the existing methods are compared across four commonly used target domain datasets: Foggy-Cityscapes, BDD100K, Clipart, and KITTI. Specifically, our method COIN significantly outperforms cloud detector by +4.6% (from 34.4% to 39.0%) on Foggy-Cityscapes and CLIP by +18.7% (from 62.1% to 80.8%) on KITTI. This demonstrates that our COIN can identify valuable knowledge for adaptation, regardless of the performance of CLIP (bad on Foggy-Cityscapes while good on KITTI). And CLIP detector improves the mAP by a large margin of +12.8% on Foggy-Cityscapes, +14.4% on BDD100K, and +17.8% on KITTI compared with CLIP, strongly demonstrating the effectiveness of the knowledge dissemination stage. Moreover, GDINO and CLIP already achieve surprising performance of 66.8% and 46.3% on Clipart, proving the superiority of CODA compared to traditional adaptation settings.

Second, since CODA enables versatile target domain adaptation with open categories and scenarios, experiments on Cityscapes for all 8 categories and Sim10K are conducted. Existing methods are not compared, as for Cityscapes they can only detect the car category while Sim10K is usually used as the source domain. From Table 4, we see that the proposed COIN achieves the best performance. Specifically, for Sim10K, when cloud detector and CLIP perform similarly, COIN still brings a significant improvement of +15.9% compared with cloud detector. The extensive quantitative results above not only demonstrate the wide applicability of CODA but also validate the effectiveness and robustness of our proposed method COIN .

4.2 Further Analysis.

Ablation study. As shown in Table 5, ablation studies are conducted on Foggy-Cityscapes and Cityscapes. Specifically, L align + CLIP detector or L align + COIN represent prompt learning for CLIP detector or target detector respectively; L con , L inc and L pri represent the distillation losses of consistent, inconsistent and private detection respectively. (1) For CLIP detector, prompt learning improves the performances from 27.4% and 35.1% to 28.2% and 35.7% on Foggy-Cityscapes and Cityscapes respectively. (2) For the proposed COIN method, all proposed components are effective which demonstrates that our method is able to achieve judicious knowledge integration.

Ablation study for decision-level fusion. To further validate the effectiveness of decision-level fusion, our proposed COIN is compared with four experimental groups, as shown in Table 6. Using

Table 6: Ablation study for decision-level fusion of inconsistent detections on Foggy-Cityscapes under GDINO. Detections are filtered by π = 0 . 7 for fair comparison. det: detector. probs: probabilities. avg: average. s-avg: score-weighted average.

the cloud detector alone achieves a mAP of 37.7%. Surprisingly, using the CLIP detector alone achieves an even higher mAP of 38.1%, attributed to the gradual parameter updates of the CLIP detector during training, allowing integrated knowledge to flow into it. Additionally, using both probabilities simultaneously with avg or s-avg approaches yield similar results. While our proposed CKG unsurprisingly achieves the best results, with a mAP improvement of +0.9% (from 38.1% to 39.0%). This strongly demonstrates the effectiveness of our proposed decision-level fusion.

Hyperparameters sensitivity analysis. We conduct sensitivity analysis on π , γ 1 and γ 2 on FoggyCityscapes, as shown in Fig.4. For parameter π , our method achieves relatively stable results over a wide range. For parameters γ 1 and γ 2 , we first set γ 2 to 0.1 and vary γ 1 across six distinct values ranging from 0.1 to 1.0. Then, we reciprocate the process for γ 2 . The outcomes are stable, with a mAP oscillating within a band between 38.0% and 39.0%. This confirms the robustness of COIN .

5 Conclusion

We proposed a novel method termed COIN for the proposed cloud object detector adaptation (CODA). The open source CLIP model is adapted to help distill knowledge in a divide-and-conquer manner. To efficiently disseminate knowledge from CLIP and cloud detector, a CLIP detector is designed and adapted to the target domain by prompt learning. Then, three kinds of detections are split and distilled to target detector respectively. Consistent and private detections are used as supervision signals without loss of generality. Prompt leaning is applied again for target detector to fit target domain. To eliminate conflicts, a consistent knowledge generation network (CKG) is proposed for decision-level fusion. A gradient direction alignment loss is proposed to learn this network in a self-promotion way. Experimental results validated the effectiveness of our method. COIN is not limited to detection task; it can also be utilized to other tasks, e.g., classification or semantic segmentation.

Acknowledgments and Disclosure of Funding

This work was supported by the National Natural Science Foundation of China (62276048, 62476169), Chengdu Science and Technology Projects (2023-YF06-00009-HZ) and Postdoctoral Fellowship Program of CPSF (GZC20233323).

References

Hanoona Bangalath, Muhammad Maaz, Muhammad Uzair Khattak, Salman H Khan, and Fahad Shahbaz Khan. Bridging the gap between object and image-level representations for open-vocabulary detection. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 33781-33794. Curran Associates, Inc., 2022.

Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Yves Lechevallier and Gilbert Saporta, editors, Proceedings of COMPSTAT'2010 , pages 177-186, Heidelberg, 2010. Physica-Verlag HD.

Qi Cai, Yingwei Pan, Chong-Wah Ngo, Xinmei Tian, Lingyu Duan, and Ting Yao. Exploring object relation in mean teacher for cross-domain detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019.

Shengcao Cao, Dhiraj Joshi, Liang-Yan Gui, and Yu-Xiong Wang. Contrastive mean teacher for domain adaptive object detectors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 23839-23848, June 2023.

Chaoqi Chen, Jiongcheng Li, Zebiao Zheng, Yue Huang, Xinghao Ding, and Yizhou Yu. Dual bipartite graph learning: A general approach for domain adaptive object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 2703-2712, October 2021.

Chaoqi Chen, Zebiao Zheng, Xinghao Ding, Yue Huang, and Qi Dou. Harmonizing transferability and discriminability for adapting object detectors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020.

Meilin Chen, Weijie Chen, Shicai Yang, Jie Song, Xinchao Wang, Lei Zhang, Yunfeng Yan, Donglian Qi, Yueting Zhuang, Di Xie, et al. Learning domain adaptive object detection with probabilistic teacher. In International Conference on Machine Learning , pages 3040-3055. PMLR, 2022.

Yuhua Chen, Wen Li, Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Domain adaptive faster r-cnn for object detection in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2018.

Zhihong Chen, Zilei Wang, and Yixin Zhang. Exploiting low-confidence pseudo-labels for source-free object detection. In Proceedings of the 31st ACM International Conference on Multimedia , MM'23, page 5370-5379, New York, NY, USA, 2023. Association for Computing Machinery.

Qiaosong Chu, Shuyan Li, Guangyi Chen, Kai Li, and Xiu Li. Adversarial alignment for source free object detection. Proceedings of the AAAI Conference on Artificial Intelligence , 37(1):452-460, Jun. 2023.

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016.

Jinhong Deng, Wen Li, Yuhua Chen, and Lixin Duan. Unbiased mean teacher for cross-domain object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4091-4101, June 2021.

Jinhong Deng, Wen Li, and Lixin Duan. Balanced teacher for source-free object detection. IEEE Transactions on Circuits and Systems for Video Technology , pages 1-1, 2024.

Jinhong Deng, Dongli Xu, Wen Li, and Lixin Duan. Harmonious teacher for cross-domain object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 23829-23838, June 2023.

Changlong Gao, Chengxu Liu, Yujie Dun, and Xueming Qian. Csda: Learning categoryscale joint feature for domain adaptive object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 11421-11430, October 2023.

A Geiger, P Lenz, C Stiller, and R Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research , 32(11):1231-1237, 2013.

Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In International Conference on Learning Representations , 2022.

Yan Hao, Florent Forest, and Olga Fink. Simplifying source-free domain adaptation for object detection: Effective self-training strategies and performance insights. arXiv preprint arXiv:2407.07586 , 2024.

Yanchao Hao, Yuanzhe Zhang, Kang Liu, Shizhu He, Zhanyi Liu, Hua Wu, and Jun Zhao. An end-to-end model for question answering over knowledge base with cross-attention combining global knowledge. In Regina Barzilay and Min-Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 221-231, Vancouver, Canada, July 2017. Association for Computational Linguistics.

Boyong He, Yuxiang Ji, Zhuoyue Tan, and Liaoni Wu. Diffusion domain teacher: Diffusion guided domain adaptive object detector. In ACM Multimedia 2024 , 2024.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016.

Mengzhe He, Yali Wang, Jiaxi Wu, Yiru Wang, Hanqing Li, Bo Li, Weihao Gan, Wei Wu, and Yu Qiao. Cross domain object detection by target-perceived dual branch distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9570-9580, June 2022.

Zhenwei He and Lei Zhang. Multi-adversarial faster-rcnn for unrestricted object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2019.

Zhenwei He and Lei Zhang. Domain adaptive object detection via asymmetric tri-way fasterrcnn. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020 , pages 309-324, Cham, 2020. Springer International Publishing.

Naoto Inoue, Ryosuke Furuta, Toshihiko Yamasaki, and Kiyoharu Aizawa. Cross-domain weakly-supervised object detection through progressive domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2018.

Matthew Johnson-Roberson, Charles Barto, Rounak Mehta, Sharath Nittur Sridhar, Karl Rosaen, and Ram Vasudevan. Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks? In 2017 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2017.

Mikhail Kennerley, Jian-Gang Wang, Bharadwaj Veeravalli, and Robby T. Tan. Cat: Exploiting inter-class dynamics for domain adaptive object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 16541-16550, June 2024.

Trinh Le Ba Khanh, Huy-Hung Nguyen, Long Hoang Pham, Duong Nguyen-Ngoc Tran, and Jae Wook Jeon. Dynamic retraining-updating mean teacher for source-free object detection. arXiv preprint arXiv:2407.16497 , 2024.

Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 15190-15200, October 2023.

Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and Anelia Angelova. Open-vocabulary object detection upon frozen vision and language models. In The Eleventh International Conference on Learning Representations , 2023.

Haochen Li, Rui Zhang, Hantao Yao, Xin Zhang, Yifan Hao, Xinkai Song, Xiaqing Li, Yongwei Zhao, Ling Li, and Yunji Chen. Da-ada: Learning domain-aware adapter for domain adaptive object detection. arXiv preprint arXiv:2410.09004 , 2024.

Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10965-10975, June 2022.

Shuaifeng Li, Mao Ye, Xiatian Zhu, Lihua Zhou, and Lin Xiong. Source-free object detection by learning to overlook domain style. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8014-8023, June 2022.

Wuyang Li, Xinyu Liu, and Yixuan Yuan. Sigma++: Improved semantic-complete graph matching for domain adaptive object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(7):9022-9040, 2023.

Xianfeng Li, Weijie Chen, Di Xie, Shicai Yang, Peng Yuan, Shiliang Pu, and Yueting Zhuang. A free lunch for unsupervised domain adaptive object detection without source data. Proceedings of the AAAI Conference on Artificial Intelligence , 35(10):8474-8481, May 2021.

Jian Liang, Dapeng Hu, Jiashi Feng, and Ran He. Dine: Domain adaptation from single and multiple black-box predictors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8003-8013, June 2022.

Chuang Lin, Peize Sun, Yi Jiang, Ping Luo, Lizhen Qu, Gholamreza Haffari, Zehuan Yuan, and Jianfei Cai. Learning object-language alignments for open-vocabulary object detection. In The Eleventh International Conference on Learning Representations , 2023.

Luojun Lin, Qipeng Liu, Xiangwei Zheng, and Zheng Lin. Slow-fast adaptation for source-free object detection. In 2024 IEEE International Conference on Multimedia and Expo (ICME) , pages 1-6, 2024.

Qipeng Liu, Luojun Lin, Zhifeng Shen, and Zhifeng Yang. Periodically exchange teacher-student for source-free object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 6414-6424, October 2023.

Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 , 2023.

Yabo Liu, Jinghua Wang, Chao Huang, Yaowei Wang, and Yong Xu. Cigar: Cross-modality graph reasoning for domain adaptive object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 23776-23786, June 2023.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021.

Zongyang Ma, Guan Luo, Jin Gao, Liang Li, Yuxin Chen, Shaoru Wang, Congxuan Zhang, and Weiming Hu. Open-vocabulary one-stage detection with hierarchical visual-language knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 14074-14083, June 2022.

Muhammad Akhtar Munir, Muhammad Haris Khan, M. Sarfraz, and Mohsen Ali. Ssal: Synergizing between self-training and adversarial learning for domain adaptive object detection. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 22770-22782. Curran Associates, Inc., 2021.

OpenAI. Gpt-4 technical report, 2023.

Qucheng Peng, Zhengming Ding, Lingjuan Lyu, Lichao Sun, and Chen Chen. Rain: regularization on input and network for black-box domain adaptation. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence , pages 4118-4126, 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 8748-8763. PMLR, 18-24 Jul 2021.

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015.

Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko. Strong-weak distribution alignment for adaptive object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019.

Zhiqiang Shen, Harsh Maheshwari, Weichen Yao, and Marios Savvides. Scl: Towards accurate domain adaptive object detection via gradient detach based stacked complementary losses. arXiv preprint arXiv:1911.02559 , 2019.

Simon Varailhon, Masih Aminbeidokhti, Marco Pedersoli, and Eric Granger. Source-free domain adaptation for yolo object detection. arXiv preprint arXiv:2409.16538 , 2024.

Vidit Vidit, Martin Engilberge, and Mathieu Salzmann. Clip the gap: A single domain generalization approach for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3219-3229, June 2023.

Vibashan VS, Poojan Oza, and Vishal M. Patel. Instance relation graph guided source-free domain adaptive object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3520-3530, June 2023.

Haoan Wang, Shilong Jia, Tieyong Zeng, Guixu Zhang, and Zhi Li. Triple feature disentanglement for one-stage adaptive object detection. Proceedings of the AAAI Conference on Artificial Intelligence , 38(6):5401-5409, Mar. 2024.

Jiaxi Wu, Jiaxin Chen, Mengzhe He, Yiru Wang, Bo Li, Bingqi Ma, Weihao Gan, Wei Wu, Yali Wang, and Di Huang. Target-relevant knowledge preservation for multi-source domain adaptive object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5301-5310, June 2022.

Size Wu, Wenwei Zhang, Sheng Jin, Wentao Liu, and Chen Change Loy. Aligning bag of regions for open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 15254-15264, June 2023.

Mingxuan Xia, Junbo Zhao, Gengyu Lyu, Zenan Huang, Tianlei Hu, Gang Chen, and Haobo Wang. A separation and alignment framework for black-box domain adaptation. Proceedings of the AAAI Conference on Artificial Intelligence , 38(14):16005-16013, Mar. 2024.

Siying Xiao, Mao Ye, Qichen He, Shuaifeng Li, Song Tang, and Xiatian Zhu. Adversarial experts model for black-box domain adaptation. In Proceedings of the 32nd ACM International Conference on Multimedia , MM'24, page 8982-8991, New York, NY, USA, 2024. Association for Computing Machinery.

Chang-Dong Xu, Xing-Ran Zhao, Xin Jin, and Xiu-Shen Wei. Exploring categorical regularization for domain adaptive object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020.

Jianfei Yang, Xiangyu Peng, Kai Wang, Zheng Zhu, Jiashi Feng, Lihua Xie, and Yang You. Divide to adapt: Mitigating confirmation bias for domain adaptation of black-box predictors. In The Eleventh International Conference on Learning Representations , 2023.

Jiuzheng Yang, Song Tang, Yangkuiyi Zhang, Shuaifeng Li, Mao Ye, Jianwei Zhang, and Xiatian Zhu. Rethinking weak-to-strong augmentation in source-free domain adaptive object detection. arXiv preprint arXiv:2410.05557 , 2024.

Ilhoon Yoon, Hyeongjun Kwon, Jin Kim, Junyoung Park, Hyunsung Jang, and Kwanghoon Sohn. Enhancing source-free domain adaptive object detection with low-confidence pseudo label distillation. arXiv preprint arXiv:2407.13524 , 2024.

Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2636-2645, 2020.

Haojian Zhang, Yabin Zhang, Kui Jia, and Lei Zhang. Unsupervised domain adaptation of black-box source models. arXiv preprint arXiv:2101.02839 , 2021.

Haojie Zhang, Yongyi Su, Xun Xu, and Kui Jia. Improving the generalization of segmentation foundation model under distribution shift via weakly supervised adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 23385-23395, June 2024.

Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision-language understanding. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 36067-36080. Curran Associates, Inc., 2022.

Jingyi Zhang, Jiaxing Huang, Xueying Jiang, and Shijian Lu. Black-box unsupervised domain adaptation with bi-directional atkinson-shiffrin memory. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 11771-11782, October 2023.

Shaojie Zhang, Chun Shen, Shuai Lü, and Zeyu Zhang. Reviewing the forgotten classes for domain adaptation of black-box predictors. Proceedings of the AAAI Conference on Artificial Intelligence , 38(15):16830-16837, Mar. 2024.

Yabin Zhang, Wenjie Zhu, Hui Tang, Zhiyuan Ma, Kaiyang Zhou, and Lei Zhang. Dual memory networks: A versatile adaptation approach for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 28718-28728, June 2024.

Liang Zhao and Limin Wang. Task-specific inconsistency alignment for domain adaptive object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 14217-14226, June 2022.

Sicheng Zhao, Huizai Yao, Chuang Lin, Yue Gao, and Guiguang Ding. Multi-source-free domain adaptive object detection. International Journal of Computer Vision , pages 1-33, 2024.

Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16793-16803, 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 16816-16825, June 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision , 130(9):2337-2348, 2022.

Wenzhang Zhou, Dawei Du, Libo Zhang, Tiejian Luo, and Yanjun Wu. Multi-granularity alignment domain adaptation for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9581-9590, June 2022.

A Appendix

A.1 More Related Works

CLIP based detector. Integrating large language-visual models (e.g., CLIP) into object detectors becomes popular. Existing methods usually use the text encoder of CLIP as the classifier, which can be broadly classified into two routes. The first route is based on knowledge distillation [17, 43, 1], which aims to distill the knowledge of the CLIP model into closed-set detectors. For example, ViLD [17] utilizes instance-level visual-to-visual knowledge distillation; HierKD [43] proposes a hierarchical distillation method with global-level language-to-visual and instance-level visual-to-visual distillation. Since CLIP is trained on images rather than object regions, the second route, region-text alignment [30, 37, 56], aims to align image region features to the fixed text region features. For instance, RegionCLIP [72] aligns image regions with region-level descriptions using a contrastive loss; VLDet [37] formulates the alignment as a set matching problem where a set of regions and a set of words are aligned. F-VLM [30] uses the CLIP vision encoder as the frozen backbone and combines the detection scores and CLIP predictions as the final output. Unlike these methods, our work combines the knowledge of CLIP to help adaptation of cloud detector. To fully explore the knowledge from CLIP, we also utilize CLIP vision encoder as backbone and ROI Head feature extractor. Moreover, prompt learning technique is embedded for adapting CLIP knowledge to target domain.

Large detection model. In order to achieve success in open-set detection, large detection models leverage massive image-text pairs for training, breaking the constraints of categories and scenes while attaining robust detection capabilities. GLIP [32], CLIPv2 [66], and GDINO [40] are representative advances. Specifically, GDINO [40] integrates detection and grounding into a unified framework. By leveraging a powerful detector pre-trained on multiple datasets, it delivers impressive performance across various downstream tasks. Without loss of generality, we choose GDINO and GLIP as the cloud detector in this work.

A.2 Methodological supplements

A.2.1 Network details

Due to the space constraints in the main text, we provide a detailed description here for the two designed networks: object detector and Consistent Knowledge Generation network (CKG).

Detector architecture is based on the two-stage Faster R-CNN [48] framework. Specifically, the ResNet50 [21] or ResNet101 [21] version of CLIP visual encoder G is split into G 1 and G 2 to be the backbone and feature extractor for ROI head following Faster R-CNN, where G 2 is the last residual block. With an input target image x , the backbone G 1 firstly produces output feature map g ∈ R H × W × C 1 , where H , W and C 1 represent the height, width and dimension of the feature map. Then, based on g , RPN generates a set of region proposals R = RPN ( g ) . For a proposal r ∈ R , ROI pooling and G 2 are utilized to extract a region feature f r = G 2 ( ROI ( g , r )) , where f r ∈ R 7 × 7 × C 2 and C 2 is the feature dimension. Since CLIP is pre-trained for classification task, f r can not be used for box regression, thus a transformation network Trans , composed of mean pooling and three linear layers (with two Leaky ReLUs), is used to endow the localization ability. To project feature into semantic space for final classification, a linear layer l c is used to obtain the box feature f = l c ( Trans ( f r ))) , where f ∈ R C and C is the dimension of semantic space. While a linear layer l b is used to get the box prediction b = l b ( Trans ( f r )) , where b ∈ R 4 . Finally, the class probability p ∈ R N c +1 of the box feature f is calculated by computing the similarity with the per-class embeddings e ∈ R ( N c +1) × C , where background is also considered to be a class. Specifically, the i -th class probability p i ∈ R is calculated by

where sim ( · , · ) is the cosine similarity function and τ = 0 . 01 is the fixed temperature.

The i -th class embedding e i is obtained as follows. Since no target domain information is wrapped in the simple prompt template, like 'a photo of a [CLS].', the embedding generated with it does not fit the target domain. So, a trainable prompt template PT , 'a photo of a { t i } i M =1 [CLS].', is designed to capture target specific attributes, where t i is a placeholder and M is fixed to 4. By wrapping the

i -th class name like 'car', a prompt P i , e.g., 'a photo of a t 1 t 2 t 3 t 4 car.' is obtained using PT . Then, the tokens T i for the i -th class are obtained by projecting P i into word embeddings, and the embedding of t i is randomly initialized. Finally, the class embedding e i = E ( T i ) is obtained based on the frozen CLIP text encoder E .

Consistent Knowledge Generation network. As shown in Fig.3, a Consistent Knowledge Generation network (CKG), noted as F θ ckg , takes the inconsistent box features ˜ f stu ∈ R | ˜ P|× C from target detector as input and output the consistent probabilities ˜ p ckg ∈ R | ˜ P|× ( N c +1) , where | ˜ P| represent the number of inconsistent boxes ˜ b m from image x . A simple description is used here since features ˜ f stu come from those proposals that matched to ˜ b m in practice.

Specifically, to facilitate the generation process, features ˜ f stu are compared with the inconsistent visual feature class prototypes ˜ e cld p ∈ R ( N c +1) × C and ˜ e clip p ∈ R ( N c +1) × C for cloud and CLIP detectors respectively, resulting in two adaptive weights w cld ∈ R | ˜ P|× ( N c +1) and w clip ∈ R | ˜ P|× ( N c +1) :

where CA 1 and CA 2 are two randomly initialized cross-attention modules [19] with the same architecture. Finally, the adaptive weights w cld and w clip are multiplied with inconsistent probabilities ˜ p cld and ˜ p clip , resulting the consistent probabilities ˜ p ckg as follows,

Algorithm 1 Our proposed COIN method.

where ⊙ represents the element-wise multiplication and δ ( · ) represents the softmax function. The architecture of the cross-attention module is represented as

where ⊗ represents the matrix multiplication, A ( ˜ f stu , X ) is the attention map and X represents class prototypes ˜ e cld p or ˜ e clip p . Q ( · ) and K ( · ) are the linear mapping functions that map the input of dimension C to dimension C ′ according to standard cross-attention, and V ( · ) is the linear mapping function that maps the class prototypes of dimension C to weights of dimension N c +1 . Thus, the generation of final weights w cld and w clip are supported by the attention map between inconsistent features and class prototypes, making it more reliable.

A.2.2 Prompt templates for CLIP model.

Naturally, target tailored prompt templates encapsulate the relevant attributes of the target domain, and the integration of multiple prompt templates can yield more precise results. Therefore, as RegionCLIP [72], we design 81 prompt templates to collect class predictions from the CLIP model, as shown in Table 7. For example, the first prompt 'Cityscapes style car.' for a class 'car' and the target domain name 'Cityscapes' is easily obtained by filling the first template '[target domain name] style [CLS].'. With these prompts, a class embedding e i,j c for the i -th class and j -th template is similarly calculated, just like calculating the i -th class embedding for CLIP detector from above. Then ensemble is used to compute the mean of these 81 embeddings, resulting in the final class embedding for the i -th class e i c = ∑ j e i,j c / 81 .

A.2.3 Optimization and algorithm

Due to the risk of disruption caused by the randomly initialized target detector on the parameters of the CLIP detector through exponential moving average (EMA), we divide the training process into two stages. In the first stage, the CLIP detector is fixed, and updates are applied to the CKG and target detector as follows,

which allows the CKG to receive effective training before distilling inconsistent detections. In the second stage, as described in the main text, updates are applied to the CKG, target detector, and CLIP detector as follows,

where γ 1 and γ 2 are two hyperparameters. λ is fixed as 10 and η is set to 0.9996. The update of CLIP detector enables the integrated knowledge in the target detector flows into the CLIP detector gradually, thus achieving better knowledge integration. The training process is summarized in Algorithm 1.

A.3 More Experiments

Detailed datasets. Cityscapes [11] consists of 2,975 training images and 500 testing images captured under normal weather with a total of 8 classes. Foggy-Cityscapes [11] contains three levels of foggy images simulated by the images of Cityscapes. 2,975 training images and 500 testing images with a foggy level of 0.02 are utilized for training and testing. Clipart [25] includes 1K clipart-style images with 20 classes. Following [49, 33], all 1K images are used for both training and testing. BDD100K [63] contains 100K videos of the scenes from New York, Berkeley, San Francisco and Bay Area. For comparison with existing methods, we follow [35, 14], and use 36,728 training images and 5,258 testing images with 7 classes for training and testing respectively. KITTI [16] contains 7,481 urban images with the car category. We use all the images for training and testing. Sim10K [26] contains 10K images collected from the computer game Grand Theft Auto V with the car category. All images are used for training and testing.

More implementation details. One 3090 GPU, a batch-size 3 and a random seed 2024 are used for all experiments. SGD [2] is used as the optimizer where the initial learning rate is 0.001 and the weight decay is 0.0001. For pre-training CLIP detector, we iterate 50K steps. For knowledge distillation, we generally iterate 45K steps using Eq.17, and then iterate 20K steps using Eq.18. The

training requires 18GB to 20GB of memory. Following [33, 7], the target detector is used for final testing. For Eq.(9) of the main text, we use L 2 loss since we find that other losses like L 1 loss or L kl loss cannot backpropagate gradients for CKG. Additionally, to reduce computation, the gradients of the transformation network are calculate rather than the entire target detector in Eq.(9).

Licenses. The models employed in this paper are available under open licenses: CLIP [47] and GLIP [32] are released under the MIT License, and GDINO [40] is under the Apache License 2.0. The datasets employed in this research are released under various licenses: Cityscapes [11] and Foggy-Cityscapes [11] are available under a non-commercial license; BDD100K [63] is provided under the BSD 3-Clause License for non-commercial use; KITTI [16] is published under the CC BY-NC-SA 3.0 License; Sim10K [26] is available under a custom non-commercial license; and the Clipart [25] is intended for academic use, with specific licensing details to be confirmed with the authors.

A.3.1 More quantitative results.

Quantitative results under GLIP. In order to comprehensively evaluate the effectiveness of the proposed COIN across different cloud detectors, we conduct experiments under GLIP [32]. Since GLIP offers class predictions in the form of confidence score, we convert confidence score into probability by label smoothing. The results are shown in Tables 8 - 9. Compared to GDINO [40], GLIP produces lower performance. However, our COIN still achieves significant improvements, such as +7.1% (from 20.6% to 27.7%) on Foggy-Cityscapes, +6.1% (from 27.4% to 33.5%) on BDD100K, +9.5% (from 24.0% to 33.5%) on Cityscapes, +30.0% (from 26.8% to 56.8%) on KITTI, +20.0% (from 17.1% to 37.1%) on Sim10K. The above results demonstrate the broad applicability of COIN across different cloud detectors.

A.3.2 More quantitative analysis.

Effectiveness under different cloud detector output types. To verify the effectiveness of our COIN under different cloud detector output types, we convert the probability outputs of GDINO to class-only format (converting probability to one-hot format) and conduct experiments on Foggy-Cityscapes, as shown in Table 10. Since confidence score is crucial for evaluating detector's performance, we observe a performance deterioration of GDINO when the output type is class-only. In addition, COIN increases the mAP by +14.3% (from 23.7% to 38.0%) - that is only a 1.0% decrease compared to the probability format. This proves that our COIN is compatible with various cloud detector outputs, making it generally applicable.

Ablation study for prompt templates of CLIP model. As shown in Table 11, we investigate the effectiveness of the proposed 81 prompt templates for CLIP model across six datasets. Experiments are categorized into four groups based on the number of templates and whether style is incorporated. The reported mAPs are calculated by the classification probabilities from CLIP model and the boxes from cloud detector. Not surprisingly, our proposed 81 templates achieve a satisfactory victory. Furthermore, one simple template with style outperforms 81 templates without style on four datasets. This not only demonstrates the importance of target customized prompt templates for the classification of CLIP model but also proves the necessity of adapting CLIP to target domain.

Ablation study for dual prompt learning. As shown in Table 12, to validate the proposed prompt learning, ablation studies are conducted on four main components. (1) For CLIP detector, training based solely on a simple template 'a photo of a [CLS].' achieves the mAP of 27.0%. While utilizing four randomly initialized placeholders improves results by +0.4% (from 27.0% to 27.4%), which suggests that our designed template can assist the CLIP detector in capturing more target domain-specific attributes. (2) When L 1 align is introduced to align the initial class prototypes - class embeddings e c from CLIP, the mAP is further enhanced by +0.3% (from 27.4% to 27.7%), and when aligning the continuously updated prototypes based on EMA and visual features, the mAP is increased by +0.8% (from 27.4% to 28.2%). As anticipated, the experimental results demonstrate that prototypes updated based on visual features capture more domain-specific attributes compared to class embeddings e c calculated by the manually customized 81 CLIP prompt templates. These findings strongly support the effectiveness of our proposed prompt learning. (3) In the knowledge distillation stage, the target detector aligns to visual class prototypes collected based on consistent detections by L 2 align instead of visual prototypes trained in CLIP detector. This alignment ensures that the target detector aligns with the shared knowledge between the cloud detector and CLIP detector. In the last two rows of Table 12, these two scenarios are compared, and as expected, aligning the shared knowledge results in a mAP improvement of +0.7% (from 38.3% to 39.0%) compared to aligning the knowledge of the CLIP detector. This confirms the effectiveness of our second prompt learning.

Ablation study for knowledge separation. To verify the effectiveness of the knowledge separation stage, we design a simple comparative experiment called "Filter and distill" as shown in Table 13. A fixed threshold is used to filter detections. High-confidence detections from any detector are viewed as consistent detections (box fusion is no longer used), while low-confidence detections are viewed as private detections. To avoid the unfair comparison from a specific threshold, we vary five thresholds between 0 and 1. As expected, the overall results are not optimistic. The best performance of 32.6% is achieved when the threshold is set to 0.7, which means our COIN surpasses it by a large margin of +6.4% (from 32.6% to 39.0%). This ablation study strongly demonstrates the effectiveness of our proposed knowledge separation stage and meanwhile validates the power of box matching for separating knowledge from different detectors.

Ablation study for knowledge dissemination. To demonstrate the significance of the knowledge dissemination stage, we conduct an ablation study as shown in Table 14. (1) When directly using CLIP instead of training a CLIP detector through knowledge dissemination, a mAP of 36.5% is achieved. This represents an improvement of +2.1% (from 34.4% to 36.5%) over the cloud detector and +21.1% (from 15.4% to 36.5%) over CLIP. This indicates that COIN can organically integrate the knowledge from both sources even without the knowledge dissemination stage. (2) When knowledge dissemination is included to train a CLIP detector, the performance is improved by +12.8% (from 15.4% to 28.2%) compared to CLIP. With the CLIP detector, COIN further increases the mAP by +2.5% (from 36.5% to 39.0%). This highlights the effectiveness of the knowledge dissemination stage - our designed detector fully utilizes the knowledge from CLIP, and prompt learning mitigates domain shifts, adapting CLIP to the target domain.

Analysis for knowledge dissemination of both CLIP and cloud detector. Since knowledge dissemination for CLIP mitigates domain shift and enhances performance, we consider whether applying knowledge dissemination to the cloud detector brings further improvements. To explore this, we analyze the method of applying knowledge dissemination to both the cloud detector and CLIP.

The results are shown in Table 15. (1) Unsurprisingly, knowledge dissemination for the cloud detector results in a +3.4% improvement (from 34.4% to 37.8%), further demonstrating the broad applicability of the knowledge dissemination stage. (2) For the knowledge separation and distillation stages, since both the cloud detector* and CLIP detector can update parameters through EMA, we list three settings in Table 15. However, the results in all three settings are not ideal. This is because both cloud detector* and CLIP detector are trained based on the same boxes (from the cloud detector), making them prone to similar false positives. This introduces significant noise into the consistent detections, leading to many incorrect predictions by the target detector. Nevertheless, our strategy of updating CLIP detector achieves the best results because updating its parameters improves its performance. In contrast, updating the cloud detector* results in performance degradation. While, when both are updated, EMA causes their parameters to gradually become similar, leading the target detector to get lost in the noise. We think these issues may be mitigated if CLIP is replaced with a model inherently capable of detection ability. (3) Compared to the best result in dual knowledge dissemination, COIN improves performance by +6.7% (from 32.3% to 39.0%), indicating the correctness and superiority of performing knowledge dissemination exclusively for CLIP.

Experimental analysis of the mechanism for gradient alignment. To demonstrate the rationality of the gradient alignment mechanism, we use the gradients generated by the ground truths of inconsistent detections as proxies to represent the direction for inconsistent detections towards the optimal target detector. Thus, we can verify the rationality of this mechanism by calculating the cosine similarity between the above gradients and the gradients of consistent detections. To this end, we compute the aforementioned similarity for each iteration, obtaining an average similarity of 0.527 across 1000 iterations on BDD100K. The corresponding vector angle for this similarity is 58.2 degrees, indicating that the gradient direction of consistent detections has a relatively small angle with respect to the direction of inconsistent detections towards the optimal target detector. This demonstrates the rationality of our gradient alignment mechanism.

Detection consistence of cloud detector and CLIP detector. We analyze the consistence frequency between cloud detector GDINO and CLIP detector. For each iteration, we keep track of whether inconsistent detections occur and calculate the frequency of instances where the CLIP detector makes correct detections but the cloud detector does not, denoted as Cloud(N)/CLIP(P), as well as the frequency of Cloud(P)/CLIP(N) and the frequency of correct detections by CKG, denoted as CKG(P). We then convert the frequencies into the probabilities and calculate the average results over 1000

iterations. The findings are presented in Table 16. We find that inconsistent detections occur in almost every iteration (99.5%), with the probability of Cloud(N)/CLIP(P) being 32.8% and CKG(P) being 80.6%. The above experimental results show that CLIP can indeed benefit knowledge distillation from cloud detector. Moreover, it also proves that CKG works in our knowledge integration process, as it achieves the best results.

Detection speed. In practical applications, the well-trained target detector is utilized in edge devices with relatively low computational power. As a result, the model size and detection speed significantly impact its practicality. As shown in Table 17, we analyze the above two terms on Foggy-Cityscapes with an input size of 600 × 1200 , where the ResNet50 version of target detector is compared. (1) As for params, compared to testing, target detector can directly employ the well-trained class embeddings for classification during deployment. So the text encoder utilized during testing is discarded at deployment, reducing target detector to 40M params, which is just 1 / 6 of the cloud detector. (2) As for detection speed, target detector can also reduce the number of proposals to accelerate detection during deployment (with negligible impact on accuracy). Compared to the 9.5 FPS of cloud detector, target detector reaches a speed of 43.5 FPS, which further underscores the significance of our proposed CODA for real-world applications.

Effectiveness across different versions of cloud detector. To verify the effectiveness of our COIN across different versions of cloud detector, the Swin-T version of GDINO [40] is selected as an alternative to compare with our default selected Swin-B version, as shown in Table 18. Compared to the Swin-B version, the Swin-T version of the cloud detector performs slightly weaker, achieving 26.9% and 36.4% on Foggy-Cityscapes and Cityscapes respectively. Interestingly, the performance of CLIP (using boxes from cloud detector) is not significantly affected, suggesting that the Swin-T version of the cloud detector may not classify correctly due to fewer parameters compared to the Swin-B version. Moreover, our COIN still achieves the best results on both datasets - 33.6% on Foggy-Cityscapes and 39.7% on Cityscapes. This demonstrates the robustness and versatility of our COIN across different versions of cloud detector.

Error bar. To facilitate the reproduction of experimental results, we use a fixed random seed of 2024 in all our experiments. To analyze the error bars introduced thereby, COIN is ran under four randomly generated seeds, and the mean and standard deviation of the results from all five seeds are calculated, as shown in Table 19. Since cloud detector and CLIP are not retrained, they produce the same results under different seeds. For the CLIP detector and COIN, statistical results indicate that their performance conforms to the presupposed Gaussian distribution. This is evidenced by the 1-sigma error bars covering 60% and 80% of the data points, respectively. The above shows that our method can achieve stable results under different random seeds.

A.3.3 More Qualitative analysis.

Qualitative Comparison. To qualitatively verify our methods, we visualize the detection results on six datasets, as shown in Figure 5, where cloud detector, CLIP (using boxes from cloud detector), CLIP detector and COIN are compared. (1) It is clear that COIN produces more true positives compared to the other three, indicating that our method achieves the best results. (2) By comparing the CLIP detector and CLIP, more ground truths are detected, which proves the effectiveness of the

knowledge dissemination stage. (3) Furthermore, by comparing cloud detector, CLIP detector, and COIN, we find that COIN achieves ideal knowledge integration while also detecting some novel boxes, demonstrating the positive impact of our knowledge integration. (4) There are a large number of false positives in the KITTI raw, but when upon magnifying the images for closer inspection, we find that they are not incorrect detections but actual existing objects with the car category. This means that our COIN even detects real objects that are not included in the annotation files! This not only proves the power of COIN but also once again highlights the significance of the proposed problem CODA.

CLIP

CLIP detector

COIN

Foggy

Cityscapes

Cityscapes

SIM

BDD100K

KITTI

A.4 Limitations

Although knowledge dissemination stage grounds detection capability to CLIP and mitigates domain shift, pre-training a CLIP detector introduces additional training time overhead. Fortunately, our COIN is a general method which is not limited to CLIP. When another auxiliary model with inherent detection capability is used, the domain shift can be alleviated with a few steps of fine-tuning, thus the issue of training time overhead is eliminated.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: As shown in abstract, introduction and contribution. The abstract and introduction accurately outline the main claims, which are substantiated by the results presented in the paper, ensuring that the contributions and scope are clearly communicated.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We provide limitations at the end of appendix.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not present any theoretical results; hence, the criteria for providing assumptions and proofs are not applicable.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide detailed model architecture, algorithm and dataset usage in appendix. Moreover, detailed implementation details are provided in main text and appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The test code is provided in supplementary material. The link to the full code will be provided upon the publication of this paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide dataset usage such as training and testing splits in detailed datasets of appendix. Moreover, detailed implementation details are provided in main text and appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report error bar at the end of appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The computer resources, including the GPU model and memory usage, are detailed in the implementation details of appendix. Detection speed (time and FPS) and model size (number of parameters and space occupied) are analyzed in one table from appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research adheres to the NeurIPS Code of Ethics in all aspects, ensuring that ethical considerations are thoroughly addressed and integrated into the study.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Given the purely academic nature of the research, which does not entail direct application or deployment, the discussion of broader societal impacts is deemed not applicable.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not involve the release of data or models that are at high risk for misuse; therefore, the discussion of safeguards is not applicable.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We provide the licenses for all used models and datasets in appendix. The version of models are introduced in implementation details of main text and appendix.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The paper introduces new assets, including test code and detection results, which are well documented as shown in supplementary material. The link to the full assets, including full code and trained models with detailed document and license will be provided upon the publication of this paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing experiments or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing experiments or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CODA, a new problem, where the goal is to adapt object detectors for specific target domains using knowledge from cloud-based detectors. The paper uses CLIP to refine the knowledge from the cloud detector. A gradient alignment method is proposed to deal with the inconsistency between the detection outputs of the adapted CLIP detector and the cloud detector. Extensive experiments show that the proposed CODA approach improves detection performance in the target domain with high computational efficiency for edge devices.\n\nSTRENGTHS:\n1. Good writing and readability. \n2. Novelity of the method: Use CLIP to help refine the knowledge distillation from a cloud detector for a new problem CODA. \n3. Extensive experiments show the effectiveness of the method. \n4. Good computational efficiency for edge devices.\n\nWEAKNESSES:\n1. The definition of the new problem “Cloud Object Detector Adaptation” should be stated more precisely. \n2. The principle of the gradient alignment mechanism should be explained more clearly.\n3. Performance degradation problem of cloud detector for some categories in experiments should be elaborated more clearly.\n\nQUESTIONS:\n1. About the definition of the new problem “Cloud Object Detector Adaptation”. As my understanding, a cloud model like GPT-4 only releases its API to output its prediction, but confidence score or probability distribution of its prediction is probably inaccessible, while the proposed method needs a probability distribution. Black-blox DDOD does not need it. Do you think this will hinder practical application of the proposed method?\n2. About the gradient alignment method for decision-level fusion of inconsistent detections. Experiments in table 4 and table 5 show that CKG with gradient alignment method works. But it still lacks a deep experimental/visualization analysis or a theoretical support of this mechanism. Can you get a conclusion like “The gradient direction of consistent detections is the direction for inconsistent detections towards optimal target detector.” or “The gradient direction of consistent detections leads to the right detection prediction for inconsistent detections with high probability.”?\n3. About the performance degradation problem of cloud detectors for some categories in experiments. For example, in table 1, for 4 categories of BDD100K dataset, there is still a gap between COIN and the cloud detector. An adapted CLIP detector is utilized to refine the cloud detector, as my understanding, is this a sign that some wrong predictions from CLIP hinders the knowledge transferring from the cloud detector for some categories? I’m also curious about the analysis of why CLIP can benefit the knowledge distillation from the cloud detector. How often the inconsistence happens? How often the adapted CLIP detector can make the right detection but cloud detector cannot?\n\nLIMITATIONS:\n1. Clarity of the new problem definition.\n2. Lack of analysis of part of mechanism design and the idea.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper proposes a new problem in the field of domain adaptation, called Cloud Object Detector Adaptation (CODA), where a cloud model is provided to help with target detector training. A novel method termed COIN is proposed to leverage CLIP for knowledge distillation in a divide-and-conquer manner. Sufficient experiments have proven the effectiveness of the proposed method.

STRENGTHS:
- **Good Presentation**. This paper is well-organized, with clear and effective writing, complemented by appealing figures and tables.
- **Sufficient Experiments**. The paper conducts experiments on six validation datasets, demonstrating that the proposed method achieves state-of-the-art performance. Ablation studies further illustrate the effectiveness of the proposed components.

WEAKNESSES:
- **Method Generalization Ability**. The observation that both the CLIP detector and the target detector are based on Faster R-CNN, which utilizes the visual encoder of CLIP, is a valid point. While this design leverages the open-set capability of CLIP, it raises a fair concern that it may limit the method's applicability to DETR-based detection approaches.
- **Detailed Problem Settings** The paper should provide more detailed problem settings for CODA, such as the differences in scenes and category gaps between the pretrained data used by the cloud detector and the validation set.

QUESTIONS:
1. Can the cloud model be replaced by a large Visual Language Model (VLM), such as GPT-4V?
1. The performance of state-of-the-art (SOTA) supervised methods on the validation dataset should be listed as the oracle performance.

LIMITATIONS:
This paper provides limitations at the end of appendix.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

comments: SUMMARY:
In this paper, the authors propose a Cloud Object Detector Adaptation framework, which leverages a strong detector in the cloud to extract discrimative knowledge of objects in the target domain. It is basically a mean teacher style for domain adaptive object detection.

STRENGTHS:
1 This paper considers an important object detection problem in the target domain.
2 The experiments are extensive.

WEAKNESSES:
The design of this work basically follows the mean teacher style for self-distillation with EMA. Basically, the contributions are not convincing. 

1) Using CLIP model to build detector. It is a straightforward extension in the Faster RCNN style as shown in Fig 2 (a). 

2) Using a strong detector in the cloud. This allows to generate more discriminative box supervisions of objects in the target domain.  It is also straightforward to obtain. 

3) The only interesting part is knowledge seperation. The idea is also simple to discover consistent, private and inconsist boxes via matching. Distillation is operated on different types of these boxes.

Hence, the overall framework brings little new insightful design by the incremental addition of object detector in the cloud.

QUESTIONS:
See the weakness section.

LIMITATIONS:
There is no potential negative societal impact of this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel task: Cloud Object Detector Adaptation (CODA). It discusses how to build a domain specific object detector with the help of a cloud detector, and local data from the target domain. Different from existing similar tasks, CODA does not have the full access of the cloud model, only detection outputs are used. Furthermore, it aims to transfer the model to a target domain with large domain gap.\n\nThe paper then presents a novel Cloud Object detector adaptation method by Integrating different source kNowledge (COIN). The key idea is to incorporate a public vision-language model (CLIP) to refine the knowledge for adaptation. Experiment results show that the proposed method achieve SOTA performance.\n\nSTRENGTHS:\n1. This paper proposes a novel task: Cloud Object Detector Adaptation (CODA). It is different from existing tasks, by considering the model privacy issue.\n\n2. The pipeline consists of knowledge dissemination, separation and distillation stages. By including CLIP, the design seems reasonable, and novel. The final results show that the method is effective.\n\nWEAKNESSES:\n1. The cloud detector is discussed in general through the whole paper. And because the model is designed purely on the detection output of the cloud detector. It should work on different detectors. So the whole architecture should be validated on different cloud detectors. And the experiment section in the main paper is too short. It should focus more on different settings about this new benchmark.\n\n2. The methodology part is a bit confusing, specifically for the two detectors. CLIP and Target detectors are both introduced in Sec 3.1 without specific discussion of how they are used or different from each other. And the training of target detector is not further discussed until Sec 3.3.\n\nQUESTIONS:\nThe most critical part is the choice of cloud detector (Weaknesses 1). As a novel benchmark, it is better to study the problem with more settings.\n\nLIMITATIONS:\nLimitations are discussed in Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Cloud Object Detector Adaptation, in which a cloud model is responsible for detection in the target domain. The proposed framework, COIN, includes successive stages including knowledge dissemination, separation, and distillation. The target detector leverages a cloud detector and CLIP through a box matching mechanism, categorizing detections into consistent, inconsistent, and private categories. The consistent and private detections are utilized to train the target detector, while the inconsistent detections are refined using a consistent knowledge generation network. A gradient direction alignment loss is proposed to optimize consistent knowledge generation\n\nSTRENGTHS:\n+ This paper presents an interesting integration of dissemination, separation, and distillation processes for the given task.\n+ The organization and presentation of the paper are good, showcasing a clear and structured approach. \n+ The experimental evaluation, conducted on foggy Cityscapes, BDD100k, and other additional datasets, demonstrates the method's effectiveness\n\nWEAKNESSES:\n+ The overall optimization loss encompasses multiple values, making the task of identifying optimal hyperparameters for each dataset hard and challenging.\n+ While this paper discusses dissemination, separation, and distillation, these concepts are not novel. \n+ Instead, the work appears to integrate existing components to enhance performance without providing a strong contribution specific to this work. The methods employed have been previously utilized in domain generalization and domain adaptation literature, e.g., works like “CLIP the Gap: A Single Domain Generalization Approach for Object Detection” (CVPR 2023) and “SSAL: Synergizing between Self-Training and Adversarial Learning for Domain Adaptive Object Detection” (NeurIPS 2021).\n+ Could the authors clarify the principle differences between their approach and prior methodologies? It is essential to highlight the specific contributions of this work to distinguish it from established techniques in a similar area.\n\nQUESTIONS:\nPlease see the weakness section. I hope to see a good rebuttal, that majorly addresses the issues regarding how the optimization loss in this work addresses the challenge of identifying optimal hyperparameters for each dataset? Also what specific contributions does this work provide that distinguish it from existing techniques in domain generalization and domain adaptation?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-16"}