Domain Adaptation for Large-Vocabulary Object Detectors

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 08:04:10

Abstract: Large-vocabulary object detectors (LVDs) aim to detect objects of many categories, which learn super objectness features and can locate objects accurately while applied to various downstream data. However, LVDs often struggle in recognizing the located objects due to domain discrepancy in data distribution and object vocabulary. At the other end, recent vision-language foundation models such as CLIP demonstrate superior open-vocabulary recognition capability. 
This paper presents KGD, a Knowledge Graph Distillation technique that exploits the implicit knowledge graphs (KG) in CLIP for effectively adapting LVDs to various downstream domains.
KGD consists of two consecutive stages: 1) KG extraction that employs CLIP to encode downstream domain data as nodes and their feature distances as edges, constructing KG that inherits the rich semantic relations in CLIP explicitly; 
and 2) KG encapsulation that transfers the extracted KG into LVDs to enable accurate cross-domain object classification. 
In addition, KGD can extract both visual and textual KG independently, providing complementary vision and language knowledge for object localization and object classification in detection tasks over various downstream domains. 
Experiments over multiple widely adopted detection benchmarks show that KGD outperforms the state-of-the-art consistently by large margins. 
Codes will be released.

Main paper: 
 introduction: 
 Object detection aims to locate and classify objects in images, which conveys critical information about "what and where objects are" in scenes. It is very important in various visual perception tasks in autonomous driving, visual surveillance, object tracking, etc. Unlike traditional object detection, large-vocabulary object detection Li et al. (2022b) ; Yao et al. (2022) ; Zhou et al. (2022) aims to detect objects of a much larger number of categories, e.g., 20k object categories in Zhou et al. (2022) . It has achieved very impressive progress recently thanks to the availability of large-scale training data. On the other hand, large-vocabulary object detectors (LVDs) often struggle while applied to various downstream tasks as their training data often have different distributions and vocabularies as compared with the downstream data, i.e., due to domain discrepancies.In this work, we study unsupervised domain adaptation of LVDs, i.e., how to adapt LVDs various downstream tasks with abundant unlabelled data available. Specifically, we observe that LVDs learn superb generalizable objectness knowledge from massive object boxes, being able to locate objects in various downstream images accurately Zhou et al. (2022) . However, LVDs often fail to classify the located object due to two major factors: 1) the classic dataset-specific class-imbalance and the resultant distribution bias across domains; and 2) different vocabularies across domains Oksuz et al. (2020) ; You et al. (2019) . At the other end, vision-language models (VLMs) Zhang et al. (2023) such as CLIP Radford et al. (2021) learn from web-scale images and text of arbitrary categories, which achieve significant generalization performance in various downstream tasks with severe domain shifts. Hence, effective adaptation of LVDs towards various unlabelled downstream domains could be facilitated by combining the superior object localization capability from LVDs and the super-rich object classification knowledge from CLIP.We design Knowledge Graph Distillation (KGD) that explicitly retrieves the classification knowledge of CLIP to adapt LVDs while handling various unlabelled downstream domains. KGD works with one underling hypothesis, i.e., the generalizable classification ability of CLIP largely comes from its comprehensive knowledge graph learnt over billions of image-text pairs, which enables it to classify objects of various categories accurately. In addition, the knowledge graph in CLIP is implicitly encoded in its learnt parameters which can be exploited in two steps: 1) Knowledge Graph Extraction (KGExtract) that employs CLIP to encode downstream data as nodes and computes their feature distances as edges, constructing an explicit CLIP knowledge graph that captures inherent semantic relations as learnt from web-scale image-text pairs; and 2) Knowledge Graph Encapsulation (KGEncap) that encapsulates the extracted knowledge graph into object detectors to enable accurate object classification by leveraging relevant nodes in the CLIP knowledge graph.The proposed KGD allow multi-modal knowledge distillation including Language Knowledge Graph Distillation (KGD-L) and Vision Knowledge Graph Distillation (KDG-V). Specifically, KGD-L considers texts as nodes and the distances among text embeddings as edges, enabling detectors to reason whether a visual object matches a text by leveraging other relevant text nodes. KGD-V takes a category of images as a node and the distances among image embeddings as edges, which enhances detection by conditioning on other related visual nodes. Hence, KGD-L and KGD-V complement each other by providing orthogonal knowledge from language and vision perspectives. In this way, KGD allows to explicitly distill generalizable knowledge from CLIP to facilitate unsupervised adaptation of large-vocabulary object detectors towards distinctive downstream datasets.In summary, the major contributions of this work are threefold. First, we propose a knowledge transfer framework that exploits CLIP for effective adaptation of large-vocabulary object detectors towards various unlabelled downstream data. To the best of our knowledge, this is the first work that studies distilling CLIP knowledge graphs for the object detection task. Second, we design novel knowledge graph distillation techniques that extracts visual and textual knowledge graphs from CLIP and encapsulates them into object detection networks successfully. Third, extensive experiments show that KGD outperforms the state-of-the-art consistently across 10 widely studied detection datasets. Zhang et al. (2021b) . Specifically, several losses have been proposed, such as Equalization losses Tan et al. (2020; 2021) , SeeSaw loss Wang et al. (2021), and Federated loss Zhou et al. (2021) . On the other hand, Yang et al. (2019a) and Detic Zhou et al. (2022) attempt to introduce additional image-level datasets with large-scale fine-grained classes for training large-vocabulary object detector (LVD), aiming to expand the detector vocabulary to tens of thousands of categories. These LVDs learn superb generalizable objectness knowledge from object boxes of massive categories and are able to locate objects in various downstream images accurately Zhou et al. (2022) . However, they often fail to classify the located objects Oksuz et al. (2020) ; You et al. (2019) accurately. In this work, we focus on adapting LVDs towards various unlabelled downstream data by utilizing the super-rich object classification knowledge from CLIP. Domain Adaptation aims to adapt source-trained models towards various target domains. Previous work largely focuses on unsupervised domain adaptation (UDA), which minimizes the domain discrepancy by discrepancy minimization Long et al. (2015) ; Vu et al. (2019) , adversarial training Gong et al. (2019) ; Vu et al. (2019) ; Luo et al. (2021) , or self-training Lee (2013) ; Zhang et al. (2019; 2021a) . Recently, source-free domain adaptation (SFDA) generates pseudo labels for target data without accessing source data, which performs domain adaptation with entropy minimization Liang et al. (2020) , self training Tarvainen & Valpola (2017) ; Li et al. (2021) , contrastive learning Huang et al. (2021); VS et al. (2022) , etc. However, most existing domain adaptation methods struggle while adapting LVDs toward downstream domains, largely due to the low-quality pseudo labels resulting from the discrepancy in both data distributions and object vocabulary.Vision-Language Models (VLMs) have achieved great success in various vision tasks Zhang et al. (2023) . They are usually pretrained on web-crawled text-image pairs with a contrastive learning objective. Representative methods such as CLIP Radford et al. (2021) and ALIGN Jia et al. (2021) have demonstrated very impressive generalization performance in many downstream vision tasks.  
method: 
 Task Definition. This paper focuses on unsupervised adaptation of large-vocabulary object detectors (LVDs). We are provided with a set of unlabeled downstream domain data D t = {x t i } Nt i=1 and an LVD pre-trained on labeled source domain detection dataset D s = {x s i , y s i } Ns i=1 . x i and y i = {(p j , t j )} M j=1 are the image and M instance annotations of i-th sample, where p j and t j denote the ground-truth category and box coordinate of j-th instance. N s and N t refer to the number of samples in D s and D t . The goal is to adapt the pretrained LVD towards the downstream domain D t by using the unlabelled images.Naïve Solution with Mean Teacher Method (MT) Tarvainen & Valpola (2017) . In this paper, we adopt Detic Zhou et al. ( 2022) as the pretrained LVD, which utilizes CLIP text embeddings as the classifier. We employ mean teacher Tarvainen & Valpola (2017) as the preliminary solution, which involves a teacher detector and a student detector where the former generates pseudo labels to train the latter while the latter updates the former in a momentum manner. Given a batch of B unlabeled target samples, the teacher detector Φ t first produces detection predictions on them, which are then filtered with a predefined threshold τ to generate detection pseudo label ŷi (consisting of classes and bounding boxes). With ŷi , the unsupervised training of student detector Φ s on the unlabeled downstream data can be formulated as the following:EQUATIONwhereL(•) = L rpn (•) + L reg (•) + L cls (•) is the detection loss function in which L rpn (•), L reg (•),and L cls (•) denote the loss for region proposal network, regression, and classification, respectively. Note both teacher detector Φ t and student detector Φ s are initialized with the pretrained LVD.Motivation. On the other hand, although the LVD is able to locate objects in various downstreamdomain images accurately Zhou et al. ( 2022), it often fails to classify the located objects, leading to very noisy detection pseudo labels when serving as the teacher detector. At the other end, vision-language models (VLMs) Zhang et al. (2023) such as CLIP Radford et al. ( 2021) learns from web-scale images-text pairs of arbitrary categories, which possesses the ability to classify objects accurately in various downstream data. Thus, we argue that effective adaptation of LVDs towards various unlabelled downstream data could be facilitated by combining the superior object localization capability from LVDs and the super-rich object classification knowledge from CLIP. To this end, we design Knowledge Graph Distillation (KGD) with Language KGD and Vision KGD, aiming to explicitly retrieves the classification knowledge of CLIP to adapt LVDs while handling various unlabelled downstream data. The overview of our proposed KGD is shown in Figure 1 .The proposed language knowledge graph distillation (KGD-L) aims on distilling knowledge graph from the perspective of text modality. KGD-L works in a two-step manner. EQUATIONwhere WNRetrieve(•) retrieves the WordNet database Miller (1995) and returns the definition of its input. In this way, a category name c i can be better defined and described with the informative yet accurate category definition from WordNet. Given a category name c i and its definition d i , we query the LLM with the following prompt template:Generate brief descriptions for the appearance of [m] types of [c i ], [d i ], in the context of [context].where [context] is a dataset description phrase (e.g., street scenes for Cityscapes Dataset) and m is fixed as 5 for all categories. With the given prompt template, we let LLM generate m domaingeneralizable prompts for category c i conditioned on its WordNet Synset definition and context:EQUATIONwhich is then combined with d i as a set of domain generalizable prompts for category c i :EQUATIONand the domain generalizable prompt set of category set C can be constructed as the following:EQUATIONWith the category-discriminative and domain-generalizable information contained in S, we formulate the proposed LKG as a weighted undirected graph G L = (V L , U L ), which is capable of capturing semantic relationships and associations between different category concepts.V L = {s i } Nc(m+1) i=1is the vertex set in which each node si refers to a description in S. And U L = {u ij } is the edge set where each edge u ij denotes the feature cosine similarity between the nodes si and sj :EQUATIONwhere T (•) refers to the text encoder of CLIP, and δ is the temperature parameter set as Radford et al. (2021) .LKG Encapsulation encapsulates the comprehensive knowledge in the extracted LKG into the teacher detector to facilitate detection pseudo label generation. Specifically, we first employ CLIP to encode the regions cropped by the teacher detector and then generate pseudo labels for each region feature conditioned on LKG. Given the image x t ∈ D t , we feed it into the teacher detector Φ t to acquire the prediction as the following:EQUATIONwhere ŷ = {(p j , tj )} M j=1 , pj denotes the probability vector of the predicted bounding box tj after Softmax activation function. M denotes the number of predicted proposals after the thresholding with τ , i.e., a predicted proposal will be discarded if its confidence score is less than τ .Next, we employ CLIP to encode the predicted object proposals in ŷ as follows:EQUATIONwhere Crop(•) crops square regions from image x t based on the longer edges of bounding boxes in ŷ, V (•) is the image encoder of CLIP, and the j-th column vector f j of matrix F is the feature of j-th proposal in ŷ.With the extracted LKG G L and the features of objects (or object proposals) F , we encapsulate the extracted LKG into Φ t by reasoning the class of each object conditioned on G L as follows:EQUATIONwhere N (•) refers to normalize data to range [0, 1]. pji is the i-th element in probability vector pj , which denotes the predicted category probability of c i . The first term in Eq. 9 denotes the original prediction probability from the teacher model while the second term in Eq. 9 stands for the prediction probability from LKG. p l ji denotes the prediction probability calibrated by LKG. In this way, KGD-L extracts and encapsulates LKG from CLIP into the teacher detector, enabling it to reason whether an object matches a category conditioned on the relevant nodes in LKG and ultimately refining the original detection pseudo labels.As LKG captures language knowledge only, we further design vision knowledge graph distillation (KGD-V) that extracts a vision knowledge graph (VKG) and encapsulates it into the teacher detector to improve pseudo label generation. Specifically, VKG captures vision knowledge dynamically along the training process, which complement LKG by providing orthogonal and update-to-date vision information.Dynamic VKG Extraction. We first initialize VKG with the CLIP text embedding and then employ the update-to-date object features to update it using manifold smoothing. Specifically, we initialize VKG as a weighted undirected graph G V = (V V , U V ), in which each node v i ∈ V V is initialized with the CLIP text embedding of category c i :EQUATIONand the graph edge u ij ∈ U V is defined as the cosine similarity between nodes v i and v j . Given a batch of {x t b } B b=1 ⊆ D t and the corresponding pseudo labels {ŷ b } B b=1 and CLIP features {F b } B b=1 , the visual embedding centroid of category c k can be obtained as the following:EQUATIONwhere pmax j is the maximum element in probability vector pj , I is the indicator function. And an affinity matrix A can be calculated as A ij = exp(-r 2 ij /σ 2 ) and A ii = 0, where r ij = ||θ i -θ j || 2 and σ 2 = Var(r 2 ij ). In each iteration, the node of VKG is preliminarily updated as:EQUATIONIn order to incorporate the downstream visual graph knowledge into VKG, we perform additional steps to smooth the node of VKG, using the affinity matrix A from the current batch as a guide:EQUATIONwhere VKG Encapsulation encapsulate the orthogonal and update-to-date vision knowledge in the extracted VKG into the teacher detector, which complements LKG and further improves pseudo label generation. With the extracted dynamic VKG G V and the object features F in image x t , we encapsulate the extracted VKG into Φ t in a similar way as the LKG Encapsulation as follows:W = (I -αL) -1 , L = D -1 2 AD -1 2 , D ii = j A ij ,EQUATION)where pji is the i-th element in vector pj , denoting the predicted probability of category c i . The first term in Eq. 14 is the prediction probability from the teacher model while the second term in Eq. 14 is the prediction probability from VKG. p v ji is the prediction probability calibrated by VKG. In this way, KGD-V extracts and encapsulates the VKG from CLIP into the teacher detector, further refining the detection pseudo labels of visual objects by conditioning on related visual nodes in VKG.Finally, with the pseudo labels p l j and p v j generated from KGD-L and KGD-V respectively, the unsupervised training loss of KGD can be formulated as the following:EQUATIONwhere ỹ = {(p j , tj )} M j=1 , and pj = N (p l j + p v j ). 
experiments: 
 This section presents experimental results. Sections 4.1 and 4.2 describe the dataset and implementation details. Section 4.3 presents the experiments across various downstream domain datasets. Section 4.4 and Section 4.5 provide ablation studies and discuss different features of KGD. 2021) is pre-trained on LVIS Gupta et al. (2019) for object localization and ImageNet-21K Deng et al. (2009) for object classification. During adaption, the updating rate of EMA detector is set as 0.9999. The pseudo labels generated by the teacher detector with confidence greater than the threshold τ = 0.25 are selected for adaptation. We use AdamW Loshchilov & Hutter (2017) optimizer with initial learning rate 5 × 10 -6 and weight decay 10 -4 , and adopt a cosine learning rate schedule without warm-up iterations. The batch size is 2 and the image's shorter side is set to 640 while maintaining the aspect ratio unchanged.Tables 1 2 3 show the benchmarking of our methods with state-of-the-art domain adaptive detection methods. As there are few prior studies on LVD adaptation, we compare our proposed method with state-of-the-art source-free domain adaptation methods for benchmarking, including Mean Teacher (MT) Tarvainen & Valpola (2017) , SHOT Liang et al. (2020) , SFOD Li et al. (2021) , HCL Huang et al. (2021), and IRG-SFDA VS et al. (2022) . For fair comparison, we incorporate CLIP Radford et al. (2021) and LLM Brown et al. (2020) into the compared methods (marked with †). Specifically, we employ LLM Brown et al. (2020) to generate category descriptions given category names, and CLIP Radford et al. (2021) to predict pseudo labels for object classification.Object detection for autonomous driving. As Table 1 shows, the proposed KGD outperforms the baseline substantially over the general autonomous driving datasets Cityscapes and Vistas (with an average improvement of 6.20 in AP50). KGD also outperforms the state-of-the-art by 2.35 on average, demonstrating the superiority of KGD in adapting pretrained LVDs toward autonomous driving scenarios with substantial inter-domain discrepancy. In addition, Table 1 shows experiments on autonomous driving data under various weather and time conditions. We can observe that KGD still achieves superior detection performance even though the unlabeled target data experience large style variation and severe quality degradation. Further, the experiments show that KGD still outperforms the state-of-the-art clearly when CLIP and LLM are incorporated, validating that the performance gain largely comes from our novel knowledge graph distillation instead of merely using CLIP and LLM. Object detection for intelligent surveillance. The detection results on intelligent surveillance datasets are presented in Table 2 . Notably, the proposed KGD surpasses all other methods by significant margins, which underscores the effectiveness of KGD in adapting the pretrained LVD towards the challenging surveillance scenarios with considerable variations in camera lenses and angles. The performance improvements achieved by KGD in this context demonstrate its effectiveness in exploring the unlabeled surveillance datasets by retrieving the classification knowledge of CLIP.Object detection for common objects. We evaluate the effectiveness of our KGD on the common object detection task using Pascal VOC and Objects365. Table 3 reports the detection results, showcasing significant improvements over the baseline and outperforming state-of-the-arts, thereby highlighting the superiority of KGD. Besides, we can observe that the performance improvements on the Pascal VOC dataset and Objects365 dataset are not as significant as those in autonomous driving. This discrepancy is attributed to the relatively smaller domain gap between common objects and the pretraining dataset of LVD.Object detection for artistic illustration. Table 3 reports the detection results on artistic illustration datasets. The proposed KGD outperforms all other methods by substantial margins, which highlights the effectiveness of KGD in adapting the pretrained large-vocabulary object detector towards artistic images that exhibit distinct domain gaps with natural images.In Table 4 , we conducted ablation studies to assess the individual contribution of our proposed KGD-L and KGD-V on the task of LVD adaptation. The pretrained LVD (i.e., Detic Zhou et al. (2022) without adaptation) does not perform well due to the significant variations between its pretraining data and the downstream data, As a comparison, either KGD-L or KGD-V brings significant performance improvements (i.e., +6.3 of AP50 and +6.2 of AP50 over the baseline), demonstrating both language and vision knowledge graphs built from CLIP can clearly facilitate the unsupervised adaptation of large-vocabulary object detectors. The combination of KGD-L and KGD-V performs the best clearly, showing that our KGD-L and KGD-V are complementary by providing orthogonal language and vision knowledge for regularizing the unsupervised adaptation of LVDs.Comparisons with existing CLIP knowledge distillation methods for detection. We compared our KGD with existing CLIP knowledge distillation methods designed for detection tasks. Most existing methods achieve CLIP knowledge distillation by mimicking its feature space, such as VILD Gu et al. (2021 ), RegionKD Rasheed et al. (2022) , and OADP Wang et al. (2023) . Qualitative experimental results. We present qualitative results of KGD over diverse downstream domain detection datasets in the Appendix. The qualitative results illustrate the effectiveness of KGD in producing accurate detection results across various domains, thereby qualitatively demonstrating its capability to adapt LVDs to unlabelled downstream domains with significant discrepancy in data distribution and vocabulary.Parameter studies. In the pseudo label generation in KGD, the reliable pseudo labels are acquired with a pre-defined confidence threshold τ . We studied τ by changing it from 0.15 to 0.35 with a step of 0.05. Distance metrics for constructing Knowledge Graph. We explore the feature distance metrics for constructing knowledge graphs. We conduct experiments that construct knowledge graphs with the following feature distance metrics: 1) Cosine Similarity Deza et al. (2009) , 2) Euclidean Distance Deza et al. (2009) , 3) Manhattan Distances Deza et al. (2009) . The results in Table 9 show that our KGD works effectively and consistently with different feature distance metrics. Besides, the cosine similarity metric performs the best, largely because CLIP is also trained with cosine similarity where using the same metric to distill its knowledge works the best reasonably. Analysis of KGD with respect to the parameter m. In Eq. 3, the parameter m controls the number of domain-generalizable prompts generated by LLM for a certain category. It plays a critical role in balancing the trade-off between generalization and discrimination in pseudo label denoising with our KGD. We study m by changing it from 1 to 9 with a step of 2 and the table below shows results on Cityscapes. It shows that the increase of m improves the performance clearly.Training and inference time analysis. We study the training and inference time of all the compared methods, and Table 11 shows the results on Cityscapes. It shows that incorporating CLIP into https://openclipart.org/https://pixabay.com/ 
 
 Appendix:  


Decision: This work exploits the implicit knowledge graphs (KG) in CLIP to effectively adapt LVDs to various downstream domains. During the rebuttal period, authors have addressed the concerns on computation overhead and insufficient and unfair experimental comparison raised by a few reviewers and all reviewers rate positively. Authors should include all results in the rebuttal in the revision. After careful consideration, The AC is inclined to accept this paper.

Reviews: review: paper_summary: paper_summary: The authors propose a method for domain adaptation for large-vocabulary object detectors. To perform the adaptation, the authors first construct a Language and a Vision graph from the set of classes of the target dataset. The language graph is built with nodes as the description of the target class and the hyponym set of the class as retrieved from WordNet, and each description is encoded with the text encoder of CLIP to define the nodes’ features. This graph is exploited to adapt and improve the predictions on an image with a graph convolutional network (GCN). The Vision graph is initialized with nodes for all classes of the target dataset, with nodes represented by the text embedding of each class which are then adapted using the visual embedding centroid of each category. The similarity between the bounding boxes’ features and these adapted visual nodes’ representation is exploited to improve the class predictions. Experiments conducted on multiple datasets with multiple baseline comparisons show state-of-the-art performance of the proposed method. Ablation studies also show the contribution of each component of the method.
strength_weakness: strength_weakness: - The main weakness of this paper may be in the terminology used by the authors. For example, the authors talk about Knowledge Graphs for things that are not aligned with the common meaning of these terms, as the edges in the LKG and VKG graphs do not convey semantic relationships but are simply affinity edges based on the distance between the nodes' representation. On the contrary, the authors could name more explicitly the underlying methods and principles adopted in their approach, e.g. the step in Eq. (7) seems to perform a semi-supervised label propagation with a graph convolutional network, and Eq. (13) is performing embedding propagation. The term “encapsulation” may not be well adapted either, and it also hides the main visual adaptation steps in Eq. (11)  and (12). Overall, this hinders the readability of the paper and induces confusion.
- Some elements of the method should be detailed (see questions)
- Some experimental results could be presented more clearly (see questions)
questions: questions: - Figure 2, what do the dashed reddish lines between LKG and VKG represent?
- In the WordNetRetrieve step, it is unclear if the hyponym set just contains the hyponym class names, their descriptions, or both. Similarly, is the initial class name used?
- p6 Eq. (12), what is the value of $\lambda$? How is it selected? How sensitive is the method to this value?
- p6, under Eq. (13), from the reference (54) it is not immediately clear what is the value of $\alpha$, how it selected, and how sensitive the method would be to this value. The authors should discuss this directly in the paper.
- Ablation studies tables (Table 3, 5, 6, 7) are somewhat redundant and hard to read due to the layout. The authors could invert rows and columns and combine all these tables into one to improve readability. Interestingly, it seems the most simple “LKG Extraction with category name” and “Static VKG Extraction” already show significant improvement over the baseline 46.5 -> 51.9, while adding all the other elements of the method brings less than one more point of performance. Do the authors have the results of using these two jointly? How far would that be from the whole KGD (Ours) results? Ablation results are only reported on the Cityscapes dataset, have the authors conducted similar analyses on the other datasets? If so, is the behavior consistent across datasets?

Typos etc:
- p6-l195: this sentence is unclear “the update-to-date object features to update it using manifold smoothing”
- p9-l306-311: use “smoothing” instead of “smooth”
limitations: limitations: Yes, in A.4.8.
paper_summary: paper_summary: This paper addresses the challenges faced by Large-vocabulary object detectors (LVDs) in recognizing objects across diverse categories, particularly due to domain discrepancies in data distribution and object vocabulary. The paper proposes a novel approach named Knowledge Graph Distillation (KGD) that leverages the implicit knowledge graphs (KG) within the CLIP model to enhance the adaptability of LVDs to various downstream domains. The paper validated the effectiveness on a wide range of datasets.
strength_weakness: strength_weakness: 1. The paper has not clearly explained the motivation for using knowledge graphs as a tool. If the goal is to employ LVDs for various downstream tasks involving unlabeled data, one potential approach could be to use CLIP to obtain pseudo-labels and then incrementally update the LVDs through semi-supervised learning. In this scenario, it appears that a specifically designed knowledge graph may not be necessary.

2. The methods used for comparison in the experiments are outdated, with no works from 2023 or 2024 included. This makes it difficult to adequately demonstrate the effectiveness of the proposed method.

3. The knowledge graph distillation technique is likely based on prior research in the field. From a design perspective, it would be beneficial if the paper could emphasize the improvements the proposed method offers over previous approaches in this area.

4. The ablation experiments are not sufficiently comprehensive. It would be advantageous to investigate the performance of using only Detic+MT without incorporating the knowledge graph distillation technique.

5. Cropping the object regions and using CLIP for recognition could considerably slow down the training speed of the model. The paper should offer a detailed explanation regarding this aspect.
questions: questions: Please refer to weaknesses.
limitations: limitations: No.
paper_summary: paper_summary: This paper highlights the problem that in domain adaptation, detectors often correctly localize but misclassify. To solve this problem, this paper proposes the Knowledge Graph Distillation method, which uses the pre-trained knowledge of VLM to supplement the relational knowledge of various categories in vision and language required for object detection. Experiments over multiple benchmarks validate the effectiveness.
strength_weakness: strength_weakness: 1 Computation overhead: Although the paper shows the training time and inference speed compared with other methods in Table 11, it does not compare memory usage and computational overhead.

2 Insufficient and unfair experimental comparison. The proposed KGD utilizes the strong generalization capability of the visual-language model (VLM) in the region (proposal). Therefore, it is unfair to compare directly with the traditional source-free domain adaptation method as they did not make any special design for VLM. The proposed KGD should be compared with methods based on visual-language models, such as RegionCLIP, which also uses the idea of ​​regional distillation. However, the comparison with methods such as RegionCLIP is only conducted on the Cityscapes dataset (Table 9), and the improvement is not particularly obvious (1.7%).
questions: questions: 1 I would like to know, if traditional methods (such as MT, etc.) are applied to methods like RegionCLIP, what is the performance gap with the proposed KGD?
limitations: limitations: Yes
paper_summary: paper_summary: This paper addresses the domain generalization problem of large-vocabulary object detectors. Without requiring additional annotations, it uses the Knowledge Graph Distillation technique to transfer knowledge from CLIP, enhancing the detector's generalization capability on downstream datasets.
strength_weakness: strength_weakness: 1. Speed: Although the appendix provides training times, considering that Equation 6 requires cropping proposals from the detector and sending them into CLIP, this step is very time-consuming (as seen in VILD[1], which took almost a few days on the COCO dataset). Given that the training set includes much larger datasets like Object365, how was this training time calculated?
2. Performance: As shown in VILD and other open-vocabulary papers, adding CLIP scores improves detection performance on novel classes. Considering this paper also uses CLIP to assist in out-of-domain detection, and the compared methods do not use CLIP, directly embedding CLIP scores as a baseline to highlight the necessity of GCN is essential.

[1] Open-vocabulary Object Detection via Vision and Language Knowledge Distillation
questions: questions: see weakness.
limitations: limitations: yes, the authors has adequately discussed the potential negative societal impact of their work and limitations.

Comments: title: Response to Weaknesses-2&3
Response to Question-5
comment: Thanks for the detailed response which has addressed most of my concerns. Nevertheless, the contribution is still believed to be not significant. I'd like to raise my rating to borderline accept.
I appreciate the author's efforts in addressing all of concerns during rebuttal - most of which with satisfactory explanations. I currently intend to keep the rating positive.
Thanks for the authors' response. I will raise my score to 5.
I have read all the reviewers' comments and the authors' answers, I believe my initial rating of `6: Weak Accept` is a fair assessment for this contribution and thus maintain it.
**Response to Weaknesses-2:**

Thanks for your comments.
As suggested, we introduce the Periodically Exchange Teacher-Student (PETS) [Ref 1] and Target Prediction Distribution Searching (TPDS) [Ref 2] as the comparison methods. 
As the table below shows, KGD consistently outperforms PETS [Ref 1] and TPDS [Ref 2]. 

In the tables below, $\star$ signifies that the methods employ WordNet to retrieve category definitions given category names, and CLIP to predict classification pseudo labels for objects. We adopt AP50 in evaluations. The results of all methods are acquired with the same baseline (Detic[81]) as shown in the first row.
We will include the new experiments into the updated paper later. Thank you for your suggestion!


[Ref 1] Liu, Qipeng, et al. "Periodically exchange teacher-student for source-free object detection." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[Ref  2] Tang, Song, et al. "Source-free domain adaptation via target prediction distribution searching." International journal of computer vision 132.3 (2024): 654-672.

Method |Cityscapes |Vistas |BDD100K-weather-rain|BDD100K-weather-snow|BDD100K-weather-overcast|BDD100K-weather-cloudy|BDD100K-weather-foggy |BDD100K-time-of-day--daytime |BDD100K-time-of-day--dawn&dusk|BDD100K-time-of-day--night 
-|-|-|-|-|-|-|-|-|-|-
Detic (Baseline) |46.5 |35.0 |34.3 |33.5 |39.1 |42.0 |28.4 |39.2 |35.3 |28.5
PETS |50.2 |35.8 |34.4 |33.9 |40.1 |43.0 |36.3 |39.7 |35.7 |27.8
PETS$\star$ |50.8 |37.4 |35.9 |36.3 |41.0 |42.8 |36.7 |40.9 |37.2 |27.7
TPDS |50.1 |36.0 |35.8 |35.2 |40.0 |42.1 |36.4 |40.4 |36.5 |28.5
TPDS$\star$ |50.3 |37.1 |35.6 |35.9 |40.5 |43.4 |36.9 |41.3 |36.7 |28.9
KGD (Ours) |53.6 |40.3 |37.3 |37.1 |44.6 |48.2 |38.0 |46.6 |41.0 |31.2


Method |Common Objects: VOC|Common Objects: Objects365 |Intelligent Surveillance: MIO-TCD |Intelligent Surveillance:BAAI  |Intelligent Surveillance:VisDrone |Artistic Illustration:Clipart1k|Artistic Illustration:Watercolor2k|Artistic Illustration:Comic2k
-|-|-|-|-|-|-|-|-
Detic (Baseline) |83.9 |29.4 |20.6 |20.6 |19.0 |61.0 |58.9 |51.2
PETS |85.9 |31.5 |20.6 |22.6 |18.2 |63.0 |60.2 |50.4
PETS$\star$ |86.3 |32.1 |21.1 |23.2 |19.3 |63.6 |61.3 |50.6
TPDS |85.5 |31.8 |20.2 |22.1 |18.8 |63.1 |60.0 |50.1
TPDS$\star$ |85.6 |32.0 |21.1 |23.2 |19.2 |64.3 |61.4 |50.6
KGD (Ours) |86.9 |34.4 |24.6 |24.3 |23.7 |69.1 |63.5 |55.6

**Response to Weaknesses-3:**
Thanks for your comment. As suggested, we conduct new experiments to compare our KGD with prior knowledge graph distillation methods[Ref 3, Ref 4] . The results in the tables below show that our KGD outperform [Ref 3, Ref 4] clearly, largely becuase the knowledge graphs in [Ref 3, Ref 4] are hand-crafted by domain experts while ours is built and learnt from CLIP. We will include the new experiments into the updated paper later. 


Method |Cityscapes |Vistas |BDD100K-weather-rain|BDD100K-weather-snow|BDD100K-weather-overcast|BDD100K-weather-cloudy|BDD100K-weather-foggy |BDD100K-time-of-day--daytime |BDD100K-time-of-day--dawn&dusk|BDD100K-time-of-day--night 
-|-|-|-|-|-|-|-|-|-|-
Detic (Baseline) |46.5 |35.0 |34.3 |33.5 |39.1 |42.0 |28.4 |39.2 |35.3 |28.5
KGE [Ref 5] |48.9 |36.0 |35.5 |34.4 |40.5 |41.2 |29.7 |40.1 |36.6 |29.0
Context Matters [Ref 6] |49.4 |36.6 |36.3 |35.0 |41.7 |42.4 |30.2 |41.5 |37.2 |29.7
KGD (Ours) |53.6 |40.3 |37.3 |37.1 |44.6 |48.2 |38.0 |46.6 |41.0 |31.2


Method |Common Objects: VOC|Common Objects: Objects365 |Intelligent Surveillance: MIO-TCD |Intelligent Surveillance:BAAI  |Intelligent Surveillance:VisDrone |Artistic Illustration:Clipart1k|Artistic Illustration:Watercolor2k|Artistic Illustration:Comic2k
-|-|-|-|-|-|-|-|-
Detic (Baseline) |83.9 |29.4 |20.6 |20.6 |19.0 |61.0 |58.9 |51.2
KGE [Ref 5] |85.4 |31.2 |20.3 |23.5 |19.4 |62.4 |58.1 |50.5
Context Matters [Ref 6] |85.9 |31.7 |20.9 |23.3 |19.9 |62.9 |59.1 |52.3
KGD (Ours) |86.9 |34.4 |24.6 |24.3 |23.7 |69.1 |63.5 |55.6

[Ref 3] Christopher Lang, Alexander Braun, and Abhinav Valada. Contrastive object detection using knowledge graph embeddings. arXiv preprint arXiv:2112.11366, 2021

[Ref 4] Aijia Yang, Sihao Lin, Chung-Hsing Yeh, Minglei Shu, Yi Yang, and Xiaojun Chang. Context matters: Distilling knowledge graph for enhanced object detection. IEEE Transactions on Multimedia, 2023, doi: 10.1109/TMM.2023.3266897.
**Response to Question-5**: 

Thanks for your detailed comments! We will check through our manuscript carefully and improve the table layout in the revised manuscript. As suggested, we conduct experiments and report the results of using ''LKG Extraction with category name'' and ''Static VKG Extraction'' jointly in the following table.
As a comparison, our proposed KGD shows clear improvements as the language and vision information extracted along the training process dynamically stabilizes and improves the model adaptation, validating the performance gain largely comes from our novel KGD designs instead of jointly using ''LKG Extraction with category name'' and ``Static VKG Extraction''. 

Method |Detic (Source only)||||KGD
-|-|-|-|-|-
LKG Extraction with category names ||$\checkmark$||$\checkmark$|
Static VKG Extraction |||$\checkmark$|$\checkmark$|
LKG Extraction with WordNet Hierarchy |||||$\checkmark$
Dynamic VKG Extraction |||||$\checkmark$
AP50 |46.5|51.9|51.9|52.4|**53.6**

As suggested, we conduct additional ablation study on 3 object detection datasets that span different downstream domains including the object detection for intelligent surveillance (BAAI), common objects (VOC), and artistic illustration (Clipart1k).
As the table below shows, the behavior consistent across datasets that span different downstream domains. 
Later, We will conduct ablation study on all the 11 object detection datasets and include the results into the revised manuscript. Thank you for your suggestion!

Method | Language Knowledge Graph Distillation | Vision Knowledge Graph Distillation |AP50|AP50|AP50|AP50
-|-|-|-|-|-|-
Dataset  | | |Cityscapes|BAAI|VOC|Clipart1k
Detic (Baseline)|||46.5|20.6|83.9|61.0
||$\checkmark $||52.8|22.2|86.1|66.5
|||$\checkmark$|52.7|22.4|86.2|67.1
KGD (Ours)|$\checkmark $|$\checkmark $|53.6|24.3| 86.9| 69.1