Open-Vocabulary Object Detection via Language Hierarchy

Authors: jiaxing  huang
jingyi  zhang
kai  jiang
shijian  lu
wordnet  hierarchy
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 18:54:55

Abstract: Recent studies on generalizable object detection have attracted increasing attention with additional weak supervision from large-scale datasets with image-level labels.
However, weakly-supervised detection learning often suffers from image-to-box label mismatch, i.e., image-level
labels do not convey precise object information.
We design Language Hierarchical Self-training (LHST) that introduces language hierarchy into weakly-supervised detector training for learning more generalizable detectors.
LHST expands the image-level labels with language hierarchy and enables co-regularization between the expanded labels and self-training. Specifically, the expanded labels regularize self-training by providing richer supervision and mitigating the image-to-box label mismatch, while self-training allows assessing and selecting the expanded labels according to the predicted reliability.
In addition, we design language hierarchical prompt generation that introduces language hierarchy into prompt generation which helps bridge the vocabulary gaps between training and testing.
Extensive experiments show that the proposed techniques achieve superior generalization performance consistently across 14 widely studied object detection datasets.

Main paper: 
 introduction: 
 Object detection aims to locate and identify objects in images by providing basic visual information of "where and what objects are". Thanks to the recent advances of deep neural networks, it has achieved great success with various applications in autonomous driving [1, 2, 3, 4] , intelligent surveillance [5, 6, 7, 8] , wildlife tracking [9, 10, 11] , etc. However, learning a generalizable object detector for various downstream tasks that have different data distributions and data vocabularies remains an open research challenge. To this end, weakly-supervised object detection (WSOD) [12, 13, 14, 15] , which allows access of large-scale image-level datasets (e.g., ImageNet-21K [16] with 14M images of 21K classes) with super rich data distributions and data vocabularies, has reignited new research interest under the context of learning generalizable detectors.While exploiting WSOD to learn generalizable detectors, one typical challenge is that the provided image-level labels do not convey precise object information [15] and often mismatch with box-level labels. Recent methods address this challenge by designing various label-to-box assignment strategies that assign the image-level labels to the predicted top-score [13, 14] or max-size [15] object proposals. However, the mismatch problem remains due to the restriction of the raw image-level labels [17] . At the other end, self-training [18, 19, 20] with the detectors pre-trained with [13, 14, 15] can generate box-level pseudo labels without the restriction of image-level labels. It allows learning from more object proposals without the image-to-box label mismatch issue, but it does not benefit much from the provided image-level label supervision.Figure 1 : Image-level labels in large-scale datasets such as ImageNet-21k [16] often do not convey precise object information [17, 15] which affects while learning generalizable detectors. Recent methods tackle this issue by various label-to-box assignment strategies [12, 13, 14, 15] as in (a) but are heavily restricted by raw image-level labels and still suffer from image-to-box label mismatch [17] . Self-training [18] with the detectors pre-trained with [13, 14, 15] could circumvent the label mismatch issue but the generated pseudo box labels are error-prone due to the lack of proper supervision as in (b). Our proposed LHST introduces language hierarchy to expand the image-level labels and enables co-regularization between the expanded labels and self-training which allows producing more accurate pseudo box labels in (c).We propose to incorporate image-level supervision with self-training for learning generalizable detectors, aiming to benefit from self-training while effectively making use of image-level weak supervision. We start from a simple observation: the image-to-box label mismatch largely comes from the ambiguity in language hierarchy, e.g., the image-level label Aquatic Mammal in Figure 1 can cover different object-level labels such as seals, dolphins, walruses, etc. With the above observations, we design a Detector with Language Hierarchy (DetLH) that combines language hierarchical selftraining (LHST) and language hierarchical prompt generation (LHPG) for learning generalizable detectors.LHST introduces WordNet's language hierarchy [21] to expand the image-level labels and accordingly enables co-regularization between the expanded labels and self-training. Specifically, the expanded labels are not all reliable though they can mitigate the image-to-box label mismatch problem by providing richer supervision. Here self-training can predict reliability scores for the expanded labels for better selection or weightage of the expanded labels. At the other end, self-training with pseudo box labels allows learning from more proposals and can circumvent the image-to-box label mismatch , but the box-level pseudo labels are usually noisy and may lead to learning degradation [15] . Here the expanded labels provide richer and more flexible supervision which can effectively help suppress prediction noises in self-training.LHPG helps bridge the vocabulary gaps between training and testing by introducing WordNet's language hierarchy into prompt generation process. Specifically, LHPG leverages the CLIP language encoder [22] to measure the embedding distances between test concepts and WordNet synsets, and then generates the prompt for a given test concept from its best matched WordNet synset. In this way, the test prompts generated by LHPG have been standardized by WordNet and are well aligned with our proposed detector that is trained with WordNet information via LHST. In another word, the combination of LHST and LHPG actually leverages WordNet as a standard and intermediate vocabulary that bridges the gaps between training and testing vocabularies, generating better prompts and leading to better detection performance on downstream applications.The main contributions of this work are threefold. First, we propose language hierarchical selftraining that incorporates language hierarchy with self-training for weakly-supervised object detection. Second, we design language hierarchical prompt generation, which introduces language hierarchy into prompt generation to bridge the vocabulary gaps between detector training and testing. Third, extensive experiments show that our DetLH achieves superior generalization performance consistently across 14 detection benchmarks. 
related work: 
 Weakly-supervised object detection (WSOD) aims to train object detectors using image-level supervision. Traditional WSOD methods [23, 24, 25, 26, 27] use image-level annotations only without any box annotations and thus focus on low-level proposal mining techniques [28, 29, 12, 30, 31, 32] , leading to unsatisfying localization performance. Semi-supervised WSOD [33, 34, 35, 36, 37, 38, 39] has been proposed to further improve the performance, which leverages both box-level and imagelevel annotated data. With better localization quality, recent methods [13, 14, 15, 40] design various label-to-box assignment strategies, such as assigning image-level labels to max-score anchors [13] , max-score proposals [14] or max-size proposals [15] . Our work belongs to semi-supervised WSOD. Different from previous methods, we tackle the image-to-box label mismatch by introducing language hierarchy into self-training.Large-vocabulary object detection [41, 13, 42, 43, 44] researches on detecting thousands of categories. Most previous papers focus on tackling the long-tail issue [45, 46, 47, 48, 49, 50] , e.g., by using equalization losses [51, 52] , SeeSaw loss [53] , or Federated Loss [54] . Recent semi-supervised WSOD methods [13, 14, 15] and our work circumvent the long-tail problem by leveraging more balanced image-level datasets such as ImageNet-21K.Open-vocabulary object detection focuses on detecting objects conditioned on arbitrary words (i.e., any category names). A common strategy [55, 56, 57, 58, 59 ] is to replace the detector's classification layer with the language embeddings of category names. Recent methods [60, 61, 62, 63, 17, 15] leverage the powerful CLIP [22] model by using its text embeddings [60, 61, 62, 63, 17, 15] or conducting knowledge distillation [60, 63, 17] . Similar to Detic [15] , our work uses CLIP text embeddings as the classifier and leverages image-level annotated data instead of distilling knowledge from CLIP.Language hierarchy has been widely studied for visual recognition tasks [64] , especially for largevocabulary visual recognition. Most existing studies [65, 66, 67] focus on image classification tasks, e.g., leveraging language hierarchy for multi-label image classification [65, 66, 67, 68, 69, 70, 71] , modelling hierarchical relations among classes [68, 69] or facilitating classification training [70, 71] . Different from previous work, we introduce language hierarchy into self-training for weaklysupervised object detection. 
method: 
 This work focuses on learning generalizable object detectors via weakly-supervised detector training [15] , which leverages additional large-scale image-level datasets to enlarge the data distributions and data vocabularies in detector training. We first describe the task definition with training and evaluation setups. Then, we present our proposed DetLH which is detailed in two major aspects on Language Hierarchical Self-training (LHST) that introduces language hierarchy into detector training, and Language Hierarchical Prompt Generation (LHPG) that introduces language hierarchy into prompt generation.Training setup. The training data consists of two parts: 1) a detection datasetD det = {(x, y det ) i } |D det | i=1, where x denotes an image while y det stands for the class and bounding box labels for x; 2) an image classification datasetD cls = {(x, y cls ) i } |D cls | i=1where y cls denotes the imagelevel label (i.e., a one-hot vector) for x. Given the two datasets, the goal is to learn a generalizable detection model F by jointly optimizing F over D det and D cls :EQUATIONwhereL det (•) = L rpn (•) + L reg (•) + L cls (•)is the fully-supervised detection loss function while L rpn (•), L reg (•), and L cls (•) denote RPN, Regression, and Classification loss functions, respectively. L weak is the weakly-supervised loss function to train detectors with image-level labels.Evaluation setup. As the goal is to learn a generalizable detection model that works well on various unseen downstream tasks, we conduct zero-shot cross-dataset evaluation2 to assess the generalization performance of the trained detection model. Note, different domain adaptation [72, 73, 74, 75] that generally uses downstream data in training, our setup is similar to domain generalization [76, 77] that does not involve downstream data in training. Open-vocabulary Detector. We modify the classification layer of the detector into an openvocabulary format such that the detector could be tested over unseen datasets. Specifically, we replace the weights of the detector's classification layer with the fixed language embeddings encoded from class names, where the object classification could be achieved by matching the object's embedding and the fixed language embeddings. We adopt the CLIP language embeddings [22] as the classification weights as in [15, 60] . In this way, the modified detector could theoretically detect any target concepts on any target data. As reported in [15] , detectors trained solely on detection datasets often exhibits constrained performance due to the small-scale training images and vocabularies. Similar to [15] , our proposed DetLH introduces large-scale image-level datasets to enlarge the data distributions and data vocabularies in detector training, leading to more generalizable detectors and better generalization performance on various unseen datasets.The proposed LHST utilizes WordNet's language hierarchy to expand the image-level labels, which enables co-regularization between the expanded image-level labels and self-training as illustrated in Figure 2 .Overview. For fully supervised detector training over the detection dataset, we feed box-level annotated samples (x, y det ) ∈ D det to the detection model F and optimize F with the standard fully supervised detection loss, i.e., the first term of Eq. 1. For weakly-supervised detector training over the image-level annotated dataset (x, y cls ) ∈ D cls shown in Figure 2 , we first leverage WordNet's language hierarchy to expand the raw image-level label y cls into y hier image (the hierarchical image-level label), and merge y hier image and the generated pseudo box label ŷbox to acquire ŷhier box (the hierarchical box-level pseudo label). Then, we optimize the detector with (ŷ hier box , w hier box ) and (y hier image , w hier image ), where w hier image and w hier box denote the predicted reliability scores of the expanded logits '1' in y hier image and y hier box and are used to weight the labels in loss calculation. Expanding image labels with language hierarchy. Given image-level annotated dataset (x, y cls ) ∈ D cls (y cls is a label vector with length C and C denotes the number of classes), we leverage WordNet's class name hierarchy [21] to expand y cls into y hier image as the following:EQUATIONwhere the function WordNet(•) recursively finds all hypernyms and hyponyms of the input (i.e., the class indicated in y cls ) and sets their positions in the label vector y cls to be '1' to expand y cls into y hier image . In this way, a single-label annotation could be expanded into a multi-label annotation within the very rich ImageNet-21K vocabulary.Generating pseudo box labels with predictions. Given the image x ∈ D cls , we feed x into the detector F to acquire the prediction as following:EQUATIONwhere p n is the probability vector of the predicted n-th bounding box after Softmax, and p c n denotes the predicted c-th category probability. Note we filter out a prediction if its max confidence score is lower than the threshold t, and N denotes the number of predicted object proposals after filtering, i.e., max({p c n } 1≤c≤C ) ≥ t, ∀n. Then the pseudo category label ŷbox = {ŷ n } 1≤n≤N for N boxes in image x is derived by:EQUATIONwhere ŷn = (ŷ(1) n , ŷ(2) n , ..., ŷ(C) n )is the predicted category label, and ∆ C denotes a probability simplex with length C.Merging image and pseudo box labels. As the predicted pseudo box label ŷbox is error-prone, we regularize it with the expanded image-level supervision by merging y hier image and ŷbox as the following:EQUATIONwhere ∨ denotes the logical "OR" operator.Assessing the expanded labels. As the labels expanded by WordNet (i.e., the expanded logits '1' inŷhier box = {ŷ c n } 1≤n≤N,1≤c≤C) are not all reliable, we predict a reliability score w c n for the expanded label to adaptively re-weight y c n ∈ ŷhier box when applying it on different pseudo boxes. We measure the reliability of y c n with prediction p c n , and w hier box = {w c n } 1≤n≤N,1≤c≤C can be derived by:EQUATIONwhere y hier image(c) ̸ = y (c)cls returns True if the c-th label logit in y hier image is expanded by WordNet, which also applies to ŷhier box as ŷhier box is expanded by mergeing it with y hier image . Given the prediction {p c n } 1≤n≤N,1≤c≤C , the merged pseudo box label ŷhier box = {ŷ c n } 1≤n≤N,1≤c≤C and its reliability score w hier box = {w c n } 1≤n≤N,1≤c≤C , we optimize the detector F as the following:EQUATIONwhere BCE(•) denotes the binary cross-entropy loss.In addition, training with the predicted pseudo box labels is not very stable as pseudo box labels vary along training process. Thus, we regularize the training of L box (F (x)) with an image-level loss defined as the following:EQUATIONwhere p image = {p c image } 1≤c≤C denotes the category probability predicted for the image-level proposal. w hier image = {w c image } 1≤c≤C denotes the reliability score for the expanded logits "1" in Training objective. The overall training objective of Language Hierarchical Self-training is defined as:EQUATIONLanguage Hierarchical Prompt Generation. As the goal is to learn a generalizable detection model that works well on various downstream tasks, one typical challenge is the vocabulary gap between detector training datasets (i.e., LVIS and ImageNet-21K) and detector testing datasets (e.g., object365 or customized data). A common solution of tackling the vocabulary gaps is to conduct prompt learning [78] to generate proper category prompts. However, prompt learning generally requires labeled target images for additional training.In this work, we tackle the vocabulary gaps by generating prompts with the help of WordNet, which introduces little computation overhead and does not require labeled target images and additional training. To this end, we design language hierarchical prompt generation (LHPG) that works by incorporating WordNet information into prompt generation process. Specifically, LHPG leverages CLIP language encoder [22] to measure the embedding distances between test concepts and WordNet synsets, and then generates the prompt for a given test concept from its best matched WordNet synset: has been standardized by WordNet and is well aligned with our proposed detector that is trained with WordNet information via LHST. In another word, the combination of LHST and LHPG makes use of WordNet as a standard and intermediate vocabulary that bridges the gaps between training and testing vocabularies, generating better prompts and leading to better detection performance on downstream applications.V WordNet test = CLIP(V test , WordNet), 
experiments: 
 We evaluate our DetLH on 14 widely adopted detection benchmarks. We follow the zero-shot crossdataset object detection setting proposed in [17, 15] . More details like Dataset and Implementation Details are provided in the appendix.Table 1 : Zero-shot cross-dataset object detection for common objects. All detectors are trained over the training datasets (LVIS and ImageNet-21K) and evaluated over target datasets (i.e., Object365 and Pascal VOC with objects from common classes and scenarios) without finetuning. "Datasetspecific oracles" denote the detectors that are fully supervised which are trained by using the training data of respective datasets.Object365 [79] Pascal VOC [80] AP We conduct extensive experiments to benchmark our proposed DetLH with state-of-the-art methods. We evaluate them on 14 widely studied object detection datasets to assess their zero-shot cross-dataset generalization ability. Tables 1-5 report zero-shot cross-dataset detection results for common objects, autonomous driving, intelligent surveillance, and wildlife detection, respectively. More details are to be described in the following paragraphs. Object detection for common objects. Table 1 shows that DetLH outperforms state-of-the-art methods clearly on common object datasets Object365 and Pascal VOC. In addition, we can observe that DetLH even brings significant gains above the dataset-specific oracle (i.e., the model that is fully trained on the target training data) on Pascal VOC (i.e., a small-scale dataset), showing the advantages of leveraging large-scale training data.Object detection for autonomous driving. As shown in Table 2 , our DetLH outperforms state-ofthe-art methods by large margins on various autonomous driving datasets, showing that DetLH still works effectively while facing large variations in camera views from autonomous driving scenarios to the base-dataset scenarios (LVIS and ImageNet-21K), e.g., autonomous driving images are captured under very different camera views. In addition, the experimental results in Table 3 show that our DetLH brings significant performance gains against state-of-the-art methods when encountering various weather and time-of-day conditions, which demonstrates the effectiveness of DetLH while detecting objects under large noises [83] , e.g., the images captured under different weather and time-of-day conditions may have very different styles and image quality.Object detection for intelligent surveillance. From Table 4 , we can observe that our DetLH outperforms state-of-the-art methods by clear margins on various intelligent surveillance datasets, indicating that DetLH is also tolerant to large changes in the camera lens and angles which often happen to intelligent-surveillance images that are captured under very different camera lens and angles (e.g., surveillance cameras are often with the wide-angle lens and used in high angle views).Object detection for Wildlife. The experimental results in Table 5 show that our DetLH performs well on various wildlife detection datasets, showing that DetLH works effectively for detecting fine-grained categories that exist widely in wildlife detection datasets. The significant performance gains largely come from the introduction of language hierarchy into detector training and prompt generation, which helps model the hierarchical relations among parent and fine-grained subcategories and thus leads to better fine-grained object detection.The superior detection performance of our DetLH is largely attributed to our two core designs, i.e., LHST and LHPG. LHST enables effective usage of large-scale image-level annotated images and significantly enlarges the data distribution and the data vocabulary in detector training, yielding robust performance under large cross-dataset gaps in data distribution and vocabulary. LHPG ingeniously helps mitigate the vocabulary gaps between detector training and testing. It improves the overall confidence of detection and benefits the detection as large data distribution gaps (or large data vocabulary gaps) often lead to low-confidence predictions and poor detection results. We perform ablation studies with Swin-B [84] based CenterNet2 [54] over the large-scale Ob-ject365 dataset as shown in Table 6 . As the core of our proposed DetLH, we examine how our designed LHST and LHPG contribute to the overall performance of zero-shot cross-dataset object detection. As shown in Table 6 , the baseline (Box-Supervised [15] ) does not perform well as it uses box-level training data only. It can be observed that LHST outperforms the baseline clearly, showing that LHST can effectively leverage the largescale image-level annotated dataset to significantly enlarge the data distribution and data vocabulary involved in detector training, leading to much better zero-shot cross-dataset detection performance. In addition, LHPG brings clear performance improvements in zero-shot cross-dataset detection by introducing language hierarchy into prompt generation, demonstrating the effectiveness of LHPG in mitigating the vocabulary gaps between training and testing. Moreover, the inclusion of both LHST and LHPG in the proposed DetLH performs clearly the best, indicating the complementary property of our two designs. Generalization across various detection tasks: We study the generalization of our DetLH by conducting zero-shot cross-dataset object detection on 14 widely studied object detection datasets. Tables 1 2 3 4 5 show that DetLH achieves superior performance consistently across all the detection applications. Besides, Table 7 summarizes the detection results averaged on 14 datasets, showing that DetLH clearly outperforms the state-of-the-art methods.We study the generalization of the proposed DetLH from the perspective of network architectures. Specifically, we perform extensive evaluations with four representative network architectures, including one Transformer-based (i.e., Swin-B) and three CNN-based (i.e., ConvNeXt-T, ResNet-50 and ResNet-18). Experimental results in Table 8 show that the proposed DetLH outperforms the state-of-the-art method consistently over different network architectures. Parameter Studies for Language Hierarchical Self-training (LHST). In generating pseudo box labels in LHST, we filter out a prediction if its max confidence score is lower than the threshold t.We study the threshold t by changing it from 0.65 to 0.85 with a step of 0.05. Table 12 reports the experimental results on zero-shot transfer object detection over object365 dataset. We can observe that the detection performance is not sensitive to the threshold t. Due to the space limit, we provide more DetLH discussions and visualizations in the appendix.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We provided detailed instructions for reproducing the main experimental results in Section 3 Method and Section 4 Experiment including the details of the proposed framework, and the datasets, base models and the parameters used for experiments in Section Dataset and Implementation Details in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
conclusion: 
 This paper presents DetLH, a Detector with Language Hierarchy that combines language hierarchical self-training (LHST) and language hierarchical prompt generation (LHPG) for learning generalizable object detectors. LHST introduces WordNet's language hierarchy to expand the image-level labels and accordingly enables co-regularization between the expanded labels and self-training. LHPG helps mitigate the vocabulary gaps between training and testing by introducing WordNet's language hierarchy into prompt generation. Extensive experiments over multiple object detection tasks show that our DetLH achieves superior performance as compared with state-of-the-art methods. In addition, we demonstrate that DetLH works well with different network architectures such as Swin-B, ConvNeXt-T, ResNet-50, etc. Moving forward, we will explore language hierarchy to further expand the labels in an open-vocabulary manner in addition to the closed ImageNet-21K's vocabulary.DAWN [82] ) is a vehicle detection dataset that focuses on diverse traffic environment. This dataset contains 1K images from real-traffic environment, including fog, snow, rain and sandstorms. The images are annotated with object bounding boxes of 6 categories. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We provided the detailed implementation details in Section Dataset and Implementation Details in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We provided sufficient information on the computation resources required for reproduce the experiments in Section Dataset and Implementation Details in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We properly credited the original owners of assets used in the paper and properly respect their license and terms of use.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Strategy studies for Language Hierarchical Self-training. Our proposed language hierarchical self-training (LHST) introduces WordNet's language hierarchy [21] to expand the image-level labels and accordingly enables co-regularization between the expanded labels and self-training. We examine the superiority of the proposed LHST by comparing it with "Self-training" [18] and "Direct WordNet Hierarchy Labeling" [21] . "Self-training" is the standard self-training algorithm as in [18] while "Direct WordNet Hierarchy Labeling" denotes directly using the expanded image-level labels (by WordNet) for weakly-supervised detection training. Table 10 reports the experimental results, which show that either "Self-training" [18] or "Direct WordNet Hierarchy Labeling" [21] does not perform well. The reasons are: 1) the box-level pseudo labels in "Self-training" are usually error-prone, making the self-training process unstable and barely improving the performance; 2) the expanded image-level labels in 'Direct WordNet Hierarchy Labeling" are not all reliable, training with which leads to unsatisfying performance. Besides, the combination of 'Self-training" and "Direct WordNet Hierarchy Labeling" still works not very well largely because the direct combination of them does not well address their own drawbacks and limitations. On the other hand, our proposed LHST performs better clearly, as shown in the last row of Table 10 .The superior performance of LHST is largely attributed the co-regularization design, which employs self-training to assess and re-weight the expanded labels according to the predicted reliability while enabling the expanded (and re-weighted) labels to regularize self-training by providing richer and flexible supervision (the flexible supervision is achieved by the adaptive re-weighting operation). Analysis of discrepancies of the taxonomies of image vs. box categories. We analyze the mismatch between image-level and box-level categories and how much it could affect the detection performance. As Table 13 shows, the mismatch between image-level and box-level categories varies across datasets, and the proposed DetLH improves more with the increase of mismatch levels. The impact of using proxy vocabulary. We conduct experiments on Object365 dataset to compare LHPG with CLIP embeddings only and LHPG with both CLIP embeddings and proxy vocabulary. As Table 15 shows, using a proxy vocabulary performs clearly better, demonstrating its effectiveness in narrowing the distribution gap between training and test labels. Comparison with RKD [17] . We note that RKD [17] explores Region-based Knowledge Distillation to better distill knowledge from the CLIP model for weakly-supervised object detection. In this paragraph, we compare our DetLH (i.e., the self-training based method) with RKD (i.e., the knowledge distillation-based method) on zero-shot cross-dataset object detection over Object365 dataset. Table 17 reports the results, which show that our DetLH clearly outperforms RKD [17] , indicating the effectiveness the proposed designs in DetLH for zero-shot cross-dataset object detection. Discussion and comparison with visual grounding-based detection methods [92, 93] . We note that GLIP [92] and DetCLIP [93] explore extra visual grounding data to train a open-vocabulary detector. In this paragraph, we compare our DetLH (i.e., the WSOD-based method) with [92, 93] (i.e., the visual grounding-based method) from the perspective of detection efficiency. Table 18 reports the results of run time in millisecond (the run times of GLIP [92] and DetCLIP [93] are acquired from [93] .). It shows that our DetLH (i.e., the WSOD-based method) runs much more efficient than the visual grounding-based detection methods (i.e., GLIP [92] and DetCLIP [93] ), largely because the visual grounding-based detection methods [92, 93] include a text encoder in their networks. Note we did not compare our DetLH (i.e., the WSOD-based method) with the visual groundingbased method [92, 93] from the perspective of detection accuracy becuase these two types of detection methods use very different training data, e.g., [92] and [93] use visual grounding data. In addition, the WSOD techniques are basically complementary to the visual grounding-based detection techniques [92, 93] because the visual grounding data could be used to further improve the WSOD-based detectors [94] . Similar to [94] , we leave this as our future research.Other image-level supervision. We follow Detic [15] to build our proposed DetLH. Therefore, similar to Detic [15] , our DetLH can also seamlessly incorporate the free-form caption text as the image-level supervision, i.e., by using the language embeddings of image captions as the detection classifier when training on image-text pair data [15] . In this way, we could further incorporate LAION dataset [95] that includes 400 million image-text pair data into detector training for learning more generalizable object detectors. On the other hand, training over large-scale LAION dataset is computation-intensive and thus we leave this as our future work.Comparison with other detection methods [92, 96, 97, 93, 98, 99] . We didn't compare with these methods [92, 96, 97, 93, 98, 99] in the main manuscript because they focus on different topics with different objectives, training data, backbones and benchmarks. Instead, we follow Detic [15] as both our DetLH and Detic belong to and are claimed as weakly-supervised object detection (WSOD), aiming to using large-scale images and classes (i.e., ImageNet-21K and LVIS) to train a general detector that can work on any detection scenarios. However, [92, 96, 97, 93, 98, 99] are not for WSOD: GLIP and DetCLIP [92, 93] introduce visual grounding and studies how to use grounding data for detection; OWL-ViT [97] focuses on fine-tuning CLIP with standard detection datasets; OmDet and UniDet [98, 99] focus on training with multiple detection datasets. We still managed to benchmark with [92, 96, 97, 93, 98, 99] , i.e., GLIP [92] , GLIPv2 [96] , OWL-ViT [97] , DetCLIP [93] , OmDet [98] and UniDet [99] . As our method and [92, 96, 97, 93, 98, 99] use various different datasets in evaluations, the benchmarking below is on the shared one, i.e., Pascal VOC (in AP). Note Florence [100] is not included as it is a foundation model that uses a very large backbone (CoSwin-H) and very large customized training data (FLD-900M and FLOD-9M). Besides, we compared our DetLH with GLIP [92] and DetCLIP [93] as in Table 18 , which shows DetLH (i.e., the WSOD-based method) runs much more efficient (about 10 times) than the visual grounding-based detection methods (i.e., GLIP and DetCLIP).Comparison on ODinW [92] . We note that ODinW [92] also benchmarks cross-dataset generalization. We benchmark on ODinW and the results below (averaged on 35 datasets in ODinW) show that our DetLH works effectively on ODinW. Note GLIP obtains higher accuracy because it introduces visual grounding and involves Language Encoder in inference. Without those extra modules, our DetLH runs much faster (over 10 times) than GLIP as discussed in Table 18 .WSDDN YOLO9000 DLWL Detic GLIP OmDet Ours 14.9 Open-vocabulary benchmark. We did not benchmark on open-vocabulary LVIS/COCO (both divide a singledataset vocabulary into base and novel classes to mimic and benchmark vocabulary generalization), because our work aims to leverage large-scale images and classes (i.e., 21K classes) to train a general detector that can work on any detection scenarios. Cross-dataset generalization benchmark fits this objective better and is more general and challenging than open-vocabulary benchmark that tackles base and novel classes within a single dataset.Comparison with class hierarchy methods [101, 70] on OpenImages [102] . We benchmark with other methods that use class hierarchy, including [101] and [70] . As shown below, our DetLH performs clearly better than hierarchy-aware losses in [101, 70] Results of WSDDN, YOLO9000 and DLWL. Note Detic implemented WSDDN, YOLO9000 and DLWL and we directly adopted Detic's implementation in evaluations. The gains in our reported results are different as we evaluated on more challenging datasets and benchmark: 1) The results in our Table 7 in the main manuscript are averaged over 14 datasets while those in Table 1 of Detic are on a single dataset LVIS; 2) Our Table 7 in the main manuscript is cross-dataset generalization benchmark while Table 1 in Detic is open-vocabulary LVIS.We provide qualitative results of zero-shot cross-dataset object detection for various detection tasks. As shown in Figures 3 4 5 6 7 . our DetLH produces good detection results consistently across different detection tasks, showing DetLH still works effectively under large cross-dataset gaps in data distribution and vocabulary. • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.zero-shot cross-dataset evaluation here means that the model is evaluated on unseen datasets, which is the same as the one defined in CLIP[22]. 


Decision: This submission proposes to expand the training data for object detection using image-level weak supervision from classification datasets, and leverages a word ontology (wordnet).

Reviewers agreed that:
1. The paper was generally well-written and easy to understand
2. The proposed method yields meaningful improvements over SOTA.

There were initial concerns about:
1. The thoroughness of ablations and experimental analysis (including suggestions to move some experiments to the main section)
2. The proposed method being somewhat trivial.

Both concerns were addressed during the discussion period and all 3 reviews ultimately recommend weak accept or accept. The AC sees no reason to override this unanimous recommendation. The authors are asked to incorporate the requests from reviewers into the final version of the paper.

Reviews: review: paper_summary: paper_summary: * The authors propose to expand the training data for object detection using classification datasets through two main contributions:

1. Combine a language hierarchy with self-training to extend the training dataset, while minimizing label noise by re-weighting categories with reliability scores. 
2. Bridge the vocabulary gaps between training and testing using a vocabulary proxy (WordNet) improving zero-shot learning.

* Extensive experiments in multiple datasets are provided showing an improvement over the state-of-the-art.
strength_weakness: strength_weakness: 1. The experimental section and ablation do not clearly show if the authors' assumptions are correct.
* The idea of the paper heavily relies on the assumption that there are discrepancies on the taxonomy levels of the categories of image and detection datasets. This assumption may hold for some datasets but not for others and it should have a significant impact on the method's results. I miss a section presenting an analysis of the taxonomies of image vs box categories, and how it may impact peformance.

2. Discussion on the specific impact of each contribution to the final result is lacking
* The ablation study (Section 4.2) does not seem to add much value to the experimental discussion. First, showing the performance increase over the baseline [68] seems misleading when the authors compare DetLH against only-box supervision, since the work of [68] already proposes to expand detector (boxed) training data with image-classification data. The work of [68] already demonstrated that adding image-based training data improves detection results, and it should not be the goal of the ablation section to show this. A far more interesting baseline could have been to train DetLH with uniform category weights to demonstrate how DetLH deals with noisy labels. 
* Another important ablation experiment would be to analize the impact of narrowing the gap between training and testing label distributions. The CLIP embeddings would already help in this regard, but how impactful is the use of a proxy vocabulary on top of it is not clearly quantified. 

3. Wording is sometimes ambiguous or could lead to contribution misunderstandings:
* For instance the authors say "These detectors usually yield constrained detection peformance when trained only with detection datasets... [68]", when the work of [68] is actually trained with both detection and classification data. If the authors want to point out that [68] already suggested this, it should be more explicitly stated. Even more so, if it's followed with a claim that the current method overcomes such limitations: "DetLH introduces large-scale image-level datasets to enlarge the data [...] in detector training.", which is misleading as this is not something the present work introduces, but the work of [68].
questions: questions: * Futher analysis and validation of main assumption. How do the category taxonomy levels differ across datasets?
* Where does the performance gain come from? Robustness to label noise or vocabulary proxy?
* In general in the experimental section I miss more insight and discussion on results, experimental setup and data setup. I feel that most discussion repeatedly states the obvious (performance increases due to leveraging larger amouts of data). Whenever more specific insight is provided, it is very speculative. For example, authors state that performance gains in Wildlife are due to the "introduction of the language hierarchy", but there are no specific experiments designed to proof such claim.
limitations: limitations: Yes, the limitations and potential negative impact are addressed.
paper_summary: paper_summary: This paper focuses on scaling the detectors’ vocabulary with image-level weak supervision. To better leverage the image-level labels in this task, this paper introduces a language hierarchical self-training (LHST) framework that incorporates language hierarchy (i.e., WordNet) with self-training. In addition, the authors introduce a language hierarchical prompt generation (LHPG) method enabling open-vocabulary object detection. The proposed method outperforms baselines across 14 commonly used benchmarks. Comprehensive ablation experiments validate the effectiveness and generalization of the proposed method.
strength_weakness: strength_weakness: - The necessity of the investigated problem (i.e., scaling the vocabulary of the detectors with image labels) should be further stated, as the recent MLLM-based detection/grounding methods inheritly have strong zero-shot capabilities.
- The proposed method is somewhat trivial and heuristic, where only the usage of WordNet is new and interesting. Most techniques are from existing WSOD and semi-supervised object detection methods. It would be better to highlight the novelty. 
- Several semi-supervised WSOD methods trained with both instance- and image-level annotations are missing. It would be better to review them for completeness. 
[a] UniT: Unified Knowledge Transfer for Any-Shot Object Detection and Segmentation, CVPR’21
[b] Cyclic Self-Training with Proposal Weight Modulation for Cross-Supervised Object Detection, TIP’23
[c] H2FA R-CNN: Holistic and Hierarchical Feature Alignment for Cross-Domain Weakly Supervised Object Detection, CVPR’22
[d] DOCK: Detecting Objects by transferring Common-sense Knowledge, ECCV’18
- In Line 149, the authors claimed ‘a single-label annotation could be expanded into a multi-label annotation’. It would be better to give some visualization examples. In Line 164, authors claimed ‘the labels expanded by WordNet are not all reliable’. Giving some examples would be helpful to improve the readability.
- Why only the AP results of oracles are reported on Object 365?
- How about the efficiency of WordNet in training and inference?
questions: questions: Please refer weakness.
limitations: limitations: The authors have discussed the limitations, and this works does not show potential negative societal impact.
paper_summary: paper_summary: The paper proposes a novel method for leveraging image-level annotations for object detection pretraining, specifically for zero-shot and open vocabulary detection settings. The proposed method, DetLH, leverages self-training to generate pseudo-labelled object proposals, and then adjust the pseudo-label for each box according to the image's class and the classes in WordNet that are hierarchically related to it. Furthermore, the authors propose a method for prompt generation that does not require additional training. The proposed method is evaluated extensively and achieves state-of-the-art results in relevant zero-shot and open-vocabulary object detection benchmarks.
strength_weakness: strength_weakness: 1. Unless I am mistaken, there is no ablation study related to the impact of the reliability score. I would be very interested to see how it affects performance.
2. While justifications are provided in the appendix, I believe more comparisons should have been included with relative methods (L661-671). More importantly, I believe that the ommision of evaluations with LVIS/COCO is a significant issue, and I am not convinced by the authors stated reason (L683-687) that they focused on better suited datasets. The paper has extensive experiments with many datasets, and it is confusing to me why they chose to leave out what is arguably the most common dataset for detection tasks.
questions: questions: My primary concern regarding the paper is related to the evaluations: a) Whereas the paper is a clear improvement over Detic, it avoids extensive comparisons with more recent methods (Detic being a ECCV 2022 paper) such as the ones included in the paper (L661-671), and b) evaluations with LVIS, a shared benchmark for most relative papers, are not conducted.

Overall, however, I am very satisfied with the paper's contribution and I believe that the conducted experiments adequately demonstrate the proposed method's effectiveness. My main reason for not giving a higher rating is because LVIS/COCO results are not included, which would be informative and are significant to facilitate comparisons with existing and future relevant works.
limitations: limitations: The authors have adequately addressed the relevant limitations.

Comments: title: Reply to authors
Reply to authors
comment: Thanks for the efforts in rebuttal. Your response has addressed my concerns.
After reading other reviewers' comments and the response, I would like to raise my rating.
Dear Reviewer PjbE,

Thank you for your insightful feedback. We have carefully considered your questions and suggestions and have addressed them accordingly. We sincerely appreciate your constructive comments, which have helped strengthen our paper. As the discussion phase is nearing its conclusion, we would appreciate it if you could let us know if there are any additional questions or suggestions.

Best regards,

Authors
Dear Reviewer VMio,

Thank you for your encouraging feedback and positive evaluation for our work. We will include the new ablation studies and the more comparisons in the main paper in the revised version. We sincerely appreciate your constructive comments, which have strengthened our paper.

Best regards,

Authors
Dear Reviewer TMzm,

Thank you for your encouraging feedback and positive evaluation for our work. We will include the updated ablation study, as well as further discussions regarding experiments (and taxonomy analysis) in the main paper in the revised version, and we agree that these are important points that should not be left in the supplementary material. We sincerely appreciate your constructive comments, which have strengthened our paper.

Best regards,

Authors
Thanks to the authors for the thorough clarification of my cocerns. I'm happy to boost my score to "Weak Accept" as long as the updated ablation study, as well as further discussions regarding experiments (and taxonomy analysis) are included in the main paper. In my opinion those are important points that should not be left in the supplementary material.
I thank the authors for addressing the issues raised by my review.

I would like to also see what other reviewers think about the responses to the issues they raised, but I am satisfied by the authors’ response to my review and am inclined to recommend acceptance (i.e. raise my score to 7 after the discussion).