Open-Vocabulary Object Detection via Language Hierarchy
Authors: Jiaxing Huang
Jingyi Zhang
Kai Jiang
Shijian Lu
Venue: NeurIPS 2024 poster (2024)
Keywords: Weakly-supervised object detection
Open-Vocabulary Object Detection
detection foundation model
Primary area: machine_vision

Abstract: Recent studies on generalizable object detection have attracted increasing attention with additional weak supervision from large-scale datasets with image-level labels.
However, weakly-supervised detection learning often suffers from image-to-box label mismatch, i.e., image-level
labels do not convey precise object information.
We design Language Hierarchical Self-training (LHST) that introduces language hierarchy into weakly-supervised detector training for learning more generalizable detectors.
LHST expands the image-level labels with language hierarchy and enables co-regularization between the expanded labels and self-training. Specifically, the expanded labels regularize self-training by providing richer supervision and mitigating the image-to-box label mismatch, while self-training allows assessing and selecting the expanded labels according to the predicted reliability.
In addition, we design language hierarchical prompt generation that introduces language hierarchy into prompt generation which helps bridge the vocabulary gaps between training and testing.
Extensive experiments show that the proposed techniques achieve superior generalization performance consistently across 14 widely studied object detection datasets.

Open-Vocabulary Object Detection via Language Hierarchy
Jiaxing Huang, Jingyi Zhang, Kai Jiang, Shijian Lu' College of Computing and Data Science Nanyang Technological University, Singapore
Abstract
Recent studies on generalizable object detection have attracted increasing attention with additional weak supervision from large-scale datasets with image-level labels. However, weakly-supervised detection learning often suffers from image-to box label mismatch, i.e., image-leve labels do not convey precise object information. We design Language Hierarchical Self-training (LHST) that introduces language hierarchy into weakly-supervised detector training for learning more generalizable detectors. L.HST expands the image-level labels with language hierarchy and en- ables co-regularization between the expanded labels and self-training. Specifically, the expanded labels regularize self-training by providing richer supervision and mitigating the image-to-box label mismatch, while self-training allows assessing and selecting the expanded labels according to the predicted reliability. In addi- tion, we design language hierarchical prompt generation that introduces language hierarchy into prompt generation which helps bridge the vocabulary gaps between training and testing. Extensive experiments show that the proposed techniques achieve superior generalization performance consistently across 14 widely studied object detection datasets.
Introduction
Object detection aims to locate and identify objects in images by providing basic visual information of "where and what objects are". Thanks the recent advances of deep neural networks, it has achieved great success with various applications in autonomous driving [1] 2 3 4. intelligent surveillance [5] 6. 7 H) wildlife tracking [9 I0 [1], etc. However, learning generalizable object detector for various downstream tasks that have different data distributions and data vocabularies remains an open research challenge. To this end, weakly-supervised object detection (WSOD) [12 13 14 15), which allows access of large-scale image-level datasets (e.g.. ImageNet-21 K (16) with 14M images of 21K classes) with super rich data distributions and data vocabularies, has reignited new research interest under the context of learning generalizable detectors.
While exploiting WSOD learn generalizable detectors, typical challenge is that the provided image-level labels do not convey precise object information [15 and often mismatch with box-level labels. Recent methods address this challenge by designing various label-to-box assignment strategies that assign the image-level labels to the predicted top-score [L3] 14 or max-size [13] object proposals.
However, the mismatch problem remains due to the restriction of the raw image- labels [17]. At
the other end, self-training [8 19 20 with the detectors pre-trained with [3] 14 [5] can generate box-level pseudo labels without the restriction of image-level labels. It allows learning from more object proposals without the mageetto-box label mismatch issue, but it does not benefit much from the provided image-level label supervision.
*Corresoonding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

WordNet Fieeardhy
Pseudo Labsls from
Pinuiped Mammal: Seal: Endiesler Seel
Aquati Mammal
(a) Laabel-t-bbox Assigument
(b) Seli-Taaninng
Language Hierarchiral Self Self-Tmining (I.HST)
Figure I: Image-level labels in large-scale datasets such as ImageNet-2Ik [[6] often do not convey precise object information [77 [5] which affects while learning generalizable detectors. Recent
methods tackle this issue by various label-to-box assignment strategies [2] [3 14 [5] as in (a) but are heavily restricted by raw image-level labels and still suffer from image-to-box label mismatch [17]. Self-training [8] with the detectors pre-trained with [3 [4] [5] could circumvent the label mismatch issue but the generated pseudo box labels are error error-prone due to the lack of proper supervision as in (b). Our proposed LHST introduces language hierarchy to expand the image-level labels and enables co-regularization between the expanded labels and self-training which allows producing more accurate pseudo box labels in (c).
We propose incorporate image-level supervision with self-training for learning generalizable detectors, aiming to benefit from self training while effectively making use of image-leve weak supervision. We start from simple observation: the image-to-box label mismatch largely comes from the ambiguity in language hierarchy, e.g., the image-level label Aquatic Mammal in Figure I caa cover different object-level labels such seals, dolphins, walruses, etc. With the above observations, we design a Detector with Language Hierarchy (DetL.H) that combines language hierarchical self- training (LHST) and language hierarchical prompt generation (LHPG) for learning generalizable detectors.
LHST introduces WordNet's 's language hierarchy enables co-regularization between the expanded labels and self-training. Specifically, the expanded labels are not all reliable though they can mitigate the image-to-box label mismatch problem by providing richer supervision. Here self-training can predict reliability scores for the expanded labels for better selection or weightage of the expanded labels. At the other end, self-training with pseudo box labels allows learning from more proposals and can circumvent the image-to-box label mismatch but the box-level pseudo labels are usually noisy and may lead to learning degradation [15. Here the expanded labels provide richer and more flexible supervision which can effectively help suppress prediction noises in self-training.
2 to expand the image-level labels and accordingly
LHPG helps bridge the vocabulary gaps between training and testing by introducing WordNet's language hierarchy into prompt generation process. Specifically, LHPG leverages the CLIP language encoder [22) to measure the embedding distances between test concepts and WordNet synsets, and then generates the prompt for a given test concept from its best matched WordNet synset. In this way, the test prompts generated by LHPG have been standardized by WordNet and are well aligned with our proposed detector that is trained with WordNet information via LHST. In another word, the combination of LHST and LHPG actually leverages WordNet as a standard and intermediate vocabulary that bridges the gaps between training and testing vocabularies, generating better prompts and leading to better detection performance on downstream applications.
The main contributions of this work are threefold. First, we propose language hierarchical self- training that incorporates language hierarchy with self-training for weakly-supervist object detection. Second, design language hierarchical prompt generation, which introduces language hierarchy into prompt generation to bridge the vocabulary gaps between detector training and testing. Third, extensive experiments show that our DetL. achieves superior generalization performance consistently across 14 detection benchmarks.
2 Related Work
Yeakky-supervisee object detection (WSOD) aims to train object detectors using image-level
supervision. Traditional WSOD methods 23 24 any box annotations and thus focus on level proposal mining techniques 28%
25 26 27 use image-level annotations only without
29
12
30
32
leading unsatisfying localization performance. Semi-supervised WSOD
B3
34
35
36
37
38
39

---

has been proposed to further improve the performance, which leverages both box- and image-
level annotated data. With better localization quality, recent methods [33 14
15 40 design various
label-to-bol box assignment strategies, such as assigning image-level labels to max-score anchors [33. max-score proposals [1] or max-size proposals [15]. Our work belongs to semi-supervised WSOD.
Different from previous methods, we tackle the image-to-box label mismatch by introducing language hierarchy into self-training.
Large-vocabulary object detection
researches on detecting thousands of
categories. Most previous papers focus on tackling the long-tail tail issue [45 46
49 50), e-g., by
using equalization losses 51 [5], SeeSaw loss [53], or Federated Loss [54] Recent semi semi-superviied
WSOD methods [T3]
14 15 and our work circumvent the long-tail problem by leveraging more
balanced image-level datasets such as ImageNet-2IK.
Open-vocabulary object detection focuses detecting objects conditioned arbitrary words (i.e., [7
any category names). A common strategy [55 [6)
588 59 is to replace the detector's 's classification
layer with the language embeddings of category names. Recent methods [60]
61
62
63
17
[5]
leverage the powerful CLIP [22] model by using its text embeddings [60 61 62
63
17
15 or
conducting knowledge distillation 60 63 17 Similar to Detic [15,, our work uses CLIP text
embeddings as the classifier and leverages image-level annotated data instead of distilling knowledge from CLIP.
Language hierarchy has been widely studied for visual recognition tasks 644, especially for large-
vocabulary visual recognition. Most existing studies [65 66, 677 focus on image classification tasks,
e.g., leveraging language hierarchy for multi-label image classification [65 66
67
68
69
70
71]
modelling hierarchical relations among classes [68 69] facilitating classification training 770
711. Different from previous work, we introduce language hierarchy into self-training for weakly- supervised object detection.
3 Method
This work focuses on learning generalizable object detectors via weakly-supervised detector train- ing (15), which leverages additional large-scale image-level datasets to enlarge the data distributions and data vocabularies in detector training. We first describe the task definition with training and evaluation setups. Then, we present our proposed DetLH which is detailed in two major aspects on Language Hierarchical Self-training (LHST) that introduces language hierarchy into detector training. and Language Hierarchical Prompt Generation (LHPG) that introduces language hierarchy into prompt generation.
3.1 Task Definition
Training setup. The training data consists of two parts: 1) a detection dataset Daer {(x,, Vdet)s)) V)er)s) where denotes an image while ydst stands for the class and bounding box labels for x; 2) an image classification dataset Date = {(x, yets) Dett where Yets denotes the image- level label (i.e., a one-hot vector) for x. Given the datasets, the goal is to learn generalizable detection model F by jointly optimizing F over Dact and Dets:
Loss =
Σ
Cdec(E(x), ydet) + Σ Lweat yete);
(I)
(D
where Latet()) Erpn(.) L.eg(.)) + L.ds(.) is the fully-supervised detection loss function while Lrra (.), L.rg(:), and L.s(()) denote RPN, Regression, and Classification loss functions, respectively. Lweak is the weakly-supervised loss function to train detectors with image-level labels.
Evaluation setup. As the goal is to learn generalizable detection model that works well on various unseen downstream tasks, we conduct zero-shot cross-datases evaluation to assess the generalization
performance of the trained detection model. Note, different domain adaptation 772 73 74
75 that
generally uses downstream data in training, our setup is similar to domain generalization 76 77 that does not involve downstream data in training.
2 Ezoto-shot cross-datase evaluation here means that the model is evaluated on unseen datasets, which is the same the one defined in CLIP 22)
3

---

(et))) Image and its Rew Image Label
Merge Labels to Expand Poeudo Box Label
Aler Obas Mrree
Imagge
Expanded Image/ Box Labels their Confidence Vectors
Poor
Prodicted Pscudo Box Label
Prodect Confidence ssess Expanded Labels
WordNet Language Hierarchy
Aler
Nler hier O
omager W Vimage)
Detector
Predictions
foox
foox
Aite Ooler W box
Pseudo Label Generation
hler ( hier)
Mi fier
Detector
Predictions
Loss
Chher Woo
Training with Generated Labels
Figure 2: The proposed language hierarchical self- teffnirain consists of two flows including Pseudo Label Generation (top box) and Training with Generated Labels (bottom box). The Pseudo Label Generation flow leverages WordNet to expand the image-level labels, and then merges the expanded image-level labels with the predicted pseudo box labels, such that the expanded image-level labels could provide richer and more flexible supervision (than the limited and rigid raw labels) to regularize the self-training which is prone errors in pseudo labeling. In addition, as the labels expanded by WordNet (i.e., the expanded logits "I" in Vimage and ylee") are not all reliable, Pseudo Label Generation predicts reliability scores for the expanded labels adaptively re-weight them when applying them on different images or pseudo boxes. In Training with Generated Labels, we optimize the detector with the generated image-level and box-level labels, where the image-level training could regularize the training with pseudo box-level labels as pseudo box labels vary along training iterations and are not very stable.
hier
Open-vocabulary Detector. We modify the classification layer of the detector into an open- vocabulary format such that the detector could be tested over unseen datasets. Specifically, we replace the weights of the detector's classification layer with the fixed language embeddings encoded from class names, where the object classification could be achieved by matching the object's 's em- bedding and the fixed language embeddings. We adopt the CLIP language embeddings (22) as the classification weights as in [15 [60]. In this way, the modified detector could theoretically detect any target concepts on any target data. As reported in (15), detectors trained solely on detection datasets often exhibits constrained performance due to the small-scale training images and vocabularies. Similar to [5], our proposed DetLH introduces large-scale image-level datasets to enlarge the data distributions and data vocabularies in detector training, leading to more generalizable detectors and better generalization performance on various unseen datasets.
3.2 Language Hierarchical Self-training
The proposed LHST utilizes WordNet's language hierarchy to expand the image-level labels, which enables co-regularization between the expanded image-level labels and self self-training as illustrated in Figure(] []
Overview. For fully supervised detector training over the detection dataset, we feed box-level annotated samples (xr, ydct) E dett to the detection model F and optimize F with the standard fully supervised detection loss, i.e., the first term of Eq. [ For weakky-suuervised detector training over the leaee-level annotated dataset (Ir, Mets) E Dat shown in Figure 2 we first leverage WordNet's language hierarchy expand the raw image-level label Mets into Vineage (the hierarchical image-level hier label), and merge Vimage and the generated pseudo box label ybor to acquire yhser (the hierarchical box-level pseudo label). Then, we optimize the detector with (uhier, whhier) and (uiier hier where wheer and bier denote the predicted reliability scores of the expanded logits '1' in yimage and yhier and are used to weight the labels in loss calculation.
Expanding image labels with language hierarchy. Given image-level annotated dataset (ar, Wets) E Dats (Wets a label vector with length and C denotes the number of classes), we leverage WordNet's class name hierarchy [21 to expand yets into hier as the following:

---

Vimage
hier
WordNettyaa.).
(2)
where the function WordNet(-)) recursively finds all hypernyms and hyponyms of the input (i.e., the class indicated in Yets) and sets their positions in the label vector yets to be '1' to expand Yets into Vimage: In this way, single-label annotation could be expanded into multi-label annotation within the very rich ImageNet-2IK vocabulary.
Generating pseudo box labels with predictions. Given the image x E Deta, we feed æ into the detector F to acquire the prediction following:
(p% hsssssssss F(x),
where Pn is the probability vector of the predicted n-th bounding box after Softmax, and yf denotes the predicted e-th category probability. Note we filter out prediction if its max confidence score is lower than the threshold t, and N denotes the number of predicted object proposals after filtering, ic., maz (p.heeec)) 2 f, Vn.
Then the pseudo category label ÿboz = (ūr hisnSN for N boxes in image is derived by:
arg max In
In po s.t. In A Vn,
where in (yhe Un
(( the predicted category label, and A denotes a probability simplex with length C.
Merging image and pseudo box labels. As the predicted pseudo box label ybow error-prone, we regularize it with the expanded image-level supervision by merging hier and Fboz as the following:
hier (n) = ūbou( V Vimage: Vn,
(5)
where V denotes the logical "OR" operator.
Assessing the expanded labels. As the labels expanded by WordNet (i.e., the expanded logits 'I' in ghier h are not all reliable, we predict a reliability score wo, for the expanded label adaptively re-weight vh. ghier when applying on different pseudo boxes. We measure the (w JensNisese can be derived by:
reliability of yh, with prediction ph, and whier
ph. if hier Vimage otherwise,
(e)
+ (c)
Vets
(6)
where yiee + Wets returns True if the c-th label logit in vlinage is expanded by WordNet, which also applies ghier ghier is expanded by mergeing it with yimage.
(c)
Given the prediction Io0 the merged pseudo box label Wbser 1 MIsSMI and its reliability score hier www.hnsss.issss we optimize the detector as the following:
N
C
Loo (())) = E(CECE( vi) x wh),
where BCE(.) denotes the binary cross-entropy loss.
In addition, training with the predicted pseudo box labels not very stable as pseudo box labels vary along training process. Thus, we regularize the training of Looz (F (F(x)) with an image-level loss defined as the following:
C Limage (F(x)) Σ (BCE(Pinage imagee hier (0)) x Wimage);
(8)
where Pimage = V lsses denotes the category probability predicted for the image-level proposal. w wheerge SiimageeSsss denotes the reliability score for the expanded logits "I" in
5

---

(c)
y Vimage. Similar Eq. 6 image Pimage if Vimage BCE(-) denotes the binary cross-entropy loss.
Yels otherwise Winage 1. Besides,
Training objective. The overall training objective of Language Hierarchical Self-training is defined as:
Lisst
Σ
Ldet (F)),, Uddt) +
Σ
(Cboz(F + Limage (F()))
(D.eides)EDaed
(Z.Nes)EDets
Language Hierarchical Prompt Generation. As the goal learn a generalizable detection model that works well on various downstream tasks, one typical challenge is the vocabulary gap between detector training datasets (i.c., LVIS and ImageNet-21K5 and detector testing datasets (e.g., object365 or customized data). A common solution of tackling the vocabulary gaps is to conduct prompt learning 78 to generate proper category prompts. However, prompt learning generally requires labeled target images for additional training.
In this work, we tackle the vocabulary gaps by generating prompts with the help of WordNet, which introduces little computation overhead and does not require labeled target images and additional training. To this end, we design language hierarchical prompt generation (LHPG) that works by incorporating WordNet information into prompt generation process. Specifically, LHPG leverages CLIP language encoder [2 to measure the embedding distances between test concepts and WordNet synsets, and then generates the prompt for a given test concept from its best matched WordNet synset: WordNet CLIP(Viests WordNet), where Vieat denotes test vocabulary, WordNet denotes WordNet test
synsets, CLIP denotes CLIP language encoder and W/Wdrdeet stands for the best matched WordNet synsets for classes Vestt- Then, we generate test prompts from Viest WoWddee As compared with V-WadNNat has been standardized by WordNet and is well aligned with our proposed detector that is trained with WordNet information via LHST. In another word, the combination of LHST and LHPG makes of WordNet standard and intermediate vocabulary that bridges the gaps between training and testing vocabularies, generating better prompts and leading to hetter detection performance downstream applications.
Viest, our test
4 Experiments
We evaluate our DetLH on 14 widely adopted detection benchmarks. We follow the zero-shot cross- dataset object detection setting proposed 17 15] More details like Dataset and Implementation Details are provided in the appendix.
Table 1: Zero-shot cross-dataset object detection for common objects. All detectors are trained
over the training datasets (LVIS and ImageNet-2IK3 and evaluated over target datasets (i.e., Object365 and Pascal VOC with objects from common classes and scenarios) without finetuning. "Dataset- specific oracles" denote the detectors that are fully supervised which are trained by using the training data of respective datasets.
Object365
79
Pascal VOC
(0
Method
AP
AP50
AP75
APs
APm
API
AP
AP50
AP75
APs
APm
API
WSDDN 12
21.0
29.1
22.7
8.7
20.9
31.2
61.6
82.7
67.5
24.8
50.9
73.5
YOLO9000 13
21.0
28.5
22.6
8.6
20.7
30.9
62.6
83.6
68.7
23.7
52.0
73.9
DLWL [1
21.3
29.1
23.0
8.8
21.0
31.5
62.4
83.4
68.3
23.8
51.2
73.8
Detic [1S]]
21.6
29.4
23.4
9.0
21.4
31.9
62.4
83.3
68.5
23.7
51.8
73.9
DetLH (Ours)
23.6
32.5
25.5
9.8
23.5
35.0
64.4
86.1
70.8
25.3
54.1
75.3
Dataset-specifee oracles
31.2
54.4
79.7
59.1
19.0
40.8
64.5
4.1
Comparison with the statc-of-the-art
We conduct extensive experiments to benchmark our proposed DetLH with state-of-the methods. evaluate them on 14 widely studied object detection datasets to assess their zero-shot cross-dataset generalization ability. Tables [ report zero-shot cross-dataset detection results for common objects, autonomous driving, intelligent surveillance, and wildlife detection, respectively. More details are to be described in the following paragraphs.

---

Table 2: Zero-shot cross-dataset object detection for autonomous driving. All detectors trained over the training datasets (LVIS and ImageNet-21K3 and evaluated over autonomous driving datasets (i.e., Cityscapes, Vistas and SODAIOM) without finetuning.
Method
Cityscapes
Vistas
SODAIOM
Average
AP
AP50
AP75
AP
AP50
AP75
AP
AP50
AP75
AP50
AP75
WSDDN 12
28.2
45.4
27.1
22.3
34,0
23.3
17.4
28.9
17.1
22.6
36.1
22.4
YOLO9000
28.8
46.2
27.4
22.5
23.4
37.0
DLWL 14
28.6
45.6
28.1
22.5
34.7
23.2
18.3
30.4
18.0
23.1
36.9
23.0
Detic 115
29.6
47.1
28.4
23.0
35.6
23.6
18.8
30.9
18.5
23.8
37.9
23.5
DetL (Ours)
31.2
50.3
29.1
26.5
44.0
25.8
25.1
38.4
26.1
27.6
44.2
27.0
Dataset- speciffc oracles
43.0
69.0
42.6
28.1
45.8
28.5
44.7
68.2
47.3
38.6
61.0
39.5
Table 3: Zero-shot cross-dataset object detection under different weather and time-of-day conditions (using metric AP50). All detectors are trained over the training datasets (LVIS and ImageNet-21IK) and evaluated over BDDIOOK and DAWN datasets without finetuning.
Method
BDDIOOK weather 31
BDD time of
DAWN
raiiy
snowy
cloudy
foggy
daytume
downAddusk
undefined
tog
WSDDN
27.9
55.0
33.6
39.5
34.8
33.4
43.8
40.1
35.1
28.7
45.0
62.1
63.7
43.6
Detic 15
34.1
33.2
39.5
41.9
27.9
45.4
39.2
35.5
28.8
48.2
52.3
54.1
56.1
41.1
DeiLif (Ours)
40.2
37.5
48.2
49.3
37.1
49.9
45.7
40.0
34.2
53.0
63.2
57.6
67.3
47.9
Dutsset-spesee oracles
52.0
52.5
56.3
56.3
21.3
65.4
57.0
50.4
48.6
27.7
56.7
48.4
26.4
47.6
Table 4: Zero-shot cross-dataset object detection for intelligent surveillance. All detectors are trained over the training datasets (LVIS and ImageNet-21K) and evaluated over surveillance datasets MIO-TCD, BAAI-VANJER DETRAC and UAVDT without finetuning.
Method
MIO-TCD
3
BAAI-VANJEE 6
DETRAC 7
UAVDTT B
Average
AP50
AP15
AP50
AP75
AP50
AP75
AP
AP50
AP75
AP
AP50
AP7S
WSDDN 12
11.3
17.6
11.6
13.1
19.6
13.3
25.6
35.3
31.1
17.1
31.9
16.0
16.7
26.1
18.0
YOLOADO 13
12.7
19.7
13.0
13.1
19.3
13.0
29.1
39.4
35.3
18.6
33.9
17.7
18.3
28.1
19.7
38.0
Detic 15
13.4
20.6
13.9
16.9
23.6
17.6
28.7
39.2
34.8
18.6
34.2
17.6
19.4
29.1
21.0
DetLi (Ours)
15.8
24.5
16.0
17.9
25.1
18.5
32.7
44.0
39.7
20.1
36.6
19.3
21.6
32.6
23.4
Datasct-speciiic oracles
58.6
60.4
63.2
48.3
Object detection for common objects. Table [ shows that DetLH outperforms state-of-the--e methods clearly on common object datasets Object365 and Pascal VOC. In addition, observe that DetLH even brings significant gains above the dutaset-specific oracle (i.e., the model that is fully trained the target training data) Pascal VOC (i.e., a small-scale dataset), showing the advantages of leveraging large-scale training data.
Object detection for autonomous driving. As shown in Table 2 our DetLH outperforms state-of- the-art methods by large margins on various autonomous driving datasets, showing that DetLH still works effectively while facing large variations in camera views from autonomous driving scenarios to the base-dataset scenarios (LVIS and ImageNet-21K), e.g., autonomous driving images are captured under very different camera views. In addition, the experimental results in Table [ show that our DetLH brings significant performance gains against state-of-the-art methods when encountering various weather and time-of-day conditions, which demonstrates the effectiveness of DetLH while detecting objects under large noises 83 e.g., the images captured under different weather and time-of-day conditions may have very different styles and image quality.
Object detection for intelligent surveillance. From Table 4 we can observe that our DetLH outperforms state-of-the-att methods by clear margins on various intelligent surveillance datasets, indicating that DetLH also tolerant to large changes in the camera lens and angles which often happen to pewllieent-survilllaanee angles (e.g.. surveillance cameras are often with the wide-angle lens and used in high angle views).
images that are captured under very different camera lens and
Object detection for Wildlife. The experimental results in Table show that our DetLH performs well on various wildlife detection datasets, showing that DetL. works effectively for detecting fine-grained categories that exist widely in wildlife detection datasets. The significant performance
7

---

Table 5: Zero-shot cross-dataset object detection for Wildlife Detection. All detectors trained over the training datasets (LVIS and (mageNet-21K) and evaluated over wildlife datasets (i.e., Arthropod Detection, AfricanWildlif and Animals Detection) without finetuning.
Method
Arthropod Detection
g
African Wildlife
10
Animals Detection
M||
Average
AP
AP50
AP75
AP
AP50
AP75
AP
AP50
AP75
AP
APS0
AP7S
WSDDN
18.1
26.3
18.8
76.7
88.2
84.0
36.0
41.7
37.5
43.6
52.0
46.7
YOLO9000 1K]
22.6
33.2
22.5
75.9
87.5
83.2
39.0
45,4
40.8
45.8
55.3
48.8
DLAVI 14
25.3
34.7
26.3
74.7
86.2
81.3
41.7
48.1
43.7
47.2
56.3
50..4
Detic 15
27.4
36.7
29.2
68.9
80.9
76.4
41.1
47.7
42.9
45.8
55.1
49.5
DetLH (Ours)
36.2
49.0
38,3
74.8
87.2
81.8
44.3
51.2
46.3
51.8
62.5
55,5
Dataset-specite oracles
75.1
86.3
79.9
82.7
90.9
89.1
64.4
74.6
69.4
74.1
83,9
79.5
gains largely come from the introduction of language hierarchy into detector training and prompt generation, which helps model the hierarchical relations among parent and fine-grained subcategories and thus leads to better fine-grained object detection.
The superior detection performance of our DetLH is largely attributed our two core designs, i.e., LHST and LHPG. LHST enables effective usage of large-scale image-level annotated images and significantly enlarges the data distribution and the data vocabulary in detector training. yielding robust performance under large cross cross-dataset gaps in data distribution and vocabulary. LHPG ingeniously helps mitigate the vocabulary gaps between detector training and testing. improves the overall confidence of detection and benefits the detection as large data distribution gaps (or large data vocabulary gaps) often lead to low-confidence predictions and poor detection results.
4.2 Ablation Studies
We perform ablation studies with Swin-B (84] based CenterNet2 [54] over the large-scale Ob-
Table 6: Ablation studies of our DetLH with Lan- guage Hierarchical Self-training (LHST) and Lan- guage Hicrarchical Prompt Generation (LHPG). The experiments are conducted with Swin-B based Center- Net2 [15] and the detectors are evaluated on Object365
ject365 dataset as shown in Table As the core of our proposed DetLH, we examine how our designed LHST and LHPG contribute to the over- all performance of zero-shot cross-dataset object detection. As shown in Table the baseline (Box- Supervised [15) does not perform well as it uses box-level training data only. It can be observed that LHST outperforms the baseline clearly, show- ing that LHST can effectively leverage the large- scale image-level annotated dataset to signifi- cantly enlarge the data distribution and data cabulary involved in detector training, leading to much better zero-shot cross-dataset detection performance. In addition, LHPG brings clear perfor- mance improvements in zero-shot cross-dataset detection by introducing language hierarchy into prompt generation, demonstrating the effectiveness of LHPG in mitigating the vocabulary gaps between training and testing. Moreover, the inclusion of both LHST and LHPG in the proposed DetLH performs clearly the best, indicating the complementary property of our two designs.
in zero-shot cross-dataset object detection setup.
Method
LHST LHPG AP50
Box-Supervised [15
26.5
31.3
31.0
DetLH (Ours)
32.5
4.3 Discussion
Table 7: Zero-shot cross-dataset object detection various datasets. Results averaged on 14 widely studied datasets.
Averaged over 14 detection datasets
Method
AP
AP50
AP75
APs
APm
API
WSDDN 12
29.9
42.5
31.4
14.8
25.9
44.2
YOLO9000 13
30.9
43.8
32.4
14.1
25.8
45.1
DLWL [4]
31.0
44.0
32.5
15.4
26.3
45.3
Detic [5]
31.0
44.0
32.8
14.6
27.5
45.5
DetLH (Ours)
34.6
49.3
36.4
16.0
28.4
49.5
8

---

Generalization across various detection tasks: We study the generalization of our DetLH by conducting zero-shot cross-datases object detection on 14 widely studied object detection datasets. Tables T show that Detl .H achieves superior performance consistently across all the detection T the detection results averaged on datasets, showing that DetLH clearly outperforms the state-of-the-arr methods.
applications. Besides, Table summarizzs
Generalization across various network architectures: We study the generalization of the proposed DetLH from the perspective of network architectures. Specifically, we perform extensive evaluations with four representative network architectures, including one Transformer-bast -based (i.e., Swin-B) -B) and three CNN-based (i.e., ConvNeXt-T, ResNet-50 and ResNet- 18). Experimental results in Table g show that the proposed DetLH outperforms the state-of-the- method consistently over different network architectures.
Table 8: Zero-shot cross-dataset object detection with different network architectures. All networks architectures trained over the training datasets (LVIS and ImageNet-21k) and evaluated over 0bject365 without finetuning.
Object365
Method
Architecture
AP
AP50
AP75
APs
APm
API
Detic 15
21.6
29.4
23.4
9.0
21.4
31.9
Swin-B
84
DetLH (Ours)
23.6
32.5
25.5
9.8
23,5
35.0
Detic 15
16,9
23.5
18.1
6.8
16,6
24.9
ConvNeXI-T
85
DetLH (Ours)
18.9
26.8
20.2
7.6
18.8
28.2
Detic 15
16.2
22.8
17.5
6.3
16.2
24.1
ResNet-50 86
DetLH (Ours)
17.7
25.5
19.0
6.9
17.9
26.4
Detic 15]
10.8
15.5
11.6
3.9
10.2
16.2
ResNet-18
86
DetLH (Ours)
11.8
17.3
12.5
4.3
11.4
17.7
Parameter Studies for Language Hierarchical Self- training (LHST). In generating pseudo box labels in LHST, we filter out prediction if max confidence score lower than the threshold t. We study the threshold / by changing it from 0.65 to 0.85 with step of 0.05. Table 12 reports the experimental results on zero-shot transfer object detection over object365 dataset. We can observe that the detection performance is not sensitive the threshold t.
Table 9: Parameter Studies for Language Hierarchical Self- training (LHST) on zero-shot transfer object detection over object365 dataset. We study the thresholding parameter / used in generating pseudo box labels LHST.
Threshold
0.65
0.70
0.75
0.80
0.85
AP50
31.1
31.3
31.3
31.3
31.2
Due to the space limit, we provide more DetLH discussions and visualizations in the appendix.
5 Conclusion
This paper presents DetLH, Detector with Language Hicrarchy that combines language hierarchical self-training (LHST) and language hicrarchical prompt generation (LHPG) for learning generalizable object detectors. LHST introduces WordNet's language hierarchy to expand the image-level labels and accordingly enables co-regularization between the expanded labels and self-training. LHPG helps mitigate the vocabulary gaps between training and testing by introducing WordNet's language hierarchy into prompt generation. Extensive experiments over multiple object detection tasks show that our DetLH achieves superior performance as compared with state-of-the-art methods. In addition, we demonstrate that DetLH works well with different network architectures such as Swin-B, Net-T, ResNet-50, etc. Moving forward, we will explore language hierarchy to further expand the labels in open- vocabbacab manner in addition to the closed ImageNet-211e 21'ss vocabulary.

---

References
[1] Marius Cordts, Mohamed Omran, Schastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Be- nenson, Uwe Franke, Stefan Roth, and Bert Schiele. The eityscapes dataset for semantic urban scene understanding. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213-3223, 2016.
[2] Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In Proceedings of the IEFE international conference on computer vision, pages 4990-4999 2017.
[3] Jianhua Han, Xiwen Liang, Hang Xu, Kai Chen, Lanqing Hong, Jiageng Mao, Chaoqiang Ye, Wei Zhang, Zhenguo Li, Xiaodan Liang, et al. SodalOm: a large-scale 2d salf/semi-superviii object detection dataset for autonomous driving. arXiv preprint arXiv:2106.1 1110....11...............1111111 2021.
[4] Jingyi Zhang, Jiaxing Huang. Zhipeng Luo, Gongje Zhang, Xiaoqin Zhang, and Shijian Lu. Da-detr: Domain adaptive detection transformer with information fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23787-23799 2023.
[5] Zhiming Luo, Frederic Branchaud-Chann Charron, Carl Lemaire, Janusz Konrad, Shaozi Li, Akshaya Mishra, Andrew Achkar, Justin Eichel, and Pierre-Mare Jodoin. Mio-ted: A new benchmark dataset for vehicle classification and localization. IEEE Transactions Image Processing, (7(10):5129-514 2018.
[6] Deng Yongqiang, Wang Dengjiang, Cao Gang, Ma Bing, Guan Xiji,, Wang Yajun, Liu Jianchao, Fang Yanming, and Li Juanjuan. Baai-vanjee roadside dataset: Towards the connected automated vehicle highway technologies in challenging environments of china. arXiv preprint arXiv: 103. 2021.
[7] Longyin Wea, Dawei Du, Zhaowei Cai, Zhen Lei, Ming SanngChing Chang. Honggang Qi, Jongwoo Lim, Ming- Ming-Hsuan Yang, and Siwei Lyu. Ua-detrac: A new benchmark and protocol for multi-object detection and tracking. Computer Vision and Image Understonding, 193: 192:102907, 2020.
[8] Dawei Du, Yuankai Qi, Hongyang Yu, Yifan Yang. Kaiwen Duan, Guorong Li, Weigang Zhang, Qingming Huang, and Qi Tian. The unmanned aerial vehicle benchmark: Object detection and tracking. In Proceedings the European conference computer vision (ECCV), pages 370- 386,,386.000 2018
[9] Geir Drange. Arthropod taxonomy orders object detection dataset, 2020.
[10] Wichayas YoLov5. African animals dataset. http::////miiieereeoooooooooooo/ african, animalsnn may 2022. visited on 2023-03-01.
[11] Kaggle. Animal detection dataset.
animal dete-ciioeeiinneeee 7vafe, apr 2022. visited on 2023-03-01.
[12) Hakan Bilen and Andrea Vedaldi. Weakly supervised deep detection networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2846-2854, 2016.
[13] Joseph Redmon and Ali Farhadi. Yolo9000: better, faster,, stronger. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7263-7271, 2017.
[14] Vignesh Ramanathan, Rui Wang, and Dhruv Mahajan. Diwl: Improving detection for lowshot classes with weakly labelled data. In Proceedings of the IEFF/CVF conference computer vision and pattern recognition. pages 9342-9352, 2020.
[15) Zhou Xingyi. Detecting twenty- thousand classes using image-level supervision. arXiv preprint 1XXi:2201.02600 2022.
[16] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248 255. lece, 2009.
[17] Hanoona Rasheed, Muhammad Maaz, Muhammad Uzair Khattak, Salman Khan, and Fahad Shahbaz Khan. Bridging the gap between object and image-level representations for open-vocabulary detection. arXiv preprint arXiv:2207.00 03342, 2022.
[18] Kihyuk Sohn, Zizhao Zhang, Chun-Liang Li, Han Zhang, Chen- Yu Lee, and Tomas Pfister, A simple semi-supervised learning framework for object detection. arXiv preprint 27X7772005.04757, 2020.
[19] Jingyi Zhang, Jiaxing Huang, Zichen Tian, and Shijian Lu, Spectral unsupervised domain adaptation for visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9829 9840, 2022.
10

---

[20] Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Cross-view regularization for domain adaptive panoptic segmentation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10133-10144, 2021.
[21] George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39 41, 1995.
[22] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[23] Xiaoyan Li, Meina Kan, Shiguang Shan, and Xilin Chen. Weakly supervised object detection with segmentation collaboration. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9735-9744, 2019.
[24] Yunhang Shen, Rongrong Ji, Yan Wang, Zhiwei Chen, Feng Zheng. Feiyue Huang, and Yunsheng Wu. Enabling deep residual networks for weakly supervised object detection. In Computer Vision-ECCH 2020: 16th European Conference, Glasgow, UK. August 28-28, 2020, Proceedings, Part VIII 16. pages 118-136. Springer, 2020.
[25] Yunhang Shen, Rongrong Ji, Yan Wang, Yongjian Wu, and Liujuan Cao. Cyclic guidance for weakly supervised joint detection and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 697-707, 2019.
[26] Fang Wan, Chang Liu, Wei Ke, Xiangyang Ji, Jianbin Jiao, and Qixiang Ye. C-mil: Continuation multiple instance learning for weakly supervised ohject detection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 2199-22222222 2208, 2019.
[27] Ke Yang, Dongsheng Li. and Yong Dou. Towards precise end-to-end weakly supervised object detection network. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 8372- 8381, 2019.
[28) Pablo Arbeláez, Jordi Pont- Tuset, Jonathan T Barron, Ferran Marques, and Jitendra Malik. Multi- scale combinatorial grouping. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 328 335, 2014.
[29] Jasper RR Uijlings, Koen RA Van De Sande, Theo Gevers, and Arnold WM Smeulders. Selective search for object recognition. International journal of computer vision, 104:154-171, 2013.
[30] Peng Tang, Xinggang Wang, Xiang Bai, and Wenyu Liu. Multiple instance detection network with online instance classifier refinement. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2843-2851, 2017.
[31] Peng Tang, Xinggang Wang, Song Bai, Wei Shen, Xiang Bai, Wenyu Liu, and Alan Yuille. Pel: Proposal cluster learning for weakly supervised object detection. IEEE transactions on pattern analysis and machine intelligence. 42(1): (766:11766711111 176-191, 2018
[32] Zeyi Huang, Yang Zou, BVK Kumar, and Dong Huang, Comprehensive attention self-distillation for weakly-supervisen object detection. Advances neural information processing systems, 33: 33:16797-16807, 2020.
[33] Bowen Dong, Zitong Huang, Yuelin Guo, Qilong Wang. Zhenxing Niu, and Wangmeng Zuo. Boosting weakly supervised object detection via learning bounding box adjusters. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 2876-2885, 2021.
[34] Shijic Fang, Yuhang Cao, Xinjiang Wang, Kai Chen, Dahua Lin, and Wayne Zhang. Wssod: A new pipeline for weakly-and semi-supervised object detection. arXiv preprint arXiv:2105. 1225,, 2021.
[35] Yan Li, Junge Zhang, Kaiqi Huang, and Jianguo Zhang, Mixed supervised object detection with robust objectness transfer. IEEE transactions on pattern analysis and machine intelligence, 41(3):639-653, 2018.
[36] Yan Liu, Zhijie Zhang, Li Niu, Junjie Chen, and Liqing Zhang. Mixed supervised object detection by transferring mask prior and semantic similarity. Advances in Neural Information Processing Systems. 34:3978-3991 2021.
11

---

[37] Jasper Uijlings, Stefan Popov, and Vittorio Ferrari. Revisiting knowledge transfer for training object class detectors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1101-1110, 2018.
[38] Ziang Yan, Jian Liang, Weishen Pan, Jin Li, and Changshui Zhang. Weakly-and semi-supervised object detection with expectation -atiattioommmaimiiattiionn algorithm. arXiv preprint a7Xiv: 2017.
[39] Yuanyi Zhong. Jianfeng Wang, Jian Peng, and Lei Zhang. Boosting weakly supervised object detection with progressive knowledge transfer. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XXVI, pages 615-63 Springer, 2020.
[40] Cheng Zhang. Tai-Yo Yu Pan, Yandong Li, Hexiang Hu, Dong Xuan, Soravit Changpinyo, Boqing Gong, and Wei-Lun Chao. Mosaicos: simple and effective use of object-centric images for long-tailed object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 417-427, 2021.
[41] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5356- 5364, 2019.
[42] Bharat Singh, Hengduo Li, Abhishek Sharma, and Larry S Davis. R-fen-3000 at 30fps: Decoupling detection and classification. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 1081-1090, 2018.
[43] Hao Yang, Hao Wu, and Hao Chen. Detecting 11k classes: Large scale object detection without fine- grained bounding boxes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9805 1805-9813,- 18, 2019.
[44] Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision language models for vision tasks: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[45] Nadine Chang, Zhiding Yu, Yu-Xiong Wang, Animashree Anandkumar, Sanja Fidler, and Jose M Alvarez. Image-level or object-level? a tale of two resampling strategies for long-tailed detection. In International conference on machine learning, pages 1463-1472. PMLR, 2021.
[46] Chengjian Feng, Yujie Zhong, and Weilin Huang. Exploring classification equilibrium in long-tailed tailed object detection. In Proceedings of the IREF/CV International conference on computer vision, pages 3417-3426, 2021.
[47] Yu Li, Tao Wang, Bingyi Kang, Sheng Tang, Chunfeng Wang. Jintao Li, and Jiashi Feng. Overcoming classifier imbalance for long-tail object detection with balanced group softmax. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10991-11000, 2020.
[48] Tai-Ya Yu Pan, Cheng Zhang, Yandong Li, Hexiang Hu, Dong Xuan, Soravit Changpinyo, Boqing Gong, and Wei-Lun Chao. On model calibration for long-tailed object detection and instance segmentation. Advances in Neural Information Processing Systems, 34:2529 2542, 2021.
[49] Jialian Wu, Liangehen Song, Tiancai Wang, Qian Zhang, and Junsong Yuan. Forest r-enm: Large- vocabulary long-tailed object detection and instance segmentation. In Proceedings of the 28th ACM International Conference on Multimedia, pages 1570-1578. 2020.
[50] Songyang Zhang, Zeming Li. Shipeng Yan, Xuming He, and Jian Sun. Distribution aligument: unified framework for long-tail visual recognition. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 2361-2370, 2021.
[51] Jingru Tan, Xin Lu, Gang Zhang, Changqing Yin, and Quanquan Li. Equalization loss v2: new gradient balance approach for long-tailed object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1685-1694, 2021.
[52] Jingru Tan, Changbao Wang, Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjic Yan. Equalization loss for long-tailed object recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11 11662- 11671, 2020.
[53] Jiaqi Wang, Wenwei Zhang, Yuhang Zang, Yuhang Cao, Jiangmian Pang, Tao Gong, Kai Chen, Ziwei Liu, Chen Change Loy. and Dahua Lin. Seesaw loss for long-tailed instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9695-9704, 2021.
[54] Xingyi Zhou, Vladien Koltun, and Philipp Krähenbühl. Probabilistic two-stage detection. arXiv preprint arXiv:2103.0761, 2021.
12

---

[55] Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa. and Ajay Divakaran. Zero-shot object detection. In Proceedings of the European Conference on Computer Vision (ECCV), pages 480, 2018.
[56] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing ( (EMNLP), pages 1532- 1543, 2014.
[57] Shafin Rahman, Salman Khan, and Nick Barnss. Improved visual-semantic alignment for zero-shot object detection. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 34, pages 11932-11939, 2020.
[58] Zhihui Li, Lina Yao, Xiaogin Zhang, Xianzhi Wang. Salil Kanhere, and Huaxiang Zhang. Zero-shot object detection with textual descriptions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8690-8697, 2019.
[59] Alireza Zarcian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14402_ 2021.
[60] Xiuye Gu. Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Open- vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104. 1399,, 2021.
[61] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Open-vocabulary image segmentation. arXiv preprint arXiv:2112.1 12143, 2021.
[62] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language-driven semantic segmentation. arXiv preprins arXiv:2201.0333 03546, 2022.
[63] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary detr with conditional matching. arXiv preprint wcXiv:2203.11877 2022.
[64] Carlos N Silla and Alex A Freitas. A survey of hicrarchical classification across different application domains. Data Mining and Knowledge Discovery, 22:31-72, 2011.
[65] Ivica Dimitrovski, Dragi Kocev, Suzana Loskovska, and Sašo Džeroski. Hierarchical annotation of medical images. Pattern Recognition,
2011.
[66] Hui Liu, Danqing Zhang, Bing Yin, and Xiaodan Zhu. Improving pretrained models for zero-shot multi- label text classification through reinforced label hicrarchy reasoning. arXiv preprint arXiv:2104.01666 2021.
[67] Ilias Chalkidis, Androutsopoulos. An empirical study on large-scale multi-label text classification including few and zero-shot labels. arXiv preprint arXiv:2010.01633, 2020.
Manos Fergadiotis, Sotiris Kotitsas, Prodromos Malakasiotis, Nikolaos Aletras, and Ion
[68] Shiming Chen, Guosen Xie, Yang Liu, Qinmu Peng, Baigui Sun, Hao Li, Xinge You, and Ling Shao. Hsva: Hierarchical semantic-visual adaptation for zero-shot learning. Advances in Neural Information Processing Systems, 34:16622 16634, 2021.
[69] Thomas Mensink, Efstratios Gavves, and Cees GM Snoek. Costa: Co-occurrence statistics for zero-shot classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2441-2448, 2014.
[70] Kai Yi. Xiaoqian Shen, Yunhao Gou, and Mohamed Elhoseiny. Exploring hierarchical graph represen- tation for large-scale zero-shot image classification. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XX, pages 1 116-132. Springer, 2022.
[71] Zhong Cao, Jiang Lu, Sen Cui, and Changshui Zhang. Zero-shot handwritten chinese character recognition with hierarchical decomposition embedding. Pattern Recognition, 107:107488, 2020.
[72] Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey. Neurocomputing, 312:135-153, 2018.
[73] Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data. Advances in Neural Information Processing Systems, 34, 2021.
13

---

[74] Pan Zhang, Bo Zhang, Ting Zhang, Dong Chen, Yong Wang, and Fang Wen. Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12414-12424, 2021.
[75] Jiaxing Huang. Dayan Guan, Aoran Xiao, Shijian Lu, and Ling Shao. Category contrast for unsupervised domain adaptation in visual tasks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 1203-1214, 2022.
[76] Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Fsdr: Frequency space domain randomization for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6891 1-9022 2021.
[77] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(4):4396 4415, 2022.
[78] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Conditional prompt learning for Dusion-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2022.
[79] Shuai Shao, Zeming Li. Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8430 8009,843,,,40 2019.
[80] Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303-308, 2009.
[81] Fisher Yu, Haofeng Chen. Xin Wang, Wenqi Xian. Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2636-2644 2020.
[82] Mourad A Kenk and Mahmoud Hassaballah. Dawn: vehicle detection in adverse weather nature dataset. arXiv preprint arXiv:2008. 05400, 2020.
[83] Jiaxing Huang. Dayan Guan, Aoran Xiao, and Shijian Lu. Rda: Robust domain adaptation via fourier adversarial attacking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8988 8999, 2021.
[84] Ze Liu, Yutong Lin, Yue Cao, Han Hu. Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022. 2021.
[85] Zhuang Liu, Hanzi Mao, Chao Yuan Wu, Christoph Feichtenhofer. Trevor Darrell, and Saining Xie. convnct for the 2020s. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 11976-11986 2022.
[86] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEFE conference on computer vision and pattern recognition, pages 770-778, 2016.
[87] Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMP- STAT pages 187-186. Springer, 2010.
[88] Siddhesh Khandelwal, Raghav Goyal, and Leonid Sigal. Unit: Unified knowledge transfer for any-shot object detection and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5951 5961, 2021.
[89] Yunqiu Xu, Chunluan Zhou, Xin Yu, and Yi Yang. Cyclic self-training with proposal weight modulation for cross-supervised object detection. IEEE Transactions Image Processing,
2002, 2023.
[90] Yunqiu Xu, Yifan Sun, Zongxin Yang, Jiaxu Miao, and Yi Yang. H2Aa r-can: Holistic and hierarchical feature alignment for cross-domain weakly supervised object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14329-1453 14339, 2022.
[91] Krishna Kumar Singh, Santosh Divvala, Ali Farhadi, and Yong Jae Lee. Dock: Detecting objects by transferring cummon-sense knowledge. In Proceedings of the European Conference on Computer Vision (ECCV), pages 492-508, 2018
[92] Liunian Harold Li Pengchuan Zhang*, Haotian Zhang Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jeng-Neng Hwang. Kai-Wei Chang, and Jianfeng Gao. Grounded language-ing -maaaagg--mageeeee pre- pre-training. In CVPR, 2022.
14

---

[93] Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu. Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detelip: Dictionary-enricher visual-concept paralleled pre-training for open-world detection. arXiv preprint urXiv:2209.09440 2022.
[94] Yutong Lin, Chen Li, Yue Cao, Zheng Zhang, Jianfeng Wang. Lijuan Wang. Zicheng Liu, and Han Hu. simple approach and benchmark 21.000-category object detection. Computer Vision-ECCO ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XI, pages 1-18. Springer, 2022.
[95] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m Open dataset of clip-filtered 400 million image-tex. text pairs. arXiv preprint arXiv:2111.02111 2021.
[96] Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang. Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipy2: Unifying localization and dision-lanuuee understanding. Advances Neural Information Processing Systems, 11136067-360800 2022.
1971 Matthias Minderer. Simple open-vocabulary object detection with vision transformers. ECCV, 2022.
[98] Tiancheng Zhao, Peng Liu, Xiaopeng Lu, and Kyusong Lec. Omdet: Language-aware object detection with scale vision-lange laiiual dutti-dataset pre pre-trainnnn trainingg arXiv preprint arXiv:2209.05946, 2022.
[99] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Simple multi-dataset detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7571-7580, 2022.
[100] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang. Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.1132, 2021.
[101] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor, Imagenet- pretraining for the 2021.
masses. arXiv preprint arXiv:2104.
[102] Alina Kuznetsova, Hassan Rom. Neil Alldrin, Jasper Uijlings. Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci. Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International Journal of Computer Vision, 128(7):1956- 1981, 2020.
15

---

Appendix
A Dataset and Implementation Details
A.1 Implementation Details
As in 17 [5]. we adopt the CenterNet2 [54] with Swin-B [84] backbone in all the experiments (except for
Table 8 where different backbone architectures were used, c.g., ConvNeXt-T, ResNet-50 and ResNet- 18).
We employ SGD 87 as the optimizer and adopt the cosine lcarning rate scheduler with warm-up of 1000
itcrations 115 We set the input sizes of box-level annotated images (i.c., LVIS) and image-level annotated
images (i.e.. ImageNet-21k as 896 x 896 and 448 x 448, respectively. As mentioned in Section 3.1] we employ the CLIP text embeddings [22] as the classifier instead of using the original one in [54]. During training, we sample box-level and image-level miinibbtcches in 1 16 ratio. We the confidence threshold (in pseudo box label generation Eq. 3 as 0.75 in experiments except parameter analysis. Note we pre-train the detector over the training datasets (i.c., training on LVIS with the conventional detection loss and on ImageNet-22 with the conventional image classification loss) such that it can generate pseudo box loxelevel labels of 2 classes for self- trff-training.
As described in the main text, we train our detector over two training datasets LVIS and ImageNet-21K, and evaluate the trained detector over 14 evaluation datasets as listed without fine-tuning.
A.2 Training Dataset
LVIS [4] is a large vocabulary dataset designed for long-tailed instance segmentation, which contains 100K images and 1203 categories. LVIS provides high-quality instance-win wise annotations, including instance masks, class labels and bounding boxes.
ImageNet-21K [66] is a large and diverse dataset over 14M images across more than 21K categories. All categories in IetteN are defined by WordNet Synsets with clear and accurate definitions and certain language hierarchy.
A.3 Evaluation Dataset
Object365 [79] a large-scale object detection dataset designed for object detection in the wild. This dataset contains 638K images across 365 categories, including 600K images training and 38K images validation.
Pascal VOC SO is a real-world dataset with two sub-datasets, i.e., PASCAL VOC 2007 and PASCAL VOC 2012. PASCAL VOC 2007 contains 2,501 training images and 2,510 validation images, and PASCAL VOC 2012 contains 5,717 training images and 5,823 validation images. This dataset provides bounding box annotations with 20 categories.
Cityscapes is dataset designed for the understanding of street scenes. The images in Cityscapes captured under normal weather conditions from 50 cities, including 2,975 training images and 500 validation images with pixel-wise instance annotations categories.
a street-level autonomous driving dataset. This dataset contains high-resolution images that cover diverse urban scenes from around the world, including 18K training images and 2K validation images with pixel-wise instance annotations.
Vista 2
SODAIOM 3 is a large-scale object detection dataset for autonomous driving, which contains 10M unlabeled images and 20K images with bounding box annotations of 6 object categories. The images in this dataset collected within 27833 driving hours covering variety time periods and locations across 32 different cities.
a large-scale driving video datasct that contains diverse driving scenarios, including different weather conditions (f.e., clear, cloudy, overcast, rainy, snowy and foggy) and times of day (é.e., dawn, daytime and night). This datasct contains 100K vidcos, including 70K training videos and 10K validation vidcos with bounding box annotations of 10 categories.
BDDI0OK (TT)
Arthropod Detection g is detection dataset for arthropods taxonomy orders identification. The images are collected from variety of agricultural settings (e.g., .., fields, greenhouses, warchouses), including over 12K images with bounding box annotations 7 categories.
African Wildlife 10] is a detection dataset which contains images of African wildlife with bounding box annotations. This dataset contains different categories of African wildlife including buffalo, elephant, rhino, zebra and each category contains 376 images.
Animals Detection []] is public dataset of various animals. This dataset contains animal images with bounding boxes of 80 different animal categories, including 6.8K training images and 1.9K validation images.
16

---

is vehicle detection dataset that focuses on diverse traffic environment. This dataset contains IK images from real-traffic environment, including fog, snow, rain and sandstorms. The images are annotated with object bounding boxes calegories.
DAWN [22
MIO-TCD (5) is an intelligent surveillance dataset for motorized traffic analysis, which contains 137,743 images captured in various times of the day and different periods of the year, and from different viewing angels. This dataset provides bounding box annotations with 11 categories.
BAAI-VANJEE 66] is an intelligent surveillance datasct which contains 5K images captured by VANJEE smart base station placed about 4.5m high. The images in this dataset vary in weather and traffic conditions, which are annotated with bounding box annotations of 12 categories.
DETRAC 7 is an intelligent surveillance dataset that contains over 14K images captured by a Canon EOS 550D camera at 24 different locations, covering various traffic patterns and conditions including urhan highway, traffic crossings and T-junctions. The images in this dataset are annotated with bounding box annotations of 4 categories, including car, bus, van, and others.
UAVDT [8] is unmanned aerial vehicle detection dataset, which contains about 80K frames from 10 hours videos. The images in this dataset are captured by unmanned acrial vehicle across various weather conditions (1.e., daylight, night and fog) and multiple camera views (i.e., front-view, side-view and bird-view). This dataset provides bounding box annotations with three categories including car, truck and bus.
B Additional Discussion
Strategy studies for Language Hierarchical Self-training. Our proposed language hierarchical self-training (LHST) introduces WordNet's language hicrarchy [21] to expand the image-level labels and accordingly enables
co-regularivation between the expanded labels and self-training. We examine the superiority of the proposed LHST by comparing it with "Selfftraining" [18] and "Direct WordNet Hierarchy Labeling" (2i). "SSefftraining" is the standard self-training algorithm in [8] while "Direct WordNet Hierarchy Labeling" denotes directly using the expanded image-level labels (by WordNet) for supervisee detection training. reppors
the experimental results, which show that either "Self-training" [18) or "Direct WordNet Hierarchy Labeling" (1 does not perform well. The reasons are: 1) the box-level pseudo labels in "Salintraining" are usually error-prone. making the self-training process unstable and barely improving the performance: 2) the expanded image-level labels 'Direct WordNet Hierarchy Labeling" are not all reliable, training with which leads to unsatisfying performance. Besides, the combination not very well largely because the direct combination of them does not well address their own drawbacks and limitations. On the other hand, our proposed LHST performs better clearly, as shown in the last row of Table To The superior performance of LHST is largely attributed the co-regularization design, which employs self-training to assess and re- re-weight the expanded labels according to the predicted reliability while enabling the expanded (and re- weighted) labels to regularize self-training by providing richer and flexible supervision (the flexible supervision is achieved by the adaptive re- re-weighting operation).
and "Direct WordNet Hierarchy Labeling" still works
Table 10: Strategy studies for Language Hierarchical Self-training on zero-shot cross-dataset object detection over object365 dataset.
Analysis for Language Hierarchical Self- training
AP50
Detic 15
29.4
Self-training 21
29.4
Direct WordNet Hierarchy Labeling 21
29.7
Self-training + Direct WordNet Hierarchy Labeling
29.9
Language Hicrarchical Self- training (Ours)
31.3
Ablation studies of Language Hierarchical Self-training. As mentioned in the main text, our proposed language hicrarchical self-training (LHST) consists of box-leve LHST (i.c., Lroz (F(x)) in Eq.7 in the main
text) and image-level LHST (i.e., Limage (=))) in Eq.8 in the main text). where image-level LHST could regularize box-level LHST. Here we conduct experiments to investigate this. The experimental results in Table show that Box-level LHST brings clear performance improvements while including Image-level LHSTT further improves the detection performance, largely becuase Image-level LHST provides stable supervision to regularize Box-leve LHST. i... Image-level LHST is much more stable as it uses the image-level proposal while the pseudo box labels in not very stable.
LHST (e.g.. the location of pseudo boxes) vary along training iterations and are
17

---

Table 11: Ablation studies of Language Hierarchical Self-training. The experiment setup is
zero-shot cross cross-dataset object detection over Object365 dataset.
Language Hierarchical Self-training (LHST)
Method
AP50
Box-level LHST
Image-level LHST
nox-Supervised [5]
26.5
Detic [15]
29.4
30.5
31.3
Parameter Studies Language Hierarchical Self- training (LHST). In generating pseudo labels in LHST, we filter out a prediction its max confidence score is lower than the threshold We study the threshold t by changing it from 0.65 to 0.85 with a step of 0.05. Table[12] transfer object detection over object365 dataset. We can observe that the detection performance is not sensitive the threshold f.
the experimental results on zero-shot
Table 12: Parameter Studies for Language Hierarchical Self- training (LHST) zero-shot transfer object detection over object365 dataset. We study the thresholding parameter t used in generating pseudo box labels in LHST.
Threshold 0.65
0.70
0.75
0.80
0.85
AP50
31.1
31.3
31.3
31.3
31.2
Analysis of discrepancies of the taxonomies of image vs. box categories. We analyze the mismatch between image-level and box-level categories and how much it could affect the detection performance. As Table [3 shows, the mismatch between image-level and box-level categories varies datasets, and the proposed DetLH improves more with the increase of mismatch levels.
Table 13: Analysis of discrepancies of the taxonomies of image vs. box categories.
ImageNet-21K
Mismatch Ratio
Baseline (AP50)
DetLH (AP50)
A
V.S. Cityscapes
0.13
47.1
50.3
+3,2
v.s. DETRAC
0.25
39.2
44.0
+4.8
V.S. MIO-TCD
0.27
20.6
24.5
+3.9
V.S. African Wildlife
0.50
80.9
87.2
+6.3
Vistas
0.67
35.6
44.0
+8.4
V.S.
Arthropod Detection
0.86
36.7
49.0
+12.3
How effective DetLH deals with noisy labels. We conduct ablation studies to analyze how effective DetLH deals with noisy labels. Specifically, we compare DetLH with without using reliability scores (the latter means uniform category weights) over Object365. As Table [4 shows, including the adaptive weighting mechanism (i.e., reliability scores) helps mitigate the label noises effectively.
Table How effective DetLH deals with noisy labels.
Method
AP50
DetLH without reliability score
31.8
DetLH
32.5
The impact of using proxy vocabulary. We conduct experiments on Object365 dataset to compare LHPG with CLIP embeddings only and LHPG with both CLIP embeddings and proxy vocabulary, As Table shows, using
18

---

proxy vocabulary performs clearly better, demonstrating its effectiveness in narrowing the distribution gap between training and test labels.
Table 15: The impact of using proxy vocabulary.
Method
AP50
Baseline
29.4
LHPG (CLIP embeddings only)
30.0
LHPG (CLIP embeddings + proxy vocabulary)
31.0
Comparisons with other semi-super WSOD methods. We conduct experiments on Object365 datasct to compare our Detl. H and
88
89
90 91.
As Table To shows, Detl H clearly outperforms [a,b,c,]].
Table 16: Comparisons with other semi-supervised WSOD methods.
Baseline
88
(89]
[90]
91
DetLH
AP50
29.4
29.5
29.9
29.8
29.6
32.5
C
Additional Comparison
Comparison with RKD 17]. We note that RKD [7 explores Region-based Knowledge Distillation to better distill knowledge from the CLIP model for weakly-supervised object detection. In this paragraph, we compare DetLH (i.e., the self- training based method) with RKD (i.c., the knowledge distillation-basee based method) on zero-shot cross-dataset object detection over Object365 dataset. Table [ the results, which show that our DetLH clearly outperforms RKD [17. indicating the effectiveness the proposed designs in DetLH for zero-shot cross-dataset object detection.
Table 17: Comparison with RKD 17 on zero-shot cross-dataset object detection over Object365 dataset.
Method
AP
RKD 17
22.3
DetLH (Ours)
23.6
Discussion and comparison with visual grounding- -basd detection methods 93,993. We that GLIP y2 DetCLIP [93 explore extra visual grounding data to train
detector. this paragraph, we compare our DetLH (i.c.. the WSOD-base method) with 992 93 (i.e., the visual grounding- based method) from the perspective of detection efficiency. Table reports the results of run time in millisecond (the run times of GL.IP ( and DetCI.IP (3 are acquired from 9 .). It shows that our DetLH (i.e., the WSOD-based method) runs much
efficient than the visual grounding -hased detection methods (i.c., GLIP (2 and DetCLIP 93), largely hecause the visual grounding-
detection methods [92 93 include a text encoder their networks. Note we did not compare our DetLH (i.e., the WSOD-base basoD-baeed method) with the visual grounding- based method [92 93 from the perspective of detection accuracy becuase these two types of detection methods very different training data, e.g.. [92 and [ visual grounding data. In addition, the WSOD techniques
are basically complementary the visual grounding
detection techniques 92 93 because the visual
grounding data could be used to further improve the WSOD-based detectors 94 Similar to [94), we leave this as our future research.
Other image-level supervision. We follow Detic [5] to build our proposed DetLH. Therefore, similar to Detic [15). our DetLH can also seamlessly incorporate the free-form caption text the image-level supervision, i.e., by using the language embeddings of image captions the detection classifier when training on image-text million image-text pair data into detector training for learning more generalizable object detectors. On the other hand, training over large-scale LAION dataset is caapuutationinnteniiin and thus we leave this as our future work.
pair data [T5]. this way. we could further incorporate LAION dataset [95] that includes
Comparison with other detection methods |92 96
97 93 98 99). We didn't 't compare with these methods [92 93 98. 99 in the main manuscript because they focus different topics with different objectives.
96 97
19

---

prouuddin-bbsed Table 18; Efficiency comparison of WSOD- and visual grounding based detection methods.
Method
Types
Run time (ms)
GLIP 92 w Swin-T
Visual Grounding-basee
8333.3
DetCLIP [3 w Swin-T
Visual Grounding-band
434.7
DetLH w Swin-B (ours)
WSOD
46.0
training data, backbones and benchmarks, Instead, we follow Detic [1] as both our DetLH and Detic belong to and are claimed as
object detection (WSOD), aiming to using large-scale images and classes (i.e., IetgeNet-21 K and LVIS) to train general detector that can work on any detection scenarios.
However, 90 96 97
introduce visual grounding studies how to use grounding data for detection; OWI.-ViT 9 focuses on fine-tuning CLIP with standard
98
99
for WSOD: GLIPP and DetCLIP
p2 [
detection datasets; OmDet and Uni 998 (9) focus on training with multiple detection datasets. We still
managed to benchmark with [92 96, 97 93 98 99] ie.. GLIP [92
GLIPv2 [96), OWL-ViT [97]. DetCLIP (93). various different datasets in
OmDet [98] and UniDet [99]. As our method and 992 96 97
93
98
99
evaluations, the benchmarking below is the shared one, i.c., Pascal VOC (in AP).
GLIP-L GLIPv2-B OWL-VIT DetCLIP
OmDet
UniDet
Ours
61.7
62.8
60.3
56.7
60.8
60.1
64.4
Note Florence [100] is not included as it is foundation model that uses very large backbone (CoSwin-IH H) and very large customized training data (FLD-900M and FLOD-9M), Besides. we compared our DetLH with GLIP [92] and DetCLIP [8] Table 18 which shows DetL. (i.c., the WSOD-based method) much more efficient (about 10 times) than the visual grounding brasdd detection methods (i.c., GLIP and DetCLIP).
We note that WDinW 12 also benchmarks cross-datasct generalization. We benchmark on ODinW and the results below (averaged on 35 datascts in ODinW) show that our DetLH works effectively on ODinW. Note GLIP obtains higher accuracy because it introduces visual grounding and involves Language Encoder in inference. Without those extra modules, our DetLH runs much faster (over 10 times) than
Comparison on ODinW (22..
GLIP as discussed Table [8
WSDDN YOLO9000
DLWL
Detic
GLIP
OmDet
Ours
14.9
16.3
17.1
17.3
19.7
16.0
18.2
Open-vocabulary benchmark. We not benchmark on open-vocabulary LVIS/COCO (both divide single- dataset vocabulary into base and novel classes to mimic and benchmark vocabulary generalization), because our work aims to leverage large-scale images and classes (i.e.. 21K classes) to train general detector that can work any detection scenarios. Cross-datase, generalization benchmark fits this objective better and is more general and challenging than open-vocabulary benchmark that tackles base and novel classes within a single dataset.
Comparison with class hierarchy methods [101. 70 on OpenImages [102.. We benchmark with other methods that use class hierarchy, including [101] and 70 As shown below, our DetL. performs clearly better than hicrarchy-awarc losses in 101 770 due to our designed co-regularization as detailed in the manuscript.
Note use OpenImages V7 (Oet 2022) instead of 2019 version.
WSDDN YOL09000
DLWL
Detic
[101]
[70]
Ours
14.8
15.8
16.2
16.4
16.5
16.5
17.6
Results of WSDDN, YOLO9000 and DLWL. Note Detic implemented WSDDN, YOLO9000 and DLWL and we directly adopted Detic's implementation in evaluations. The gains in our reported results are different as we evaluated more challenging datasets and benchmark: 1) The results in our Table in the main manuscript averaged over 14 datascts while those in Table I of Detic are single dataset LVIS; 2) Our Table 7 in the main manuscript cross-dataset generalization benchmark while Table in Detic is open- opea-vocabulary LVIS.
D Additional Qualitative Result and Comparison
We provide qualitative results of zero-shot cross cross-datase dataset object detection for various detection tasks. As shown in Figures 7 our DetLH produces good detection results consistently across different detection tasks, showing DetLH still works effectively under large cross dass-et gaps in data distribution and vocabulary.
20

---

Figure 3: Qualitative results of DetLH over zero-shot cross-dataset object detection for common
objects. Zoom in for details. Top: Detic [5] Bottom: DetLH (Ours).
Figure 4: Qualitative results of DetLH over zero-shot cross-dataset object detection for autonomous driving. Zoom in for details. Top: Detic [15). Bottom: DetLH (Ours).
Figure 5: Qualitative results of DetLH zero-shot cross-dataset object detection under different
weather and time-of-day conditions. Zoom in for details. Top: Detic 15] Bottom: DetL. (Ours).
Figure 6: Qualitative results of DetLH over zero-shot cross- cross-dataset object detection for intelligent surveillance. Zoom in for details. Top: Detic [5] Bottom: DetLH (Ours).
E
Broader Impacts and Limitations
This work strives for exploiting weakly-supervised object detection (WSOD) to lear generalizable detectors by addressing the image-level label mismatch issue. We propose incorporate image- level supervision with self-training for learning generalizable detectors, aiming benefit from self eeffiigi while effectively making use of image-level weak supervision. Our proposed technique provides great advantages by avoiding the need of massive object-level annotations and allowing learning effective and generalizable detectors with level supervision. It thus makes very valuable contribution the computer vision
Broader Impacts.
21

---

Figure 7: Qualitative results of DetLH over zero-shot cross cross-datase dataset object detection for Wildlife Detection. Zoom in for details. Top: Detic [15]. Bottom: DetLH (Ours).
research community by providing novel and efficient weakly-supervised object detection method. The feature of scaling detectors with image-level labels enables effective and generalizable detectors that could work well in various downstream tasks, broadening the applicability of object detectors significantly.
L Limitations. As discussed in Sections 3 and 4 of the main and Section Dataset and Implementation Details the appendix, our proposed WSOD method adopts ImageNet-21K with image-level labels scale up detectors. It avoids the need of massive object-level annotations and allowing learning effective and generalizable detectors with image-level supervision. At the other end, we could further scale detector training by involving the
recent image-text pair data for WSOD training, which may further improve the performance significantly. We will investigate how involve recent image-text pair data for WSOD training our future work.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately describe the paper's contributions and scope.
Guidelines:
The answer NA means that abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcorctical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discussed the limitations of the work in Section E. of the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors are encouraged create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (e.g.. independence assumptions, noiseless settings, model well rell-specifications asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should he articulated.
The authors should reflect the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
23

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provided detailed instructions for reproducing the main experimental results in Section 3 Method and Section 4 Experiment including the details of the proposed framework, and the datasets, base models and the parameters used for experiments in Section Dataset and Implementation Details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example. if the contribution is novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility. which may depend on the nature of the contribution. For example
(a) If the contribution is primarily reproduce that algorithm.
algorithm, the paper should make clear how
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (c.g.. with open- (d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-soune models, it may be that access to the model is limited in some way (e.g.. registered users). but it should be possible for other researchers have some path reproducing or verifying the results.
datasct instructions for how construct the dataset).
5. Open access to data and code
Question: Does the paper provide open faithfully reproduce the main experimental results, as described supplemental material?
the data and code. with sufficient instructions
Answer: [No]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see NeurlPS code and data submission guidelines http::://niis.cccppbb//
guides/Codens https//ooooSuuiiiiooonoe
for more details.
While we encourage release of code and data, we understand that might not possible,
so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g.. for a new open-sourge benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: : ://nis/ cc/public/ qu..dess/oooeSSutnlessssooooo... for more details.
24

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from script and why.
At submission time, preserve anonymity, authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommeneed, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provided the detailed implementation details in Section Dataset and Implementation Details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropriate informa- tion about the statistical significance of the experiments?
Answer: [NA]
Justification:
We conducted the experiments with multiple runs and did not observe clear variance.
Guidelines:
The answer means that paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful symmetric error bars would yield results that are out of range (e.g. negative error rates). error bars reported tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
show in tables figures
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: We provided sufficient information on the computation resources required for reproduce the experiments in Section Dataset and Implementation Details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
25

---

The paper should provide the amount of compute required for each of the individual experimental well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer:
[Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is special consideration due laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed the broader impacts of the work in Section E of the Appendix.
Guidelines:
The answer NA means that there is no socictal impact of the work performed.
the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system Iearns from feodback over time, improving the efficiency accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, scraped datascts)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions to access the model implementing safety filters.
26

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly eredited and are the license and terms of usc explicitly mentioned and properly respected?
Answer: [Yes]
Justification: properly credited the original owners of assets used in the paper and properly respect their license and terms of use.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperewi thcode. cwm/ca////aa//oc daaa/dattstt has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing research with human subjects.
Guidelincs:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least minimum wage in country of data collector.
15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human Subjects
27

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country which research is conducted, approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
28


Reviews:
[{"review_id": "ZSHrglmQif", "reviewer": "Reviewer_TMzm", "summary": "* The authors propose to expand the training data for object detection using classification datasets through two main contributions:\n\n1. Combine a language hierarchy with self-training to extend the training dataset, while minimizing label noise by re-weighting categories with reliability scores. \n2. Bridge the vocabulary gaps between training and testing using a vocabulary proxy (WordNet) improving zero-shot learning.\n\n* Extensive experiments in multiple datasets are provided showing an improvement over the state-of-the-art.", "questions": "* Futher analysis and validation of main assumption. How do the category taxonomy levels differ across datasets?\n* Where does the performance gain come from? Robustness to label noise or vocabulary proxy?\n* In general in the experimental section I miss more insight and discussion on results, experimental setup and data setup. I feel that most discussion repeatedly states the obvious (performance increases due to leveraging larger amouts of data). Whenever more specific insight is provided, it is very speculative. For example, authors state that performance gains in Wildlife are due to the \"introduction of the language hierarchy\", but there are no specific experiments designed to proof such claim.", "limitations": "Yes, the limitations and potential negative impact are addressed.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.The authors clearly exposed their ideas and the paper is reasonably well written. \n\n2.Even though using a language hierachy to improve object recognition is an old idea, [Marszalek, et al. Semantic Hierarchies for Visual Object Recognition] it's appealing to see it applied to a modern, large-scale, object detection setting. Moreover, the combination with self-training is original and well justified.\n\n3.The results show a significant improvement over the state-of-the-art.\n\n4.The evaluation is very extensive and performed over an impressive quantity of diverse datasets.", "weaknesses": "1. The experimental section and ablation do not clearly show if the authors' assumptions are correct.\n* The idea of the paper heavily relies on the assumption that there are discrepancies on the taxonomy levels of the categories of image and detection datasets. This assumption may hold for some datasets but not for others and it should have a significant impact on the method's results. I miss a section presenting an analysis of the taxonomies of image vs box categories, and how it may impact peformance.\n\n2. Discussion on the specific impact of each contribution to the final result is lacking\n* The ablation study (Section 4.2) does not seem to add much value to the experimental discussion. First, showing the performance increase over the baseline [68] seems misleading when the authors compare DetLH against only-box supervision, since the work of [68] already proposes to expand detector (boxed) training data with image-classification data. The work of [68] already demonstrated that adding image-based training data improves detection results, and it should not be the goal of the ablation section to show this. A far more interesting baseline could have been to train DetLH with uniform category weights to demonstrate how DetLH deals with noisy labels. \n* Another important ablation experiment would be to analize the impact of narrowing the gap between training and testing label distributions. The CLIP embeddings would already help in this regard, but how impactful is the use of a proxy vocabulary on top of it is not clearly quantified. \n\n3. Wording is sometimes ambiguous or could lead to contribution misunderstandings:\n* For instance the authors say \"These detectors usually yield constrained detection peformance when trained only with detection datasets... [68]\", when the work of [68] is actually trained with both detection and classification data. If the authors want to point out that [68] already suggested this, it should be more explicitly stated. Even more so, if it's followed with a claim that the current method overcomes such limitations: \"DetLH introduces large-scale image-level datasets to enlarge the data [...] in detector training.\", which is misleading as this is not something the present work introduces, but the work of [68].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Response 1] Analysis of discrepancies of the taxonomies of image vs. box categories:**\n\nThank you for pointing out this issue! As suggested, we analysed the mismatch between image-level and box-level categories and how much it could affect the detection performance. As the table below shows, the mismatch between image-level and box-level categories varies across datasets, and the proposed DetLH improves more with the increase of mismatch levels. We will include a subsection to discuss this issue in the updated paper.\n\n|ImageNet-21K |Mismatch Ratio| Baseline (AP50) | DetLH (AP50) |  $\\Delta$ |\n|:----:|:----:|:----:|:----:|:---:|\nv.s. Cityscapes\t|0.13 \t|47.1\t|50.3\t|+3.2|\nv.s. DETRAC\t|0.25 \t|39.2\t|44.0\t|+4.8|\nv.s. MIO-TCD\t|0.27 \t|20.6\t|24.5\t|+3.9|\nv.s. African Wildlife\t|0.50 \t|80.9\t|87.2\t|+6.3|\nv.s. Vistas\t|0.67 \t|35.6\t|44.0\t|+8.4|\nv.s. Arthropod Detection|\t0.86 \t|36.7\t|49.0\t|+12.3|\n\n**[Response 2] More ablation studies:**\n\nThank you for your comment! As suggested, we conduct new ablation studies to analyse how effective DetLH deals with noisy labels. Specifically, we compare DetLH with and without using reliability scores (the latter means uniform category weights) over Object365. As the table below shows, including the adaptive weighting mechanism (i.e., reliability scores) helps mitigate the label noises effectively.\n\n| | AP50| \n|:----:|:----:|\nDetLH without reliability score | 31.8|\nDetLH | 32.5 |\n\nRegarding the impact of using proxy vocabulary, we conduct new experiments on Object365 dataset to compare LHPG with CLIP embeddings only and LHPG with both CLIP embeddings and proxy vocabulary. As the table below shows, using a proxy vocabulary performs clearly better, demonstrating its effectiveness in narrowing the distribution gap between training and test labels.\n\n| | AP50| \n|:----:|:----:|\nBaseline | 29.4|\nLHPG (CLIP embeddings only) | 30.0 |\nLHPG (CLIP embeddings + proxy vocabulary) | 31.0 |\n\n**[Response 3] Misleading Claims:**\n\nThank you for your suggestion! To avoid confusion, we will rephrase the text \"These detectors usually yield constrained detection... [68]\" to \"As reported in [68], detectors trained solely on detection datasets often exhibits constrained performance.\", and the claim \"DetLH introduces large-scale image-level datasets  [...].\" to \"Similar to [68], DetLH follows semi-supervised WSOD and introduces large-scale image-level datasets for detector training.\".\n\n**[Response 4] More insights and analysis should be provided:**\n\nThank you for pointing out this issue. We would clarify that we provided extensive discussion and related insights in the appendix (e.g., Section B Additional Discussion, and Section C Additional Comparison), including more ablation studies, strategy studies, parameter studies, comparisons, etc. In addition, we benchmark DetLH across multiple datasets and discuss its generalization ability and parameter sensitivity In Section 4.1 Comparison with the SOTA and Section 4.3 Discussion. We will revise the two Sections to remove repetitive text and include some insight analysis from the appendix. \n\nRegarding Wildlife datasets, our statement is largely based on the intuition that these datasets with many fine-grained categories should suffer from clear vocabulary mismatch, and the language hierarchy introduced by DetLH should help mitigate such mismatch and improve the detection. The new experiments in Response 1 show that wildlife datasets do suffer from more severe mismatch and DetLH helps mitigate the mismatch effectively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DldKBsNP2l", "reviewer": "Reviewer_PjbE", "summary": "This paper focuses on scaling the detectors’ vocabulary with image-level weak supervision. To better leverage the image-level labels in this task, this paper introduces a language hierarchical self-training (LHST) framework that incorporates language hierarchy (i.e., WordNet) with self-training. In addition, the authors introduce a language hierarchical prompt generation (LHPG) method enabling open-vocabulary object detection. The proposed method outperforms baselines across 14 commonly used benchmarks. Comprehensive ablation experiments validate the effectiveness and generalization of the proposed method.", "questions": "Please refer weakness.", "limitations": "The authors have discussed the limitations, and this works does not show potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ This paper is well organized and written. The overall paper is easy to follow and technically sound.\n+ On extensive benchmarks, the proposed method achieves non-trivial performance improvement and shows strong generalization.", "weaknesses": "- The necessity of the investigated problem (i.e., scaling the vocabulary of the detectors with image labels) should be further stated, as the recent MLLM-based detection/grounding methods inheritly have strong zero-shot capabilities.\n- The proposed method is somewhat trivial and heuristic, where only the usage of WordNet is new and interesting. Most techniques are from existing WSOD and semi-supervised object detection methods. It would be better to highlight the novelty. \n- Several semi-supervised WSOD methods trained with both instance- and image-level annotations are missing. It would be better to review them for completeness. \n[a] UniT: Unified Knowledge Transfer for Any-Shot Object Detection and Segmentation, CVPR’21\n[b] Cyclic Self-Training with Proposal Weight Modulation for Cross-Supervised Object Detection, TIP’23\n[c] H2FA R-CNN: Holistic and Hierarchical Feature Alignment for Cross-Domain Weakly Supervised Object Detection, CVPR’22\n[d] DOCK: Detecting Objects by transferring Common-sense Knowledge, ECCV’18\n- In Line 149, the authors claimed ‘a single-label annotation could be expanded into a multi-label annotation’. It would be better to give some visualization examples. In Line 164, authors claimed ‘the labels expanded by WordNet are not all reliable’. Giving some examples would be helpful to improve the readability.\n- Why only the AP results of oracles are reported on Object 365?\n- How about the efficiency of WordNet in training and inference?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Response 1] The necessity of the investigated problem:**\n\nThank you for your suggestion. We would clarify that, compared with recent MLLM-based detection/grounding methods, our approach of scaling the vocabulary of detectors using image labels offers significant advantages in training efficiency. As shown in Table 14 (as copied and posted below), DetLH is up to ten times more efficient than current visual grounding/MLLM-based methods such as GLIP and DetCLIP. The superior efficiency is critical to the scalability of detectors in many practical tasks.\n\n\n| Method | Types | Run time (ms)|\n|:----:|:----:|:----:|\nGLIP w Swin-T |Visual Grounding-based |8333.3|\nDetCLIP w Swin-T |Visual Grounding-based |434.7 |\nDetLH w Swin-B (ours) | WSOD |46.0|\n\n**[Response 2] The novelty of the proposed method:**\n\nThank you for your suggestion! We would share that, different from existing WSOD and semi-supervised WSOD methods, DetLH tackles the image-to-box label mismatch by introducing language hierarchy into self-training: 1) it introduces WordNet's language hierarchy to expand the image labels and accordingly enables co-regularization between the expanded labels and self-training. As shown in Table 10 (in the submitted manuscript), just introducing language hierarchy or simply combing self-training and language hierarchy does not work without our co-regularization design; 2) it introduces language hierarchy into prompt generation to bridge the vocabulary gaps between detector training and testing, leading to better prompt generation and detection performance. Extensive experiments validate the effectiveness of our designs in DetLH. We will highlight the above points in the updated manuscript.\n\n**[Response 3] Comparisons with other semi-supervised WSOD methods:**\n\nThanks for sharing [a,b,c,d]. We will review them and highlight how DetLH differs in the revised paper. As the table below shows, DetLH clearly outperforms [a,b,c,d] on Object365.\n\n| | Baseline | [a] | [b] | [c] | [d] | DetLH|\n|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n|AP50 |29.4 |29.5 |29.9|29.8|29.6|**32.5**|\n\n\n**[Response 4] Visualizations of label expanding:**\n\nThank you for your suggestion! **Figure 1** in the attached PDF shows that the labels expanded by WordNet are not all reliable. We will include the suggested visualization in the updated manuscript or appendix.\n\n**[Response 5] Missing oracle values on Object365 in Table 1:**\n\nWe reported AP value only for Object365 as we referenced to the Detic paper which only reports AP value. For Pascal VOC, we trained the model to get the oracles of all AP metrics. We will train models on Object365 to obtain the complete AP values and update them in the updated manuscript.\n\n**[Response 6] The efficiency of WordNet in training and inference:**\n\n|  | Training Speed (second/image) | Inference Speed (second/image)|\n|:----:|:----:|:----:|\n|Detic (w/o WordNet language hierarchy)| 0.7009 | 0.045|\n| Our DetLH |0.7110| 0.046|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6SybreEl55", "reviewer": "Reviewer_VMio", "summary": "The paper proposes a novel method for leveraging image-level annotations for object detection pretraining, specifically for zero-shot and open vocabulary detection settings. The proposed method, DetLH, leverages self-training to generate pseudo-labelled object proposals, and then adjust the pseudo-label for each box according to the image's class and the classes in WordNet that are hierarchically related to it. Furthermore, the authors propose a method for prompt generation that does not require additional training. The proposed method is evaluated extensively and achieves state-of-the-art results in relevant zero-shot and open-vocabulary object detection benchmarks.", "questions": "My primary concern regarding the paper is related to the evaluations: a) Whereas the paper is a clear improvement over Detic, it avoids extensive comparisons with more recent methods (Detic being a ECCV 2022 paper) such as the ones included in the paper (L661-671), and b) evaluations with LVIS, a shared benchmark for most relative papers, are not conducted.\n\nOverall, however, I am very satisfied with the paper's contribution and I believe that the conducted experiments adequately demonstrate the proposed method's effectiveness. My main reason for not giving a higher rating is because LVIS/COCO results are not included, which would be informative and are significant to facilitate comparisons with existing and future relevant works.", "limitations": "The authors have adequately addressed the relevant limitations.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well written and its key ideas clearly communicated.\n2. The proposed method is well presented and novel, and constitutes an improved way to leverage labelled object centric images to facilitate object detector training.\n3. The conducted experiments are extensive and demonstrate that the proposed method consistently achieves state-of-the-art performance in various settings and datasets.", "weaknesses": "1. Unless I am mistaken, there is no ablation study related to the impact of the reliability score. I would be very interested to see how it affects performance.\n2. While justifications are provided in the appendix, I believe more comparisons should have been included with relative methods (L661-671). More importantly, I believe that the ommision of evaluations with LVIS/COCO is a significant issue, and I am not convinced by the authors stated reason (L683-687) that they focused on better suited datasets. The paper has extensive experiments with many datasets, and it is confusing to me why they chose to leave out what is arguably the most common dataset for detection tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Response 1] The affect of the reliability score:**\n\nThank you for your suggestion. As suggested, we conduct the new ablation study to examine the effect of the reliability score in the proposed DetLH. As the table below shows (on Object365), DetLH effectively mitigate the label noises with the design of reliability score.\n\n| | AP50| \n|:----:|:----:|\n|DetLH without reliability score | 31.8|\n|DetLH | 32.5 |\n\n**[Response 2] More comparisons:**\n\nWe would clarify that we did not provide much comparisons with those methods as mentioned in Lines 661-671 because they focus on different tasks with different objectives (also use different training data, backbones and benchmarks). We largely focus on benchmarking DetLH with Detic as both methods work on the same task WSOD and use the same training data (ImageNet-21K and LVIS) and backbone. As suggested, we will provide more comparisons between DetLH and those detectors and include the comparisons in the updated manuscript or appendix.\n\nIn addition, we would clarify that we focus on cross-dataset zero-shot evaluations [48, 68] (i.e., evaluation on unseen datasets like CLIP) as discussed in Lines 117-119. We did not benchmark on LVIS/COCO (both use the same data but different annotations) since we used LVIS/COCO in network training. \n\nAs suggested, we benchmark DetLH on LVIS and COCO as shown below and will include these results in the updated manuscript or appendix.\n\n|LVIS Dataset | AP50| \n|:----:|:----:|\n|MosaicOS  |28.3 |\n|CenterNet |34.9 |\n|AsyncSLL |36.0 |\n|SeesawLoss  |37.3  |\n|Baseline |40.7 |\n|Detic |41.7 |\n|**DetLH** | **42.2** |\n\n\n|COCO Dataset | AP50| \n|:----:|:----:|\n|Baseline |39.3 |\n|Self-training |39.5 |\n|WSDDN |39.9 |\n|DLWL |42.9 |\n|Detic |44.7 |\n|**DetLH** |**45.3** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for your positive comments on our work. In particular, it is encouraging that the reviewers acknowledge that 1) our work is novel and appealing [TMzm,VMio]; 2) the proposed method is effective [TMzm,PjbE,VMio]; 3) the evaluation is extensive [TMzm,PjbE,VMio] and 4) the paper is well written [TMzm,PjbE,VMio].

We address the questions and concerns raised by each reviewer point-by-point in the respective threads. Additionally, attached please find a PDF file that contains the figure that is related to the rebuttal.

Meta-review: This submission proposes to expand the training data for object detection using image-level weak supervision from classification datasets, and leverages a word ontology (wordnet).

Reviewers agreed that:
1. The paper was generally well-written and easy to understand
2. The proposed method yields meaningful improvements over SOTA.

There were initial concerns about:
1. The thoroughness of ablations and experimental analysis (including suggestions to move some experiments to the main section)
2. The proposed method being somewhat trivial.

Both concerns were addressed during the discussion period and all 3 reviews ultimately recommend weak accept or accept. The AC sees no reason to override this unanimous recommendation. The authors are asked to incorporate the requests from reviewers into the final version of the paper.
Decision: Accept (poster)