GOOD: Exploring geometric cues for detecting objects in an open world
Authors: Haiwen Huang
Andreas Geiger
Dan Zhang
Venue: ICLR 2023 poster (2023)
Keywords: open world
object detection
geometric cues
mid-level representations

Abstract: We address the task of open-world class-agnostic object detection, i.e., detecting every object in an image by learning from a limited number of base object classes. State-of-the-art RGB-based models suffer from overfitting the training classes and often fail at detecting novel-looking objects. This is because RGB-based models primarily rely on appearance similarity to detect novel objects and are also prone to overfitting short-cut cues such as textures and discriminative parts. To address these shortcomings of RGB-based object detectors, we propose incorporating geometric cues such as depth and normals, predicted by general-purpose monocular estimators. Specifically, we use the geometric cues to train an object proposal network for pseudo-labeling unannotated novel objects in the training set. Our resulting Geometry-guided Open-world Object Detector (GOOD) significantly improves detection recall for novel object categories and already performs well with only a few training classes. Using a single ``person'' class for training on the COCO dataset, GOOD  surpasses SOTA methods by 5.0% AR@100, a relative improvement of 24%. The code has been made available at https://github.com/autonomousvision/good.

Published as a conference paper at ICLR 2023
GOOD: EXPLORING GEOMETRIC CUES FOR DETECT- ING OBJECTS IN AN OPEN WORLD
Haiwen Huang!. Andreas Geiger Dan Zhang 1,4
1 Bossh Lab, University of Tübingen
2 *Autonomous Vision Group, University of Tübingen Tübingen AI Center "Bosch Center for Artificial Intelligence
(haiwen. huang, a.geiger) Gunn ceeeeennnnttttttte tuebingen. de, Dan. 2 www.bbs..c.....o... com com
(a) Ground truth (20)
(b) OLN (13)
(c) GGN (14)
(d) GOOD (18)
(c) Ground truth (22)
(f) OLN (6)
(g) GGN (6)
(h) GOOD (15)
Comparison of GOOD with different baselines. Images in the first column are from validation sets of ADE20K (Zhou et al. 2019). From the second to fourth columns we show the detection results of three open-world object detection methods: OLN Kim al. (202I). GGN Wang et al. (002)), and our soometrry-gidee Open-world Object Detector (GOOD). The shown detection results are true- pos-iosiieve proposals from the top 100 proposals of each method. The numbers of true positive proposals or ground truth objects are denoted in parentheses. All models are trained on the RGB images from the PASCAL- VOC classes of the COCO dataset (Lin al. (014), which do not include houses, trees, or kitchen furniture. Both OLN and GGN fail to detect many objects not seen during training. GOOD generalizes better to unseen categories by exploiting the geometric cues.
Figure 1:
ABSTRACT
We address the task of open-world class-agnostic object detection, i.e., detecting every object in an image by learning from a limited number of base object classes. State-of-the-an RGB-based models suffer from overfitting the training classes and often fail at detecting novel-looking objects. This is because RGB-based mod- els primarily rely on appearance similarity to detect novel objects and are also prone to overfitting short-cut cues such as textures and discriminative parts. To address these shortcomings of RGB-based object detectors, we propose incorpo- rating geometric cues such as depth and normals, predicted by general-purpose monocular estimators. Specifically, we use the geometric cues to train an ob- ject proposal network for pseudo-labeling unannotated novel objects in the train- ing set. Our resulting Geometry-guided Open-world Object Detector (GOOD) significantly improves detection recall for novel object categories and already performs well with only a few training classes. Using a single "person" class for training on the COCO dataset, GOOD surpasses SOTA methods by 5.0% AR @ 100, a relative improvement of 24%. The code has been made available at https:: //g thub. wtt.bb/cwm/aatr.ooooooovviiiiaea

---

Published as a conference paper at ICLR 2023
Non-VOC
0
60%
56
AR8100
50%
40%
Novel
OLN GOOD-D GOOD-N
Figure 2: Geometry cues are complementary to appearance cues for object localization. The depth and normal cues of the RGB image are extracted using off-the-shelf general pprrpss monoc- ular predictors. Left: Geometric cues abstract away the appearance details and focus on more holis- tic information such as object shapes and relative spatial locations (depth) and directional changes (normals), Right: By incorporating geometric cues, GOODs generalize better than the RGB-based model OLN Kim et al. 202I), i.e. much smaller AR gaps between the base and novel classes.
INTRODUCTION
The standard object detection task is to detect objects from predefined class list. However, when deploying the model in the real world, it rarely the case that the model will only encounter objects from its predefined taxonomy. In the open-world setup, object detectors required to detect all the objects in the scene even though they have only been trained on objects from a limited num- ber of classes. Current state-of-the-art object detectors typically struggle in the open-world setup. As a consequence, open-world object detection has gained increased attention over the last few years (Jaiswal et al., [021]] Kim et al. [202]] Joseph et al., 2021; Wang et al.) [2022). In this work, we specifically address the task of open-world class-agnostic object detection, which is a funda-
mental task for downstream applications like open-world world multi-object tracking (Liu et al. 2022
robotics (Jiang et al.) 2019), and autonomous Al agents (Liu et al. (201)).
One reason for the failure of current object detectors in the open- world setting is that during training, they are penalized for detecting unlabeled objects in the background and are thus discouraged from detecting them. Motivated by this, previous works have designed different architectures (Kim al,,
2021 Konan et al.] 2022) and training pipelines (Saito et al., 2021 Wang et al.) 2022) to avoid suppressing the unannotated objects the background, which have led to significant performance improvements. However, these methods still suffer from overfitting the training classes. Training only on RGB images, they mainly rely on appearance cues to detect objects new categories and have great difficulty generalizing to novel-looking objects. Also, there known short-cut learning problems with regard to training on RGB images Geirhos et al.) (2019) 2020); [Suuer & Geiger (2021) there no constraint for overfitting the textures the discriminative parts of the known classes during training. In this work, we propose to tackle this challenge by incorporating geometry cues extracted by seneral-purposs monocular estimators from the RGB images. We show that such cues significantly improve detection recall for novel object categories on challenging benchmarks.
Estimating geometric cues such depth and normals from single RGB image has been an active research area for a long time. Such mid-level representations possess built-in invariance to many changes (e.g., brightness, color) and are more class-agnostic than RGB signals, see Figure 2 In other words, there is less discrepancy between known and unknown objects in terms of geometric cues. recent years, thanks to stronger architectures and larger datasets (Ranftl et al.) 202 2022 Eftckhar et al.) (021). monocular estimators for mid-level representations have significantly advanced in terms of prediction quality and generalization to novel scenes. These models are able to compute high-quality geometric cues efficiently when used off-the-shelf as pre-trained models new datasets. Therefore, becomes natural to ask if these models can provide additional knowledge for current baseb open- open-world world object detectors to overcome the generalization problem.
In this paper, we propose to use a pseudo-labeling method for incorporating geometric cues into open-world object detector training. We first train object proposal network the predicted
2

---

Published as a conference paper at ICLR 2023
depth or normal maps to discover novel unannotated objects in the training set. The top-ranked novel object predictions are used as pseudo boxes for training the open-world object detector on the original RGB input. We observe that incorporating the geometry cues can significantly improve the detection recall of unseen objects, especially those that differ strongly from the training objects, as We speculate that this is due to the complementary nature of geometry
shown in Figure [ and [ cues and the RGB-based detection cues: the geometry cues help discover novel-loking objects that RGB-based detectors cannot detect, and the RGB-based detectors can make use of more annotations with their strong representation learning ability to generalize to novel, unseen categories.
Our resulting Gecoetry-guiides Open- Object Detector (GOOD) surpasses the oute-of-the- of-the-art performance on multiple benchmarks for open open-world class-agnostic object detection. Thanks to the rich geometry information, GOOD can generalize to unseen categories with only few known classes for training. Particularly, with a single training class "person" on the COCO dataset (Lin jt aall||2011 2014), GOOD can surpass SOTA methods by 5.0% AR@ 100 (a relative improvement of 24%) on detecting objects from non-person classes. With 20 PASCAL-VOC classes for training, GOOD surpasses SOTA methods even by 6.1% AR@ 100 in detecting non-VOC classes. Furthermore, we also analyze the advantages of geometric cues and show that they are less sensitive to semantic shifts across classes, and are better than other strategies for improving generalization.
2 RELATED WORK
Open-world class-agnostic object detection is the task of localizing all the objects in an image by learning with only a limited number of object classes (base classes). The core problem with standard object detection training is that the model trained to classify the unannotated objects as background and thus is suppressed to detect them at inference time. To solve this issue, Kim jet al.] (202T) proposed object localization network (OLN), which replaces the classification heads of Faster RCNN (Ren et al. 2015) with class-agnostic objectness heads that the training loss is only calculated on positive samples, i.e., known objects, and thus not suppressing the detection of unannotated novel objects. [Sato et al. (2021) synthesized a training set by sopy-pasting known objects onto synthetic backgrounds. However, the model struggles with the synthetic- to-rell domain gap in solving the object detection task. Besides background non-suppression, a more proactive approach is to exploit unannotated objects for training. Wang et al. (2022) built upon traditional learning-fre free methods and developed pairwise affinity predictor to discover unannotated objects. Their object detector, GGN, is then trained using the newly-discovered object masks and ground truth base class annotations as supervision. Finally, another promising direction is to use open- world knowledge from large pretrained modal models. Recently, Minderer et al. (2022); |Maz et al. (2022) made use of pretrained language- vision model (Radford et al.) 2021) objects using text queries. Our work is most related to OLN and GGN. We used the OLN architecture and training loss, but additionally incorporated geometry cues through our pseudo-labeling method. GGN used the pairwise affinity for pseudo labeling. However, since the pairwise affinity is trained on RGB inputs using the base class annotations, GGN still suffers from the overfitting problems of RGB-based methods. Our experiments showed geometric cues better source of pseudo boxes.
detect open- world
Incorporating geometric cues for generalization. The estimation of geometric cues has been an active research area for decades. With the introduction of deep neural networks, the seminal work (2015) significantly improved over early works (Hoiem)
by [igen et al. (2014); Eigen & Fergus
al. 2005a by 2007 2008 Saxena et al. 2005 2008a b). Recent progress in estimating geomet- ric cues can be attributed to the use of modern architectures (Ranfil et al., 2022). stronger training strategies (Zamir et al. 2020) and large-scale datasets (Eftekhar et al. (021). In particular, Om- nidata (Eftekhar et al. 2021) has made significant headway in prediction quality and cross-dataset generalization. Since geometric cues abstract away the appearance details and retain more holistic information about the objects, such as shapes. they have been incorporated into many applications for generalization. For example, Xiang et al. (2022) incorporated them into the 3D shape completion pipeline to generalize to novel classes. Yu et al. (2022) used them to guide the optimization of neural implicit surface models for tackling scenes captured from sparse viewpoints. Chen et al. (2021) ap- plied mid-level visual representations to reinforcement training and gained robustness under domain shifts. In this work, we propose to incorporate geometric cues through ascudo-labeling and also demonstrate large performance gains open-world class-agnostic object detection benchmarks.
3

---

Published as a conference paper at ICLR 2023
Phase-1
Phase-ll
Input: RGB image with known class boxes
Merge bounding boxes
Known class
boxes
Proposal Network
Generate pscuto boxes
RGB image with peeudo boxes
image
deplh
Object Detector
Protrained Omnidata model
normal
Figure 3: Overview of the geometry-guided pseudo labeling method. It consists of two training phases. Phase the RGB input firstly preprocessed by the off-the-shelf model to extract the ge- ometry cues, which are then used to train object proposal network with the base class bounding box annotations. The proposal networks then pseudo-label the training samples, discovering unan- notated novel objects. The top-ranked pseudo boxes added the annotation pool for Phase II training, i.e., a class-agnostic object detector is directly trained on the RGB input using both the base class and pseudo annotations, At inference time, only need the model from Phase II.
METHOD
Our goal is to incorporate geometric cues for improved open-world class-agnostic object detec- tion performance. Concretely, we propose a pseudo labeling method, which can effectively utilize the geometric cues to detect unannotated novel objects in the training set and then them for training the object detector. See the overview of our method in Figure
3.1 OPEN-WORLD CLASS-AGNOSTIC OBJECT DETECTION PROBLEM
Current state-of-the-art object detection methods work well under the closed-world assumption. They are trained with set of object bounding box annotations {t,} from a pre-specified list K. of semantic classes, i.c., the base classes. At test time, their generalization is evaluated by detecting the objects from the known base classes. The standard training loss of object detection for an image has two parts: classification loss and bounding box regression loss
C(I)
Nela EedssPe
1
1
Creg t7),
Nreg iEEi
where is the index of an anchor from the candidate set B, {pe, t.) are the predicted label and bounding box coordinates, and [p] are the corresponding ground truths. Nets is the total number of anchors in the candidate set B. Nreg is the size of the subset Bx, which only contains the anchors with the ground truth label p; = 1. Note P. equals 1 only when the anchor can be associated to an annotated object bounding box from the known classes K; otherwise equals 0 (background).
From the closed-worid to the open-world setup, the generalization goal extends to localizing every object the image, which can belong to an unknown novel class u Utd. Under the training loss in (T), an unannotated object will be classified "background". As result, the model will treat similar types objects as "background" at inference time. To avoid suppressing the detection of novel objects in the background, Kim et al. (202T) proposed replace the classification loss in (1) by the objectness score prediction loss, yielding
(I Nreg iEBx
Σ
Σ (t, + Nreg LEBK
1
(ot of),
(2)
4

---

Published as a conference paper at ICLR 2023
where 0₁ and of the anchors with p are involved in training, completely removing any "background" prediction. At inference time, the objectness score is used to rank the detections. However, since these anchors only capture the annotated objects from the base classes, this loss modification cannot effectively mitigate the overfitting to the base classes. We further resort to adding additional "objects" into training, especially novel ones with very different appearances than objects from the base classes.
the predicted objectness score and its ground truth of anchor i. In doing so, only
3.2 EXPLOITING GEOMETRIC CUES
Models trained on RGB images tend to over-rely on the appearance cues for object detection. There- fore, is hard for them to detect novel objects that appear very differently from the base classes. For instance, model trained on cars is likely to detect trucks, but unlikely to also detect sand- wiches. Involving such novel objects, e.g., food, into training is then an effective way mitigate the appearance bias towards the base classes, e.g., vehicles. To this end, we exploit two types of geometric cues, i.e., depth and normals, for detecting unannotated novel objects in the training set, see some examples in Figure 2 Both of them are common geometric cues that capture local infor- mation. Depth focuses on the relative spatial difference of objects and abstracts away the details on the object surfaces. Surface normals focus the directional difference and remain the same flat surfaces. Compared with the original RGB image, they discard most appearance details and focus on the geometry information such object shapes and relative spatial locations. Models trained with them can thus discover many novel-looking objects that RGB-based ones cannot detect.
We use off-the-shelf pretrained models to extract geometric cues. Specifically, models (Eftekhar et al., 2021) trained using cross-task consistency (Zamir et al. 2020) and 2D/3I) data augmentations (Kar al. 2022). The training dataset for the models is the Omnidata Starter Dataset (OSD) (Eftekhar et al. 2021 which contains 2200 real and rendered scenes. Despite the
Omnidata
difference between the OSD to the object detection benchmark datasets, the Omnidata model can produce high-quality results, implying that the invariances behind these geometric cues are robust.
3.3 PSEUDO LABELING METHOD
To use the geometric cues to discover unannotated novel objects in the training set, we first train an object proposal network on the depth or normal input using the same training loss in (2). i.e, P training in Figure 3 Then, this object proposal network will pseudo-label the training images using its detected bounding boxes. After filtering out the detected bounding boxes which overlap with the base class annotations, we then add the remaining top-k boxes to the ground truth annotations. Here à E 2,3 determined for each detector on a small holdout validation set. Finally, we train new class-agnostic object detector using the RGB image input and the extended bounding box annotation pool as ground truth, i.c., Phase-II in Figure The training loss
LOooD
1
Σ
Frog (Li, (2) Nreg &EBRUBN
Σ
Lorgloi, of).
Nreg
LEBRUBN
Compared with (2), the anchors that overlap with the pseudo boxes of the detected novel objects, ie.,, E By, also involved in training. The pseudo boxes can be acquired from single source, i... one of the geometric cues, and from both, i.e., pseudo label ensembling. We name our method GOOD-X when using specific geometric cue X the pseudo labeling source, whereas GOOD- Both stands for ensembling the pseudo labels from both the depth and normals.
Inspired by previous works in self-training (Xie et al. 2020; Sohn al.) 2020 Xu et al. (021), use strong data augmentation during Phase-II to counteract the noise in pseudo boxes and further boost the performance of GOOD. Specifically, for Phase-II training, we use AutoAugment (Cubuk et al. 2019) which includes random resizing, flip, and cropping.
4 EXPERIMENTS
Benchmarks, We target two major challenges of open- world class-agnosd object detection: cross- category and cross dataset generalization. For the cross-category evaluation, we follow the common practice in the literature (Kim al. 2021 Wang et al. 2022) to split the class category list into two
5

---

Published as a conference paper at ICLR 2023
parts. One is used as the base class for training, whereas the other is reserved only for testing cross- category generalization. Specifically, we adopt two splits of the 80 classes in the COCO dataset (Lin et al. 2014). The first benchmark splits the COCO classes into single "person" class and 79 non- person classes. This is to stress-test the generalization ability of the methods. We follow Wang et al. (2022) to choose the "person" category the training class because it contains diverse viewpoints and shapes. The second benchmark splits the COCO classes into 20 PASCAL-VOC (Everingham al. 2010) classes for training and the other 60 for testing. For the cross-dataset evaluation, we use the ADE20K dataset (Zhou et al. 2019) for testing. We compare models trained using only 20 PASCAL-VOR classes or all 80 COCO classes on detecting objects in ADE20K. This is to evaluate open-world class-agnostic object detectors when used in the wild.
Implementation. We use the same architecture as OLN in (Kim et al. 2021 for both Phase and Phase II training. OLN is built on top of a standard Faster RCNN (Ren et al. 2015)) architecture with a ResNet-50 backbone pretrained on ImageNet (Deng et al. 2009). We implement our method using MMDetection framework (Chen et al. (2019) and use the SGD optimizer with an initial learning rate of 0.01 and batch size of 16. The models with data augmentation are all trained for 16 epochs. Other models are trained for 8 epochs. We did not find training for longer epochs beneficial for models without data augmentation. The optimal number of pseudo boxes, i.e., h E (1, 3)), varies across input types and is determined on a small holdout validation set. See Appendix
A for further details.
Evaluation metries. Following (Kim et al. 2021, Wang et al. 2022), ) we use Average Recall (AR @k) over multiple IoU thresholds (0.5:0.95), and set the detection budget k as 100 by default. All ARs are shown in percentage. AR and A respectively denote the AR score on detecting all classes (including the base and novel ones) and on detecting the novel classes. To evaluate AR v. we do not count the boxes associated to the base classes into the budget k. The same protocol is applied when evaluating per-class ARs and ARs for small, medium and large size of objects, i.e., AR./m//
4.1 DETECTING UNKNOWN OBJECTS IN AN OPEN WORLD
Table la and lb compare open-world class-agnostic object detectors on two cross-category bench- marks and two cross dataset benchmarks, respectively. Our method GOOD, which incorporates geo- metric cues, considerably outperforms state-of -fhee---rr-h--tttt RGB- besed open- open-world world class llass-agnostic object detection methods, i.e., OLN (Kim et al., 202T) and GGN (Wang et al. 2022). OLN did not involve any novel object into training. Although GGN proposed to use an intermediate pairwise affinity (PA) representation for pseudo labeling, the PA predictor still trained on the RGB images, therefore still has the bias towards the known classes as other RGB-basee methods.
On the cross-category benchmarks shown in Table Ia with a single training class "person" on the COCO dataset, GOOD can surpass SOTA methods by 5.0% ARx @ 100 on detecting objects from non-person classes, a relative improvement of 24%. With 20 PASCAL-VOC classes for training, GOOD surpasses SOTA methods by 6.1% AR @ 100 in detecting non- VOC classes, a relative im- provement over 18% On the cross-dataset benchmarks, Table Tb shows that GOOD achieves 2.4% to 4.7% gain A @ 100 in different setups. We observe that GOOD is particularly strong when there are fewer training classes, i.e., person -> non-person and VOC - ADE20k. For RGB-based methods, the overfitting problems become more severe as the object diversity from the base classes reduces. In contrast, the geometric cues can still detect novel-looking objects, which are particularly helpful to training with only a limited number of base class annotations. Finally, ensembling both geometric cues offers additional performance gains.
4.2 ADVANTAGES OF GEOMETRIC CUES
Geometric cues are less sensitive to appearance shifts across classes. We first compare per-novel- class AR@ 5 of the object proposal network trained on geometric cues with that trained on the RGB image. Here, AR @5 is of interest as geometric cues are used to discover novel-looking objects during P and no more than five pseudo boxes per image will be used in Phase-II, see Figure
Figure shows that geometric cues can achieve much higher per-novel-class AR@5 than RGB in many categories. An example is the novel supercategory "food", including classes such as "hot dog" and "sandwich". The base classes, belonging to the supercategory "person", "animal", "vehicle", and "indoor", have very different appearances to the "food" supercategory. The RGB-based model
6

---

Published as a conference paper at ICLR 2023
Person -Non-Person
VOC CLe--Non--VOO
Method
ARA
A
AR
ARg
AR
AR
AR
AR
AR
AR
FRCNN (oracle)
55.9
53.4
37.9
59.5
73.0
55.9
52.6
37.1
60.0
73.1
FRCNN (cls-aen)
25.7
12.2
8.7
12.4
18.2
38.5
27.3
10.8
30.2
55.8
OLN (Kim et al.. 2021
30.9
16.5
8.7
14.7
33.4
47.5
33.2
18.7
39.3
58.6
GGN Wang 2012
30.3
20.7
12.0
25.6
29.6
39.8
31.5
11.8
37.4
63.8
SelfTran-KGB
32.5
18.7
11.3
18.6
32.6
48.1
37.4
22.8
43.9
57.7
GOOD-Depth
37.0
25.6
12.8
30.4
42.4
49.6
39.0
21.1
47.5
63.2
GOOD-Normal
35.6
23.7
13.9
27.6
36.0
49.6
38.9
21.2
47.9
62.0
GOOD-Both
37.3
25.9
14.2
32.6
38.9
49.5
39.3
21.6
48.2
62.4
(a) Cross-category benchmarks
VOC +AD22K
COCO >DE20K
Method
ARA
AR'
ART
AR
ARA
AR
ART
AR
FRCNN (cls-agn)
22.6
15.5
23.7
26.5
25.9
20.5
28.5
27.4
OLN Kim al. 2021
29.2
19.7
30.7
34.4
32.9
25.1
35.9
35.6
GGN
Wang al. 2022
27.0
16.9
27.5
33.6
29.8
18.9
29.1
38.2
SelfTran-RRon
27.7
18.4
29.7
32.4
33.8
26.5
37.6
35.4
GOOD-Depth
33.9
21.1
35.9
41.2
35.3
25.7
38.0
39.6
GOOD-Normal
33.4
22.0
36.2
38.8
33.5
25.7
37.1
GOOD-Both
34.0
21.9
37.0
39,9
35.3
(b) Cross-dataset benchmarks
Table I: Detecting unknown objects in an open world. FRCNN (oracle) is standard Faster R- CNN detector trained all COCO classes and serves as performance upper bound the cross- category benchmarks. FRCNN (cls-agn) is Faster R-CNN trained in a class-agnostic manner, serving as a baseline for comparison. Our geometry-guided methods GOOD-Xs are compared with SOTA open- open-world class-agnostic object detectors, i.e., OLN and GGN. SelTTrin-RGB is RGB- based self-training, i.e., using the RGB image instead of geometric cues for pseudo labeling. We only report A in b) as the classes in ADE20K do not exactly match those in COCO.
has difficulty detecting food using appearance cues. In contrast, the geometric cues can generalize across supercategories. For categories that geometric cues are worse than RGB, we find that they typically are of small sizes, such as knives, forks, and clocks. This shows that while abstracting away appearance details, geometric cues may also lose some information about small objects. This again shows complementariness of RGB and geometric cues.
Geometric cues are better than edges and pairwise affinities. We further compare the geometric cues with two other mid-level representations: 2D edge and PA. The 2D edge map is extracted using the Holistically nested Edge Detection (HED) model (Xie & Tu 2015). which shows more robust performance across the datasets than more recent methods. The HED model is trained on the Berke- ley Segmentation Dataset and Benchmark (BSDS500) dataset (Arbelaez al. 2011) which contains 500 images with segmentation annotations. PA can be thought of as a learned object boundary pre- dictor. Wang et al. (2022) trained directly RGB images and then grouped the predictions into object masks using a combination of traditional grouping methods (Shi & Malik 2000; Arbeláez 2006 Arbeláez
2014). We compare these four data modalities as the source of pseudo label- ing. From Table 2 we can see that depth and normals outperform 2D edge and PA on detecting novel objects. We speculate that this because 2D edge and PA mainly capture object boundaries, whereas depth and normals have an extra spatial understanding of the scene and can thus better detect objects in complex scenes. Owing to their better pseudo labels, GOOOD-Depth/Normal outperform GOOD-Edge/PA the final AR @ 100, i.c.. 39.0%/38. 0066/399% vs. 38317 1 ,, Appendix D
Geometric eues are better than adding shape bias and multi-scale augmentation. In the previ- ous part, we showed that geometric cues can detect novel objects from very different supercategories, owing to their less sensitivity to detailed appearance changes in objects such as textures. It is thus natural compare with
model with inductive shape bias counteract the texture bias. We adopt the stylized ImageNet pretrained ResNet-50 from (Geirhos et al. 2019) as the backbone for SelfTrain-Rde RGB. This backbone trained using heavy style-based augmentation to mitigate the

---

Published as a conference paper at ICLR 2023
Deptr vs RGP
sandwich
ret gerator
Normal vs RGB
umbrrella
AKgS citt (%6)
not dog
toothbrush
150
toaster
laptop
100
knire
clock
Vovel class category
Figure 4: Per-novel-class AR@5 difference comparison of pseudo boxes on COCO VOC Non- VOC. We train the object proposal network the geometric cues (Phase-I in Figure 3) and also directly on the RGB image. We show their per-novel-class AR differences, which is defined as (ARx ARRGs)/ARRe
Normal). with E )ormal, The geometric cues outperform the RGB image on those classes above the zero difference line. We also highlight some classes where RGB and geometric cues have big differences.
Modality
ARx @
ARx @ 5
AR @ 5
AR @5 5
AR's @5
RGB
4.0
12.6
5.4
16.4
21.8
Depth
4.7
13.2
2.4
16.0
31.6
Normal
4.3
12.8
3.0
16.6
27.7
Edge
3.2
10.4
3.5
14.5
18.5
Pairwise affinity (PA)
3.2
8.1
0.5
9.1
30.6
Table 2: Comparison on different data modalities for pseudo labeling. We report AR N @k achieved by the object proposal network trained on different modalities in Phase-l, where the bench- mark is VOC >> Non- NOC. Geometric cues (depth and normal) are stronger in discovering novel objects than the edge and pairwise affinity (Wang et al. 2022).
texture bias, where texture is one of the most discussed appearance features prone to overfit by RGB-based models. showed performance improvements on the COCO object detection bench- mark under the closed-world assumption (Geirhos et al. (219). Moreover, we observe from Figure that RGB is relatively stronger in detecting smaller objects. This leads to the question of whether augmenting SelfTrain-Roe RGB with pscudo boxes extracted different input scales can already help this RGB-based method achieve comparable performance to those incorporating geometric cues.
that while the shape bias backbone improve the AR of SelfTrain- RGB, it degrades the performance on novel classes, i.e., AR,, indicating that shape bias obtained from training ImageNet may not be very helpful in generalizing novel objects. As for the multi-scale augmentation, although can improve the detection recall medium and large objects, the overall performance is still worse than models that incorporate geometry Overall, these comparisons show that geometry provides strong cues for object localization.
From Table
Method
AR
AR
AR
AR
AR
SelfTrain RGBB
48.1
37.4
22.8
43.9
57.7
ShapeBias
48.8
36.5
21.4
42.5
58.6
ScaleAug
48.5
37.9
21.2
44.7
62.2
GOOD-Both
49.5
39.3
48.2
62.4
Table 3: Comparison of GOOD and two other strategies VOC to non-VOC. ShapeBias placed the backbone of SelfTrain-RGB with a stylized ImageNet pre-trained backbone. ScaleAug applied multi-scale augmentation the RGB input for collecting pseudo boxes different scales.
8

---

Published as a conference paper at ICLR 2023
1055
15
11.2
Abuihtte
2400
30%
I05
Thepth
Norual
GOOD-Belh
Plase Iilreelll
(c) Varying the number of
(a) Effectiveness of pseudo labeling. (b) Effectiveness of data augmentation.
base classes.
Figure 5: Ablation studies. (a) and (b) are conducted on COCO VOC to non- non-VOC benchmark. The study the number of base classes (c) uses ADE20K the evaluation benchmark.
+Outdoor, Animal
Aocessory,
+Kitchen, Food
+Furniture, Electronic, Appliance. Indoor
Supercategories
Person
Vehicle
Sports
# Training classes
1
24
56
# Training images
64115 609666
74152 654460
92169 715582
93939 727207
107036
117266 860001
# Training instances
824535
Table 4: Base class choices for studying the influence of the number of base classes.
4.3 ABLATION STUDIES
In this section, we conduct ablation studies to understand the effectiveness of pseudo-labeling, data augmentation, and the influence of the number of base classes. Figure 5a) shows the AR, @ 100 achieved by the object proposal network (Phase-I) and the object detector (Phase-II). The latter uses the pseudo boxes generated by the former, where the former can be trained with different data modalities. The RGB-based object proposal network outperforms the depth- and normal-based one on x @ 100. This indicates that depth and normal maps cannot simply replace RGB image for object detection. As evidenced Table 2 and Figure 4 they are competitive on AR y @k with k 5, meaning their top predictions of high quality. Pseudo labeling is thus effective way for geometry-guideee guided open- open-world object detector training.
We further study the effect of AutoAugment (Cubuk et al. 2019). Using it during Phase-Il II training, we achieve higher A the noise in pseudo labels. However, using Augment class annotations, we observe only improvement recalling the base class objects (from 58.4% to 61.7% AR Base 100), but degradation in novel object detection. This shows that data augmentation via random resizing, cropping, and flipping cannot improve generalization across categories.
y @ 100 as shown in Figure 5b). Data augmentation helpful to counteract
train OLN the ground truth base
Finally, we study the influence of the number of base classes. As shown in Table 4 we split the COCO dataset based on the supercategories to create six different splits. We then train GOOD and OLN on these splits and evaluate them on ADE20K. More base classes in training allow object detectors to learn a more generic sense of objectness so that they can better detect novel objects. As shown in Figure both GOOD and OLN achieve better AR y 100 along with the number of base classes, and their performance gap reduces. However, the annotation cost also grows along with the number classes. GOOD can achieve a similar AR 100 as OLN with only half the number of base classes, e.g., 39 vs. 80. This shows that GOOD is more effective at learning generic sense of objectness and less prone to overfitting to the base classes
CONCLUSION
In this paper, we proposed method GOOD for tackling the challenging problem of open-world class-agnostic object detection. It exploits the easy-to-obtain geometric cues such as depth and normals for detecting unannotated novel objects the training set. As the geometric cues focus on object shapes and relative spatial locations, they can detect novel objects that RGB-based methods cannot detect. By further involving these novel objects into RGB-bart KGB-based object detector training, GOOD demonstrates strong generalization performance across categories well as datasets.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENT
Haiwen Huang and Dan Zhang were supported by Bosch Industry Campus Lab at University of Tübingen. Andreas Geiger was supported by the ERC Starting Grant LEGO-3D (850533) and the DFG EXC number 2064/1 project number 390727645. Haiwen Huang would like to thank Tianlin Ye for her emotional support throughout the project and Jojumon Kavalan for generously providing access to his Internet during the rebuttal period.
REFERENCES
Arbeláez, J. Pont-Tuset, J. Barron, F. Marques, and J. Malik. Multiscale combinatorial grouping. In Computer Vision and Pattern Recognition, 2014.
Pablo Arbeláez. Boundary extraction in natural images using ultrametric contour maps. In 2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'06) 066), pp. 182-182.
IEEE, 2006.
Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hier- archical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell,, 33(5) May 2011 1. ISSN 0162-8828. doi: 00.1099TPAM..200.16.. URL http://.......g....111.1..
TPAMI 2010.161 161
Bryan Chen, Alexander Sax, Francis Lewis, Iro Armeni, Silvio Savarese, Amir Zamir, Jiten- dra Malik, and Lerrel Pinto. Robust policies via mid-level visual representations: An experi- mental study in manipulation and navigation. In Jens Kober, Fabio Ramos, and Claire Tom- lin (eds.). Proceedings of the 2020 Conference on Robot Learning, volume 155 of Proceed- ings of Machine Learning Research, pp. 2328 2346. PMLR, 16-18 Nov 2021. URL https: //proceediggs.c / chen5/cch2215 html
Kai Chen, Jiaqi Wang. Jiangmiao Pang. Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: / /00.:090000555 2019.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 113-123, 2019.
J. Deng, W. Dong, R. Socher, L.-J. Li. K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir Zamir. Omnidata: scalable pipeline for making multi-task mid-level vision datasets from 3d scans. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 10796-10796. 2021.
David Eigen and Rob Fergus. Predicting depth, surface normals and semantic labels with common multi-scale convolutional architecture. In Proceedings of the IEEE international conference on computer vision, pp. 2650-2658. 2015.
David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from single image using multi-scale deep network. Advances in neural information processing systems, 27, 2014.
M. Everingham, L. Van Gool, K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes ( (VOC) challenge. International Journal of Computer Vision, 88(2):303-332) June 2010.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-traing CNNs are biased towards texture; increasing shape bias im- proves accuracy and robustness. In International Conference on Learning Representations, 2019. htt.:http/:/pppeee oorreeeeeoooooooredooooooaa
10

---

Published as a conference paper at ICLR 2023
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665-677) 2020.
Derek Hoiem, Alexei A Efros, and Martial Hebert. Automatic photo pop-up. In ACM SIGGRAPH 2005 Pupers, pp. 577-584. 2005a.
Derek Hoiem, Alexei A Efros, and Martial Hebert. Geometric context from single image. In Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume I, volume 1. pp. 654-661. IEEE, 2005b.
Derek Hoiem, Alexei A Efros, and Martial Hebert. Recovering surface layout from an image. International Journal of Computer Vision, 75(1):151-17, 2007.
Derek Hoiem, Alexei A Efros, and Martial Hebert. Putting objects in perspective. International Journal of Computer Vision, 30-1):3-15, 2008
Ayush Jaiswal, Yue Wu, Pradeep Natarajan, and Premkumar Natarajan. Class-agnostic object de- tection. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 919-928, 2021.
Yuqian Jiang, Nick Walker, Justin Hart, and Peter Stone. Open-world reasoning for service robots. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 29, pp. 725-7233 2019.
K. J. Joseph, Salman Khan, Fahad Shahbaz. Khan, and Vineeth N. Balasubramanian. Towards Open World Object Detection. arXiv:2103.02m0 002003 [cs], May 2021. URL http::/arxiv.cr/ org/ arXiv: 2103.02603.
abs/2103. 02603
Oguzhan Fatih Kar, Teresa Yeo, Andrei Atanov, and Amir Zamir. 3d common corruptions and data augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18963-18974, 2022.
Dahun Kim, Tsung- Yi Lin, Anelia Angelova, In So Kweon, and Weicheng Kuo. Learning Open- World Object Proposals without I Learning to Classify. 1rXiv:2108. [cs], August 2021. URL http::/ http::///aaii..oo//b/b//////////
wwt./oogg//a/......ooooo /210 06753 arXiv: 2 2108.06753.
Sachin Konan, Kevin Liang, and Li Yin. Extending one-stage detection with open-world proposals. arXiv preprint arXiv:2001.02302, 2022.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), ECCV, pp. 740-755, 2014.
Bing Liu, Eric Robertson, Scott Grigsby, and Sahisnu Mazumder. Self-initiated open world learning for autonomous ai agents. arXiv preprint arXiv:21 10. 1385, 2021.
Yang Liu, Idil Esen Zulfikar, Jonathon Luiten, Achal Dave, Deva Ramanan, Bastian Leibe, Aljoša Ošep. and Laura Leal-Taixé, Opening up open world tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19045-19055, 2022.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, Fahad Shahbaz Khan, Rao Muhammad Anwer, and Ming- Hiing- Yang. Class-agnostic object detection with multi-modal transformer. In 17th European Conference on Computer Vision. Springer, 2022.
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. Sim- ple voca-locyy object detection with vision transformers. arXiv preprint arXiv:2205.062300 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
11

---

Published as a conference paper at ICLR 2023
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. ArXiv preprint, 2021a.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. ICCV, 2021b.
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transac- lions on Pattern Analysis and Machine Intelligence, 44(3), 2022.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28, Curran As- neurips http://ps...cpppppr/////////
sociates, Inc., 2015. URL 4455266b116865ee5552222222ooooo3
Kuniaki Saito, Ping Hu, Trevor Darrell, and Kate Saenko. Learning to Detect Every Thing in an Open World. arXiv:2112.01699 [cs], December 2021. URL http:://axii..../t//////// abs/
2112.0165 6968
arXiv: 2112.01698 version: 1.
Axel Sauer and Andreas Geiger. Counterfactual generative networks. In International Confer- ence on Learning Represenlations, 2021. URL
BXewfAYMmJw
Ashutosh Saxena, Sung Chung, and Andrew Ng. Learning depth from single monocular images. Advances in neural information processing systems, 18, 2005.
Ashutosh Saxena, Sung H Chung, and Andrew Y Ng. 3-d depth reconstruction from a single still image. International journal of computer vision, 76(1):53-66 69, 2008a.
Ashutosh Saxena, Min Sun, and Andrew Y Ng. Make3d: Learning 3d scene structure from a single still image. IEEE transactions on pattern analysis and machine intelligence, 31(5):824-844 2008b.
Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transactions on pattern analysis and machine intelligence, 22(8))888 -006,90552100 2000.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying sumi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-608, 2020.
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. FCOS: Fully convolutional one-stage object detection. In Proc. Int. Conf. Computer Vision (ICCV), 2019.
Weiyao Wang, Matt Feiszli, Heng Wang, Jitendra Malik, and Du Tran. Open-world instance seg- mentation: Exploiting pseudo ground truth from leameed pairwise affinity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4422 4432, 2022.
Yongqin Xiang, Julian Chibane, Bharat Lal Bhatnagar, Bert Schiele, Zeynep Akata, and Gerard Pons-Moll. Any-shot gin: Generalizing implicit networks for reconstructing novel classes. In 2022 International Conference on 3D Vision (3DV). IEEE, 2022.
Qizhe Xie, Minh-Thang Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10687-10698, 2020.
Saining Xie and Zhuowen Tu. Holistically-nesteed edge detection. In Proceedings of the IEEE international conference on computer vision, pp. 1395-1403, 2015.
Mengde Xu, Zheng Zhang, Han Hu, Jianfeng Wang, Lijuan Wang. Fangyun Wei, Xiang Bai, and Zicheng Liu. End-to-end sumi-supervised object detection with soft teacher. Proceedings of the IEFF/CVF International Conference Computer Vision (ICCV), 2021.
12

---

Published as a conference paper at ICLR 2023
Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Ex- ploring monocular geometric cues for neural implicit surface reconstruction. arXiv:002200665,, 2022.
Amir Zamir, Alexander Sax, Nikhil Cheerla, Rohan Suri, Zhangjie Cao, Jitendra Malik, and Leonidas Guibas. Robust learning through cross-task consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11197-11206, 2020.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 3223333 2019.
13

---

Published as a conference paper at ICLR 2023
LVIS COCO -Non-OCC
COCO -VVO
Method
ARA
ARx
AR
AR
AR
ARA
AR3
ART
AR'
FRCNN (cls-agn)
26.6
21.0
14.9
32.7
36.2
42.3
22.2
38.3
52,0
OLN (Kim et al. 2021
32.2
27.4
17.9
44.7
53.1
49.2
35.0
48.7
55.1
GGN Wang
2022
27.1
22.5
15.7
35.5
38.4
45.6
25.6
43.2
54,9
SelfTran-RGGB
32.6
28.3
19.0
45.7
52.2
48.7
35.8
48.8
53.5
GOOD-Depth
32.8
28.3
18.3
46.8
54.1
50.3
35.6
49.9
56.4
GOOD-Normal
33.4
29.2
19.3
49.8
53.3
49.8
35.8
50.0
55.0
C000D-Both
33.2
29.0
19.0
47.7
53.0
50.3
36.1
50.2
55.4
Table 5: More benchmarks. The same methods in Table 1 are compared.
A IMPLEMENTATION DETAILS
GOOD uses OLN Kim et al. 2021 as the architecture for both Phase-I and Phase-Il training. OLN is built top of Faster RCNN (Ren et al. (015). For open-world object detection, the classification heads are replaced with the objectness score prediction heads, i.e., predicting the centerness and IoU of each bounding box proposal at the two stages, respectively. We use the objectness score v ycenterness x IoU for ranking the pseudo boxes and selecting the top A pseudo boxes per image. The optimal A choice for GOOD-Depth and GOOD-Normal is and for SelfTrain-RGB is
For Phase-l training, we trained the proposal network with loss as given in Eq. 2 and used the SGD optimizer with an initial learning rate of 0.01 and batch size of 16 for 8 epochs. unless otherwise stated, optimizer with an initial learning rate of 0.01 and batch size of for 16 epochs with AutoAugment. For GOOD-Both, we merge the pseudo boxes generated by object proposal networks separately trained on depth and normal maps by filtering out the overlapping boxes. Specifically, the loU of two pseudo boxes is larger than 0.5, they seen overlapping with each other, and the one with lower objectness score will be filtered out. For other ensembling experiments, not specified, pseudo boxes also merged as described above.
Phase-Il training,
trained the object detector with loss as given in Eq. 3 and used SGD
We use the DPT-Hybrid models from Omnidata repository (Eftekhar et al. 2021) for off-the-shelf inference of geometric cues. The DPT-Hybrid models (Ranftl al., 2021aa aa) have hybrid archi- tecture of attention layers and convolutional layers. They are trained on the Omnidata Starter Dat- (Eftekhar et al. 202T) for one week with 2D and 3D data augmentations (Kar al. 2022). and one week with cross-task consistency (Zamir et al. 2020) on 4 V100 GPUs. To infer on RGB im- ages, we first pad images to sizes divisible by 32 without resizing, then feed them to the DyT-Hybrid model. Note although the original models are trained on 384x384 image patches, we find that in- ferring on the original resolution COCO produces better visual results than on the resolution of 384 384.
B MORE BENCHMARKS
We further evaluate our approach more benchmarks. Specifically, we evaluated the baselines and GOOD on the cross-category benchmark LVIS COCO to non-COCO and cross-dataset benchmark COCO to UVO. The results shown in Table 5 As expected, the performance gains are smaller than those observsed on benchmarks with smaller number of base classes such as COCO VOC to non-VOC. But still, GOOD surpasses all the state-of-the-art RGB- based methods. This shows that even extreme conditions when the number of base classes is large, GOOD can still helpful in improving the open-world performance.
14

---

Published as a conference paper at ICLR 2023
PAOL
100
300
54%
200
375
060
120-VC
to ADEAK
saMOOCO AUEaEL
VSC ADEaC
s==0000 COCO ADEAR.
Dittick apria for of cake pwets abeia
Bittack peczetik ond Bioee FGG pneserte cae
(a) Choice of ensembling geometric cucs.
(b) Fusing inputs: whether to use RGB.
KG5
343
002
60
V00 Ma-
V0C to ADIDIK
OOCO 000000 0000 Abx23K
1000
31505
30003
I G0000s B0
(d) Histograms of top1 pseudo box sizes from models trained on COCO VOC with different modalities.
(c) Ensemble pseudo boxes: whether to use RGB.
Figure 6: More ablation studies.
C MORE ABLATION STUDIES
C.I ENSEMBLING WAYS GEOMETRIC CUES
There are two possible ways to ensemble geometric cues: (I) Stack the two geometric cues together and train a single object proposal network these stacked inputs in Phase-l; (2) Train ob- ject proposal networks and extract pseudo boxes separately, then merge them into a single pseudo box pool for Phase-Il II training. The details of the merging process is described in Appendix [A] We conduct ablation studies these two methods. From Figure [aa we demonstrate that empirically, ensembling pseudo labels slightly better than using stacked inputs for Phase-l training. Through- out the paper, we use the pseudo label ensembling for GOOD-Both.
C.2 ON INCORPORATION RGB IN PHASE-I
is natural to think of incorporating RGB in Phase-I. To do so, we can also consider the two ways for ensembling geometric cues. If stack RGB with geometric cues train the proposal network, the model will tend to make more use of RGB to optimize the target localization loss. This because RGB is much stronger input signal than geometric cues in the closed-world setup AR @ 100 is 58.3 for RGB inputs alone and 44.9 when stacking depth and normals on COCO VOC classes. In the extreme case, the model can even completely ignore geometric cues. This reliance on RGB inputs prevents the model from making the best use of geometric cues to discover novel objects in Phase-I, which is crucial for open- world object detection. As shown in Figure 6b| stacking RGB with geometric cues leads to inferior performance across many benchmarks.
Alternatively, we can train separate object proposal on RGB inputs, extract pseudo boxes, and merge them with those extracted from models trained on geometric cues. We can name this method "GOOD-AII". In Figure 6c] we compare GOOD-All with GOOD-Both and find that adding pseudo boxes from RGB (i.e., GOOD-All) either leads to performance gains even worsens the per- formance on benchmarks like VOC to ADE20K. To understand this, we note that object proposal networks trained on RGB favor smaller detection boxes, evidenced in our visualizations (Fig- ure [0 [T) and more quantitatively in histograms (Figure (d).. These smaller detection boxes can either be small objects just textures and parts of larger objects, which can potentially hurt the
15

---

Published as a conference paper at ICLR 2023
VOC-=Non-VOL
VOC-AD
Method
AR A
AR
AR
AR
AR
AR
AR
ART
AR'
GOOD-Both
49.5
39.3
21.6
48.2
62.4
34.0
21.9
37.0
39,9
GOOD-AIl
48.5
39.3
22.7
47.6
60.8
33.3
22.9
36.2
38.0
Table 6: More comparison of GOOD-Both and GOOD-AII. For GOOD-All, the performance gains in detecting small objects (AR*) are too small to compensate for the losses in detecting larger objects (AR" and ()R')), leading to overall inferior performance.
TOC--No-VOO
VOC AAC-AA
Method
ARA
ARx
AR
AR
AR
ARA
AR
ART
AR
FCOS
43.6
29.3
14.6
35.7
50.2
25.6
17.2
26.5
30.9
OLN
47.5
33.2
18.7
39.3
58.6
29.2
19.7
30.7
34.4
GOOD-Both (FCOS)
48.4
36.3
18.4
46.5
58.0
32.5
20.1
34.5
39.5
GOOD-Both (OLN)
49.5
39.3
21.6
48.2
62.4
34.0
21.9
37.0
39.9
Table Architecture choice. FCOS is a single-stage proposal-free object detector, and OLN is two-stage proposal-based object detector (modified from Faster R-CNN), GOOD can significantly improve the open- world performance of both architectures.
performance of the final detector to detect large objects. This consistent with our observations in Table 6 Compared to GOOD-Both the gains in AR usually too small to compensate for the losses in AR (A)) leading to the inferior overall performance of GOOD-AIL.
C.3 ARCHITECTURE CHOICE
In principle. our approach is model agnostic and is therefore compatible with both proposal-free and proposal-based object detectors. To demonstrate this, besides the two-stage proposal-based detectors in the main paper, also experiment with more recent single-stage proposal frree object detector FCOS (Tin et al., 2019). Specifically, we kept Phase unchanged, i.c., generating the geometric cue-based pscudo boxes using OLN as the architecture. In Phase II, we train a class-agnostic FCOS using the extracted pseudo boxes together with the groundtruth annotations of the base classes. The experiment results shown in Table We can that GOOD can significantly improve FCOS in terms of detecting novel objects and OLN is stronger architecture than FCOS to be used in GOOD.
D MORE DISCUSSION ON DIFFERENT MODALITIES FOR GOOD
In this section, we further discuss how different modalities behave and can be further ensembled to boost the performance. We first compare different modalities used for Phase-I training and pseudo labeling in GOOD on the COCO VOC to non-V non-VOC benchmark. In Table g we show that pseudo- labeling using the geometric cues leads to stronger performances. This agrees with our study in Table 2 where we found that pseudo boxes from proposal networks trained on geometric cues have higher y @5, indicating that they are of higher quality.
D.1 COMPLEMENTARINESS OF DIFFERENT MODALITIES
In the main paper, we have combined the pseudo boxes only from the geometric cues, i.e., depth and normals. GOOD-Both provides additional performance gains over GOOD-Depth and GOOD- Normal. As we have more than two sources of pseudo labeling, it is natural to examine further if they are complementary and thus can be jointly exploited. We first evaluate the overlap of their top-1 ranked pseudo boxes. Their most confident novel object detections best convey their bias in generalization. We can observe from Figure 7a the overlap is low across all the input types. Ta- ble and Table 8 further reveal their complementariness in detecting different sizes of objects. Both
16

---

Published as a conference paper at ICLR 2023
Modality
AR
AR
AR
AR
V
SelfTrain-RGB
37.4
22.8
43.9
57.7
GOOD-Edge
38.1
21.8
45.7
60.2
GOOD-PA
37.1
18.6
43.9
65.3
GOOD-Depth
39.0
21.1
47.5
63.2
GOOD-Normal
38.9
21.4
47.9
62.4
Table 8: Comparison of GOOD using different modalities on COCO VOC to non-VOC bench- mark.
Nomse
Adg
4
100
40
100.0%
21.1 %
25.0%
30.1
5.3%
30
00pe9
21.1%
1000
98.4%
20
78%
AR 9110
Abmmal
26.0 %
28.9 %
000.0%
24.9 %.
6.9%
37
m
24.3%
200.0%
5.0%
36
SellItiin-RG
SelfTrain- RGB (eus)
100.0%
(a) Overlap of topl pseudo boxes.
(b) Influence of top-k when ensembling.
different representations. (b), SelfTran--RGB using the top-k pseudo boxes from single RGB-based object proposal netowrk for retraining, and SelfTrain- RGB (ens) is using pscudo boxes merged from independently trained RGB-based object proposal netowrks for retraining.
Figure 7: Complementariness
observations motivate us to ensemble different sources of pseudo boxes, exploiting their diversity for training the object detector.
To decide which modality to ensemble first, we designed a simple greedy algorithm based on the overlap of pseudo boxes and the potential performance gain of adding the modality the ensemble. Specifically, starting with the best-performing modality (depth), we incrementally ensemble more sources of pseudo boxes by selecting the source with the highest Utility Uniqueness score, where U Utility is the performance gain against a vanilla OLN, and Uniqueness is the overlap of the pseudo boxes with the current ensemble pseudo boxes. The performance is evaluated on a holdout validation set. The chosen ensemble order for the COCO VOC to non- VOC benchmark is: depth, normal, PA, edge, RGB.
We show the ensembling results in Figure 7b Two baselines for using multiple pseudo boxes from RGB are considered: SelfTrai--RGB is using the top-k pseudo boxes from a single RGB-based object proposal netowrk for retraining, and SelfTrain-RGB (ens) is using pseudo boxes extracted and merged from independently trained can see that ensembling pseudo sources from multiple modalities is better than adding more pseudo boxes from single source (RGB).
object proposal networks for retraining. We
E MORE VISUALIZATION
E. VISUALIZATION OF GEOMETRIC CUES
We visualize more examples of geometric cues in Figure g and Figure 9 We demonstrate that the inferred geometric cues of high quality in diverse scenes.
17

---

Published as a conference paper at ICLR 2023
E.2 VISUALIZATION OF PSEUDO BOXES FROM PHASE-I
We also provide visualization of pseudo boxes in Figure I0 and Figure 11 generated from OLNs trained RGB, depth, and normals, respectively. We find that pseudo boxes from RGBB-baed models generally tend to target small objects, textures, and parts of objects. This again shows that RGB-based models over-rely on appearance cues and can overfit textures and discriminative parts of the training classes.
The pseudo boxes are
E.3 VISUALIZATION OF GOOD DETECTIONS ON NOVEL ORJECTS
We further added more visualization examples of GOOD detection results in Figure 12 images contain objects that are seen neither in the GOOD training set (COCO) the Omnidata model training set. The presented examples include new technology devices, spaceships, dinosaurs, aliens, and sea scenes. We predictions even though these objects have never appeared in the training set.
The test
that GOOD still make reasonable object bounding box
18

---

Published as a conference paper at ICLR 2023
Figure 8: Visualization of geometric cues. From left to right: RGB, depth, normals.
19

---

Published as a conference paper at ICLR 2023
EXHIBIT WEST
Figure 9: Visualization of geometric cues. From left to right: RGB, depth, normals.
20

---

Published as a conference paper at ICLR 2023
Figure 10: Top3 pseudo boxes after filtering out those that overlap with known (VOC) class bounding boxes. The pscudo boxes generated from OLNs trained RGB, depth, and normals, respectively. OLN trained on RGB tends to detect small objects and parts of objects.
21

---

Published as a conference paper at ICLR 2023
GANNOS
Figure 11: Top3 pseudo boxes after filtering out those that overlap with known (VOC) class bounding boxes. The pseudo boxes generated from OLNs trained RGB, depth, and normals, respectively. OLN trained on RGB often detect textures or small parts the objects.
22

---

Published as a conference paper at ICLR 2023
Figure 12: GOOD detections novel objects. Only top 20 detection boxes shown with the images. The novel objects are seen neither in GOOD training, nor in Omnidata training set.
23


Reviews:
[{"review_id": "RGi60DSbQa", "reviewer": "Reviewer_FTvx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studied to improve the generalization of open-world class-agnostic object detection with geometric cues. In practice, depth and normal maps predicted from pretrained Omnidata model were used to train an object proposal network for pseudo-labeling unannotated novel objects. The resulting Geometry-guided Open-world Object Detector (GOOD) significantly improves detection recall for novel object categories even with only a few training classes.", "summary_of_the_review": "I tend to reject this paper because of the limited novelty in this paper. \n\nupdate: After discussion, I would like to raise my rating from 5 to 6 because of their great performance. ", "strength_and_weaknesses": "[Strength]\n1. This paper shows that depth and normals help overcome the overfitting problem of the only RGB-based proposal network, thus generating more unlabeled pseudo boxes which finally improve the detection performance.\n\n[Weaknesses]\n1. The novelty of this paper is limited. The network architecture and training losses used in this paper are the same as OLN. It looks that the only difference is that this paper uses additional geometric cues to train the proposal network, while OLN only uses RGB images. However, to my knowledge, geometric cues have been widely used in many 2D tasks to improve performance. Thus, this paper does not give new messages and insights.\n2. In experiments, it is recommended to study how does the number of base classes influence the performance.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OMYTFQ7P0e", "reviewer": "Reviewer_e8x4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on open-world class-agnostic object detection tasks. This paper proposes a pseudo-labeling approach to alleviate the issue that base object categories are limited. Different from conventional methods that acquire pseudo labels from RGB-based images, this paper acquires pseudo labels from depth or normal images. The method, named GOOD, achieves state-of-the-art results on several datasets.", "summary_of_the_review": "The paper proposes a novel and effective pseudo-labeling method for open-world class-agnostic object detection. The results acquired are superior as well. However, the authors need to conduct the necessary experiments as requested in the weaknesses for a comprehensive comparison. The reviewer will increase the rating if the above concerns are well addressed.", "strength_and_weaknesses": "*Strength\n1. GOOD reveals that Depth and normal images are superior to edge and PA images for generating pseudo-labels in Table 2. Whereas, it would be better if RGB images are taken as a comparison.\n\n2. The idea to leverage the depth and normal images in the open-world class-agnostic object detection task is simple and innovative. Low-level information deserves more attention in open-world tasks.\n\n3. GOOD achieves state-of-the-art results in several datasets.\n\n*Weaknesses\n1. The experiments are insufficient. For example, in cross-dataset benchmarks in Table 1, GOOD only selects ADE2k as the target test set. However, most approaches [1, 2] mainly consider UVO [3] as the target dataset for the cross-dataset benchmark.\n\n2. Heuristically, pseudo-labeling methods work on the partially labeled training set, such as VOC categories in COCO. The effects need to be further validated if the training set is exhaustively (e.g. UVO dataset) or nearly exhaustively (e.g. COCO to Non-COCO in LVIS dataset) labeled. The authors need to add at least one experiment under such scenes.\n\n3. There are several mistakes in the paper, such as the legend in Fig. 7b.\n\n[1] Wang, W., Feiszli, M., Wang, H., Malik, J. & Tran, D. Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinity. CVPR (2022).\n[2] Saito, K., Hu, P., Darrell, T. & Saenko, K. Learning to Detect Every Thing in an Open World. Arxiv (2021).\n[3] Wang, W., Feiszli, M., Wang, H. & Tran, D. Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentation. ICCV (2021).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "E3GrG8dlr6Y", "reviewer": "Reviewer_SvRu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an interesting and novel idea of using geometric cues like depth and normal maps to obtain object proposals for object detection. The authors propose a two phase approach for improving object proposals for unseen objects. In the first stage they train an object proposal generation method which uses geometric cues like depth and normal maps to provide pseudo-labeling. The generated pseudo-labels are added to the training set and the final proposal method is trained using the original boxes and these generated boxes. ", "summary_of_the_review": "I am currently recommending a \"weak-reject\" rating based on the weaknesses mentioned above. However, I would be happy to consider upgrading it to an \"accept\" if the authors are able to satisfactorily address the points mentioned above. ", "strength_and_weaknesses": "The idea of explicitly using geometric cues for object proposal generation is novel and interesting. The authors have proposed a two phase training pipeline which first uses geometric cues to generate proposals for unlabeled objects. These pseudo-labels and the original ground-truths are used to fine-tune the proposal network in the second phase. The authors show that the proposed approach can achieve improved performance.\n\nThe main weaknesses of the paper are lack of details about the decisions, and an absence of discussion on how to extend the proposed method to more recent proposal-free object detection methods. In particular, the authors should address the following in a revision/rebuttal:\n\n1. How exactly is the proposal network trained in Phase 1? Is the same network trained for both depth and normal? How exactly? Are the depth and normal maps stacked? Or are they just used as separate/independent images? \n\n2. Connected with the first point above - Why not use RGB images as well in the first step, along with depth and normal? Does that lead to worse performance? Why/Why not? Further, what other geometric cues can be used in addition to depth and normal? \n\n3. I want the authors to consider and discuss, how can this proposed approach be extended to proposal-free object detectors? What is the utility of proposal-based object detectors if DETR and other similar object detectors give better performance? Further, I want the authors to consider if the presented approach can help proposal-based object detectors achieve a higher performance than more recent proposal-free object detectors. For example, can a Mask-RCNN containing an RPN trained with the proposed approach get better performance than a DETR model? \n\n4. The authors should demonstrate the effectiveness of the proposed approach on larger and more challenging datasets like LVIS. This will help in making their claims stronger. \n\n\n\nAfter discussion with the authors and consulting with the AC and other reviewers, I am increasing my rating. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DFo-a8Z1z0", "reviewer": "Reviewer_3ZGc", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a self-training approach for detecting novel objects in an open-world setting using off-the-shelf monocular depth and normal estimators. The presented method improves over prior work on open-world settings on COCO and ADE20K. ", "summary_of_the_review": "Overall, this is a good paper that should be accepted. Given the nice idea and promising results, the work leaves me wanting more (results with instance segmentation, results on LVIS, further analysis, etc.), but the work passes the bar for acceptance as is. ", "strength_and_weaknesses": "Strengths: \n- Intuitive, novel method that outperforms prior work on important benchmarks\n- Many reasonable ablations that compare against baselines that help show the value of geometric cues vs. self-training in general.\n- Good analysis of where the method improves, and where the RGB self-train method falls short or outperforms vs. the proposed method.\n\nWeaknesses: \n- Segmentation: One would expect that a key win from dense geometric cues would be improved segmentation of unseen objects. This work (and possibly the results) would be much stronger if evaluated on instance segmentation rather than bounding-box object detection. Admittedly, this is a more complex task, but would increase the impact of this paper.\n- As one might expect, the accuracy improvements drop off significantly as the # base classes increases. It might be nice to plot accuracy on novel classes as the # base classes increases (you already have two points for this plot in Table 1 (b) at n=20 and n=80). \n- Relatedly, it would have been nice to see experiments on the LVIS dataset, which covers many categories, and has a vocabulary that matches to COCO. This would have allowed authors to report AR_N for LVIS, unlike for ADE20k in Table 1 (b).\n- It would have been nice to see how well the model generalizes to objects which are not in the training sets of the depth/normal estimation models. This could even be qualitative, on a few images of objects known to be missing from the original training sets (e.g., a new object from a movie, or a recently introduced tech device).\n- Nits:\n    - I would highly recommend removing Fig 5, or at least replacing it with 5-10 randomly chosen images, instead of 1 (likely carefully selected) image. It’s difficult to generalize from this one example, and can lead unsuspecting readers to lend excess credence to the conclusion drawn from that example.\n    - Fig 7b – (1) do you take more pseudo boxes from a single RGB model? Wouldn’t a more fair comparison train K independent RGB models?   (2) When using multiple modalities, do you do any filtering? i.e., if the top box from depth and from edges is the same, do you use both as pseudo-groundtruth?\n    - Appendix A – any idea why k is higher for RGB than depth and normals?\n    - Table 3 – Given RGB helps improve for smaller classes, does it make sense to train a “Good-All” which uses RGB, depth and normals? Or to sample pseudo-groundtruth based on size (small boxes from RGB model, larger boxes from depth and normals)?  [No need to run this experiment, just curious about the authors’ thoughts]\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster