VoxDet: Voxel Learning for Novel Instance Detection
Authors: Bowen Li
Jiashun Wang
Yaoyu Hu
Chen Wang
Sebastian Scherer
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Unseen object detection
instance perception
voxel representation
TL;DR: This work proposes VoxDet, a novel instance detector based on geometry-invariant voxel learning.

Abstract: Detecting unseen instances based on multi-view templates is a challenging problem due to its open-world nature. Traditional methodologies, which primarily rely on $2 \mathrm{D}$ representations and matching techniques, are often inadequate in handling pose variations and occlusions. To solve this, we introduce VoxDet, a pioneer 3D geometry-aware framework that fully utilizes the strong 3D voxel representation and reliable voxel matching mechanism. VoxDet first ingeniously proposes template voxel aggregation (TVA) module, effectively transforming multi-view 2D images into 3D voxel features. By leveraging associated camera poses, these features are aggregated into a compact 3D template voxel. In novel instance detection, this voxel representation demonstrates heightened resilience to occlusion and pose variations. We also discover that a $3 \mathrm{D}$ reconstruction objective helps to pre-train the 2D-3D mapping in TVA. Second, to quickly align with the template voxel, VoxDet incorporates a Query Voxel Matching (QVM) module. The 2D queries are first converted into their voxel representation with the learned 2D-3D mapping. We find that since the 3D voxel representations encode the geometry, we can first estimate the relative rotation and then compare the aligned voxels, leading to improved accuracy and efficiency. In addition to method, we also introduce the first instance detection benchmark, RoboTools, where 20 unique instances are video-recorded with camera extrinsic. RoboTools also provides 24 challenging cluttered scenarios with more than $9 \mathrm{k}$ box annotations. Exhaustive experiments are conducted on the demanding LineMod-Occlusion, YCB-video, and RoboTools benchmarks, where VoxDet outperforms various $2 \mathrm{D}$ baselines remarkably with faster speed. To the best of our knowledge, VoxDet is the first to incorporate implicit 3D knowledge for 2D novel instance detection tasks.

VoxDet: Voxel Learning for Novel Instance Detection
Bowen Li
Jiashun Wang'
Yaoyu Hu'
Chen Wang?
Sebastian Scherer'
Carnegie Mellon University
State University of New York Buffalo
Abstract
Detecting unseen instances based on multi-view templates is challenging problem due to its open-world nature. Traditional methodologies, which primarily rely on 2D representations and matching techniques. are often inadequate in handling pose variations and occlusions. To solve this, we introduce VoxDet, a pioneer 3D geometry-aware framework that fully utilizes the strong 3D voxel representation and reliable voxel matching mechanism. VoxDet first ingeniously proposes tem- plate voxel aggregation (TVA) module, effectively transforming multi-view 2D images into 3D voxel features. By leveraging associated camera poses, these fea- tures are aggregated into compact 3D template voxel. In novel instance detection, this voxel representation demonstrates heightened resilience to occlusion and pose variations. We also discover that a 3D reconstruction objective helps pre-train the 2D-3D mapping in TVA. Second, to quickly align with the template voxel, VoxDet incorporates a Query Voxel Matching (QVM) module. The 2D queries are first converted into their voxel representation with the learned 2D-3D mapping. We find that since the 3D voxel representations encode the geometry, the relative rotation and then compare the aligned voxels, leading to improved accuracy and efficiency. In addition method, also introduce the first instance detection benchmark, RoboTools, where 20 unique instances are video-recorded with camera extrinsic. RoboTools also provides 24 challenging cluttered scenarios with more than 9k box annotations. Exhaustive experiments are conducted on the demanding -.clluuio,,,,,,o,, YCB-video, and RoboTools benchmarks, where VoxDet outperforms various 2D baselines remarkably with faster speed. To the best of our knowledge. VoxDet is the first to incorporate implicit 3D knowledge for 2D novel instance detection tasks. Our code, data, raw results, and pre-trained models are public at https: //ggtt/bbbtttt com/ haaeJJaaxxxus--ee Meeeauss--e/MMM Me/VoxDet.
first estimate
Introduction
Consider common scenarios of locating the second sock a pair in a pile of laundry or identifying luggage amid hundreds of similar suitcases at airport. These activities illustrate the remarkable capability of human cogmition to swiftly and accurately identify a specific instance among other similar objects. Humans rapidly create mental picture of a novel instance with few glances even they see such an instance for the first time or have never seen instances of the same type. Searching for instances using mental pictures is fundamental ability for humans, however, even the latest object detectors [1-7] still cannot achieve this task.
We formulate the above tasks as novel instance detection, that is identification of an unseen instance cluttered query image, utilizing its multi-view support references. Previous attempts mainly work in 2D space, such as correlation [8. 5], attention mechanisms [6]. or similarity matching [9], thereby localizing and categorizing the desired instance, as depicted in Fig. 1 gray part. However, these techniques struggle to maintain their robustness when faced with significant disparities between the query and templates. comparison to novel instance detection, there a vast amount of work centered around few-shot category-level object detection [7, 1, 2]. Yet, these class-leve matching techniques prove insufficient when it comes to discerning specific instance-level features.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Query Image
Architecture Comparison
Resulfs Robo Tools
Query
2D Correlation / Attention Matching
Support
Previous 2D Framework
Support Templates
Proposals
Open- World
Query
Det.
Enc.
Voxel
Voxel
Support
Reconstraction
Aggregation
Matching
Score
Our 3D-inspired VoxDet
Figure 1: Architecture comparison between previous 2D methods (gray) and proposed VoxDet (black). Previous methods resorts to pure 2D corrllation/atentionnnaaae contrast, VoxDet is 3D-inspired, leveraging reconstruction objective to learn the gcometry-awand voxel representation, which enables more effective and accurate voxel-based instance detection. In the challenging newly built RoboTTools benchmark shown on the right, VoxDet exhibits surprising robustness to severe occlusion and orientation variation.
for novel instance detection. In
Humans exhibit the remarkable capability to swiftly formulate mental model of an unfamiliar instance, facilitated by a rapid comprehension of its 3D geometric structure [10-12]. Leveraging such mental representation, once presented with single query image, human probably search and identify the same instance despite alterations in distance, occlusion, and even approximate the instance's 's orientation. Motivated by this, we propose VoxDet, pioncer 3D geometry- detection framework as shown in Fig. bottom. In contrast to state-of-the-art methods [7, 5, 6, 9, 13], VoxDet adapts two novel designs: (1) a compact 3D voxel representation that is robust to occlusion and pose variations and (2) an effective voxel matching algorithm for identifying instances.
instance
VoxDet consists of three main modules: a template voxel aggregation (TVA) module, open- world detection module, and a query voxel matching (QVM) module. Initially, the TVA module transforms multi-view 2D features of an instance into individual 3D template voxels [10]. These template voxels are then accumulated using relative rotations, thus incorporating both geometry and appearance into condensed template voxel. As VoxDet learns this 2D-3D mapping via reconstruction objective, TVA effectively encapsulates both the geometry and appearance of any instance into a compact template voxel. When presented with query image, VoxDet employs open world detector [14] that universally identifies potential objects within the image as 2D proposals. These proposals are then converted to query voxels via the learned 2D-3D mapping and compared with the template voxel by the QVM module. QVM initiates this comparison process by first estimating the relative rotation between a query voxel and the template, which is then used to align the two voxels. Finally, the comparison between aligned voxels is delivered by carefully designed voxel relation module.
Besides methodology, we also construct large-scale synthetic training dataset, Oprn-World Instance Detection (OWID). OWID comprises 10k instances sourced from the ShapeNet [15] and Amazon Berkeley Objects (16) datasets, culminating in 55k scenes and 180k query bounding boxes. Trained on OWID, VoxDet demonstrates strong generalization ability on novel instances, which we attribute to the meticulously designed voxel-based framework and the large-scale OWID training set.
To validate VoxDet, we further build RoboToools new instance detection benchmark compiled from a diverse range of real-world cluttered environments. RoboTools consists of 20 unique instances, 24 test scenes, and over 9,000 annotated bounding boxes. As shown in Fig. right, in the demanding RoboTools benchmark. VoxDet can robustly detect the novel instances under severe occlusion or varied orientation. Evaluations are also performed on the authoritative Linemod-Oue Linemod--Qcelusion [17] and YCB-video [18] for more compelling results. The exhaustive experiments on these three benchmarks VoxDet not only outperforms various previous works [5-7] and different 2D baselines [19, 9] but also achieves faster inference speed.
demonstrate that our 3D geometry-ane
2 Related Works
Typical object detection [20-26] thrive in category-level tasks, where the instances belonging to a pre-defined class are detected. Typical object detection can be divided into two-stage approaches and

---

one-stage approaches. For the former one, RCNN [20] and its variants [21, 22] serves as foundations, where the regions of interest (ROI) are first obtained by the region proposal network. Then the detection heads classify the labels of each ROI and regress the box coordinates. On the other hand, the YOLO series [23-25] and recent transformer-base based methods [4, 3] are developing promisingly as the latter stream, where the detection task is tackled as an end-to-end regression problem.
Few-shot/One-shoot object detection [1. 27, 28, 2, 29, 7] can work for unseen classes with only few labeled support samples, which are closer to our task. One stream focuses on transfer-learning techniques [28, 27], where the fine-tuning stage is carefully designed make the model quickly generalize to unseen classes. While the other resorts to meta-learning strategies [1. 7, 2, 29), where discovered and leveraged. Since the above
various kinds of relations between supports and queries methods are category-level. they assume more than one desired instances exist in an image, so the classficaaion/mmaccingg designs are usually tailored for Top-100 precision, which is not a very strict metric. However, they can easily fail in our problem, where the Top-/ accuracy is more important.
Open-world//Zero-shor object detection [30-32, 14] finds any objects on an image, which is class-agnostic and universal. Some of them learn objectiveness [30, 14] and others [32] rely on large-scale high-quality training sets. These methods can serve as the first module in pipeline, which generates object proposals for comparison with the templates. Among them, we adopt [14] with its simple structure and promising performance.
Instance detection requires the algorithm to find unseen instance in the test image with some corresponding templates. Previous methods [6, 5, 8] usually utilize pure 2D representations and 2D matching/relation techniques. For example. DTOID [6] proposed global object attention and local pose-specific branch to predict the template-guids guided heatmap for detection. However, they easily fall short when the 2D appearance variates due occlusion or pose variation. Differently, VoxDet leverages the explicit 3D knowledge in the multi-view templates to represent and match instances, which is geometry-ingee invariant.
Multi-view 3D representations Representing 3D scenes/instances from multi-view images is a long-standing problem in computer vision. Traditional methods resort to multi- view geometry, where structure from motion (SfM) [33] pipeline has enabled joint optimization of the camera pose and 3D structure. Modern methods usually adopts neural 3D representations [34, 11, 35-37, 12, 10], including deep voxels [35, 12, 10, 38] and implicit functions [36, 37], which have yielded great success in 3D reconstruction novel view synthesis. Our framework is mainly inspired by Video Autoencoder [10], which encodes video by separately learning the deep implicit 3D structure and the camera trajectory. One biggest advantage of [10] is that the learned Autoencoder can encode and synthesize test scenes without further tuning or optimization, which greatly satisfies the efficiency requirement of our instance detection task.
3 Methodology
3.1 Problem Formulation
Given training instance set Obase and an unseen test instance set Onovel, where Orase N NO Oovel o. the task of novel instance detection (open-world detection) is to find an instance detector trained on Obse and then detect new instances in O Ouovel with no further training or finetuning. Specifically, for each instance, the input to the detector is a query image I E 3 xH and a group of M support templates ID E I X3xWxXX of the target instance. The detector is expected to output the bounding box b E TR of an instance on the query image. We assume there exists exactly one such instance in the query image and the instance is located near the center of the support images.
3.2 Architecture
The architecture of VoxDet is shown in Fig. 2, which consists of an open- world detector, a template voxel aggregation (TVA) module, and query voxel matching (QVM) module. Given the query image, the open-wordd world detector aims to generate universal proposals covering all possible objects. TVA aggregates multi-view supports into a compact template voxel via the relative camera pose between frames. QVM lifts 2D proposal features onto 3D voxel space, which then aligned and matched with the template voxel. In order to empower the voxel representation with 3D geometry, we first resort to a reconstruction objective in the first stage. The pre-trained models serve as the initial weights for the second instance detection training stage.
3

---

Stage Reconstruction
Stage
Query To
Instance Detection
1
Proposals FQ
Result
Cls Reg
Open- WpendWorld Detection
TVA
Initialize 2D-3D Mapping
Support 7
TVA
QVM
R + Rot
Template Voxel Aggregation
Query Voxel Matching
Dec
R
Rot
Voxel Relation
R
Voxel Relation
Lreeon
Legan
F
Mapping
2D-3D
V
F
Mapping
2D-3D
Rot
MLP
Eperoep
Figure 2: Architecture of VoxDet. VoxDet mainly consists of three modules, namely, open-world detection, template voxel aggregation (TVA), and query voxel matching (QVM), We first train TVA via the reconstruction stage, where the 2D-3D mapping learns to encode instance geometry. Then the pre-trained mapping serves as initial weights in the TVA and QVM modules for detection training.
3.2.1 Open-World World Detection
Since the desired instance is unseen during training, directly regressing its location and scale is non-trivial. To solve this, we first use an open-world detector [14] to generate the most possible candidates. Different from standard detection that only finds out pre-defined classes, an open- world detector locates all possible objects in an image, which is class-agnostic.
As shown in Fig. 2, given a query image 2 a 2D feature map f extracted by a backbone network (-)) To classify each pre-defined anchor foreground (objects) background, the region proposal network (RPN) [22] is adopted. Concurrently, the boundaries of each anchor are also roughly regressed. The resulting anchors with high classification scores termed region proposals P P1 P2: ... PN] E R where is the number of proposals. Next, to obtain the features F for these candidates, we use region of interest pooling (ROIAlign) [22], FQ ROLAlign(P) (P, Q) E IR xxNxxxxx where C' denotes channel dimensions and w is the spatial size of proposal features. Finally, we obtain the final classification result and bounding box by two parallel multi-layer perceptrons (MLP), known as the detection head, which takes the proposal features Q as input, and outputs the binary classification scores and the box regression targets. The training loss is comprised of RPN classification loss CRPN, RPN regression loss L
CRPN head classification loss
gHead and head regression loss D
els
To make the detector work for open- world objects, the classification branches (in RPN and head) are guided by objectiveness regression [14]. Specifically, the classification score is defined (supervised) by Intersection over Union (IoU), which showed a high recall rate over the objects in test images, even those unseen during training. Since they have learned the class-agnostic "objectiveness" we assume the open-world proposals probably cover the desired novel instance. Therefore, we take the top-ranking candidates and their features as the input of the subsequent matching module.
3.2.2 Template Voxel Aggregation
To learn geometry-invarian representations, the Template Voxel Aggregation (TVA) module com- presses multi-view 2D templates into a compact deep voxel. Inspired by previous technique [10] developed for unsupervised video encoding, relative orientation in the physical 3D world. To this end, we first generate the 2D) feature maps
propose encode our instance templates via their
F$ ( M using a shared backbone network vi(-) used the query branch and then map the 2D features to 3D voxels for multi-view aggregation.
2D-3D mapping: To map these 2D features onto shared 3D space for subsequent orientation-based aggregation, we utilize an implicit mapping function M(.). This function translates the features to 3D voxel features, denoted by V -MFF)) BR M X D where V is the 3D voxel feature

---

from the 2D feature, C. is the feature dimension, and D, L indicate voxel spatial size. Specifically, we first reshape the feature maps to F ( where d is the pre-defined implicit depth, then we apply 3D inverse convolution to obtain the feature voxel.
Note that with multi-view images, we can calculate the relative camera rotation easily via Structure from Motion (SfM) [33] or visual odometry [39]. Given that the images are object-centered and the object stays static in the scene, these relative rotations in fact represent the relative rotations between the object orientations defined in the same camera coordination system. Different from previous work [10] that implicitly learns the camera extrinsic for unsupervised encoding, we aim to explicitly embed such geometric information. Specifically, our goal is to first transform every template into the same coordinate system using their relative rotation, which is then aggregated:
M Σ (Rot), R,))), M
V
where V, E RCCxDxLXL is the previously mapped i-th independent voxel feature, R denotes the relative camera rotation between the i-th support frame and the first frame. Rot(:-) the 3D transform used in [10], which first wraps unit voxel to the new coordination system using R. and then samples from the feature voxel V, with the transformed unit voxel grid. Therefore, all the M voxels transformed into the same coordinate system defined in the first camera frame. These then aggregated through average pooling to produce the compact template voxel vs.
By explicitly embedding the 3D rotations into individual reference features, TVA achieves geometry- aware compact representation, which is more robust to occlusion and pose variation.
3.2.3 Query Voxel Matching
Given the proposal features F from query image Io and the template voxel CS from supports IS, the task of the query voxel matching (QVM) module is to classify each proposal as foreground (the reference instance) or background. As shown in Fig. 2, in order to empower the 2D features with 3D geometry, we first use the same mapping to get query voxels, VQ = .MFF)) E R. DxLLL VoxDet next accomplishes matching v and QQ through two steps. First, we need estimate the relative rotation between query and support, so that v can be aligned in the same coordinate system as v Second, we need learn a function that measures the distance between the aligned two voxels. To achieve this, we define a voxel relation operator Ry):;)):
Voxel Relation Given two voxels II, V2 E Rexaxaxa where is the channel and a is the spatial di- mension, this function seeks to discover their relations in every semantic channel. To achieve this, we first interleave the voxels along channels as In(v1, v2) = [v]. v2j, vî, v².-. .vi vo] E 22exxxxxa where vt, v is the voxel feature in the k-th channel. Then, we apply grouped convolution as (V1, V2) = Conv3D(InnV1. V2); group = e). the experiments, we found that such a design makes relation learning easier since each convolution kernel is forced to learn the two feature voxels from the same channel. With this voxel relation, we can then roughly estimate the rotation matrix R E x3x3 of each query voxel relative to the template as:
RC Q ML Vo)),
where S is copied N times to get VS In practice, we first predict 6D continuous vector [40] as the network outputs and then convert the vector a rotation matrix. Next, we can define the classification haed with the Voxel Relation as:
$ = MLP (VS, ...RRRt.. R9)). (R))V.)
where Rot (VQ, RO) rotates the queries to the support coordination system to allow for reasonable matching. In practice, we additionally introduced global relation branch for the final score, so that the lost semantic information in implicit mapping can be retrieved. More details are available the supplementary material. During inference, we rank the proposals P according to their matching score and take the Top-k candidates as the predicted box b.
3.3 Training Objectives
As illustrated in Fig. 2, VoxDet contains two training stages: reconstruction and instance detection. Reconstruction To learn the 3D geometry relationships, specifically 3D rotation between instance templates, we pre-trai traitrain the implicit mapping function MM() using a reconstruction objective. We
5

---

The instances and scenes in the newly built Tools benchmark. The 20 unique instances are recorded as multi-view videos, where the relative camera poses between frames are provided. RoboTools consists of various challenging scenarios, where the desired instance could be under severe occlusion or in different orientation.
Figure 3:
divide M multi-view templates S into input images I E (MMM-K)axXXx x and outputs I E IK K X Next, we construct the voxel representation V using Z via the TVA module and adopt decoder network Dec to reconstruct the output images through the relative rotations:
_ Dec(Rot(V RR)) .j e (1,2, .K}.
where 25, denotes the j-th reconstructed (fake) output images and R, is the relative rotation matrix between the 1-st to j j-th camera frame. We finally define the reconstruction loss as:
Wrecon Crecon + Wgan Cgan Upeccpppppercpp
where Lroron denotes the reconstruction loss, i.e, the LI distance between TO and 7s. Lyan is the generative adversarial network (GAN) loss, where additionally train discriminator classify I and 78. Lpecce means the perceptual loss, which is the LI distance between the feature maps I and 78 each level of VGGNet [41]. Even though the reconstruction is only supervised on training instances, thus reason that the pre-trained voxel mapping can roughly encode the geometry of an instance.
observe that it can roughly reconstruct novel views for unseen instances. We
Detection base training: In order to empower M(.) with geometry encoding capability, we initialize with the reconstruction pre- pre-trained weights and conduct the instance detection training stage. In addition to the open- world detection loss [14], we introduce the instance classification loss L and rotation estimation loss Clus to supervise our VoxDet.
We define Clus the binary cross entropy loss between the true labels E (0, 1 and the predicted scores & E x2 from the QVM module. The rotation estimation loss is defined as:
Tot
Ins
R
(6)

---

Table 1: Overall performance comparison synthectic-real datasets LM-O [17] and YCB-V [18]. Compared with various 2D methods, including correlation [5], attention [6]. and feature matching [9, 19], our VoxDet holds superiority in both accuracy and efficiency. OLN* means the open- world object detector (OW Det.) [14] is jointly trained with the matching head while OLN denotes using fixed modules. the model is trained both synthetic dataset OWID and real images.
Test/Metric
LM-O
YCB-V
Avg.
Method
OW Det.
Train
mAR
ARso
ARrs
ARaG
mAR
ARso
AR7s
ARar
mAR AR
AR7s Speed
VoxDet
OLN
OWID
29.2
43,1
33.3
0.8
31.5
51.3
33,4
1.7
30.4
47.2
33,4
6,5
OLN [14, 5j
OLN*
OWID
22.3
34,4
24.7
0.5
24.8
41.1
26.1
0.7
23.6
37.8
25.4
5.5
DTOID [6]
N/A
OWID
9.8
28.9
3.7
<0.1
16.3
48.8
4.2
<0.1
13.1
38.9
4.0
2.8
OS2D [7]
N/A
OWID
0.2
0.7
0.1
<0.1
5.2
18.3
1.9
<0.1
2.7
9.5
1.0
5.3
OLNeLIP [14, 19]
OLN
OWID
16.2
32.1
15.3
0.5
10.7
25,4
7.3
0.2
13.5
28.8
11.3
2.8
OLNDINO [14, 9]
OLN
OWID
23.6
41.6
24.8
0.6
25.6
53.0
21.1
0.8
24.6
47.3
23.0
2.8
Gen6D
N/A
OWID
12.0
29.8
6.6
<0.1
12.1
37.1
5.2
<0.1
12.1
33.5
5.9
BHRI. [42]
COCO
14.1 21.0
15.7
0.5
31.8
47.0
34.8
23.0
34.0
25.3
N/A
Table 2: Overall performance comparison on the newly built real image dataset, Robo' For fairness, we only compare with the models fully trained on synthetic dataset here, more compari- son see Appendix D. VoxDet shows superiority even under sim-to-real domain gap compared with other 2D representation-base methods [14, 5-7].
Table 3: Per module efficiency comparison. All the four methods share the same open-world de- tector [14]. Compared with 2D baselines that adopt cosine similarity [9, 19] or learnable corre- lation [5], our Voxel matching is more efficient, which shows ~ 2 x faster speed. The numbers presented below are measured in seconds.
Metric
Det.
ARso
ART5
ARgs
Methed/Module
Open-World World
Matching
Total
VoxDet
OLN*
18.7
23.6
20.5
5.1
VoxDet
0.032
0.154
OLNoore. 51
OLN*
14.4
18.1
15.7
3.8
OLNeLIP [14, 191
0,248
0,370
DTOID [6]
N/A
3.6
9.0
2.0
<0.1
OLNDINO [14, 91
0.122
0,235
0,357
OS2D [7]
NEA
2.9
6.5
2.0
<0.1
OLar [14,5]
0.060
0.182
where RD is the ground-truth rotation matrix of the query voxel. Note that here we only supervise the positive samples. Together, our instance detection loss is defined as:
La Cas
RPN + w2
RPN + Cy
Head + w.
HHead + uy clns + W6 Coo
Remark 1: In both training stages, we only use the training objects, Obase- During inference, VoxDet doesn't 't need any further fine-tuning or optimization for O novell
4
Experiments
4.1
Implementation Details
Our research employs datasets composed of distinct training and test sets, adhering to Obase Onovel = & to ensure no overlap between semantic classes of Obase and Onovel.
Synthetic Training set: In response to the scarcity of instance detection traing sets, we've compiled a comprehensive synthetic dataset using 9,901 objects from ShapeNet [15] and ABO [16]. Each instance is rendered into a 40- 40-frame, object-centric 360" video via Blenderproc [43]. We then generate a query scene using to 15 randomly selected objects from the entire instance pool, each initialized with a random orientation. This process yielded 55,000 scenes with 180,000 boxes for training and an additional 500 images for evaluation, amounting to 9,800 and 101 instances respectively. We've termed this expansive training set "open-world instance detection" (OWID-10K) signifying our model's 's capacity to handle unseen instances. To our knowledge, this is the first of its kind.
Synthetic- -Rell Test set: We utilize two authoritative benchmarks for testing. LineMod- Occlusion [17] (LM-O) features 8 texture-less instances and 1,514 box annotations, with the primary difficulty being heavy object occlusion. The YCB-Vides [18] (YCB-V) contains 21 instances and 4,125 target boxes, where the main challenge lies in the variance in instance pose. These datasets provide real test images while lacks the reference videos, we thus render synthetic videos using the CAD models in Blender.
Fully- Test set: To test the sim-to-real transfer capability of VoxDet, we introduced a more complex fully real-world benchmark, RoboTools, consisting of 20 instances, 9,109 annotations,
7

---

Obj. 8
Obj.
Obj- 18
Ratio to Defiault
1,00
Ratio to Default
1.0
Ratio to Default
1.0
0.77
--370%
0.9
-9.1%
0.-
-1.77%
-87.1%
0.8
-27.8%
0.25
M 10
8
M 10
10
Num. Templates Analysis
VoxDet
OINNINO
Figure 4: Number of templates analysis of VoxDet and 2D baseline, OLNDINO [14, 9] on YCB-V benchmark. Thanks to the learned geometry-aware 2D-3D mapping, VoxDet can work well with very few reference images, while 2D method suffers from such setting, dropping up to 87%.
Top K Analysis on YCB-V Benchmark
VoxDet OS2D
10AR
60
-9.4%
AR50
5.00%
-38.5%
K
K
100
Figure 5: Top-K analysis of VoxDet and One-shot object detector [7]. By virtue of the instance-level matching method, QVM, VexDet can better classify the proposals, so that 90% the true positives lie in Top- 10, while for OS2D, this ratio is only 60%.
and 24 challenging scenarios. The instances and scenes presented in Fig. 3. Compared with existing benchmarks [17, 18), Robo'Tools is much more challenging with more cluttered backgrounds and more severe pose variation. Besides, the reference videos of RoboTools are also real-images, including real lighting conditions like shadows, We also provide the ground-truth camera extrinsic.
Baselines: Our baselines comprise template-driven instance detection methods, such as correla- tion [5] and attention- based approaches [6]. However, these methods falter in cluttered scenes, like those in LM-O, YCB-V, and Robo'Tools. Therefore, we' 'e self-constructed several 2D baselines, namely, DINO OLNCLIP, and OLNcor.. In these models, we initially obtain open-world 2D proposals via our open-world world detection module [14]. We then employ different 2D matching methods to identify the proposal with the highest score. OLNDINO and OLNCLIP, we leverage robust fea- tures from pre- rre-trained backbones [9, 19] and use cosine similarity for matching. For OLN designed a 2D matching head using correlation as suggested in [5]. These spen-world detection based 2D baselines significantly outperform previous methods [5, 6]. In addition to these mnstance-spectie methods, we also include a llaes one-shot detector, OS2D [7) and BHRI. [42] for comparison.
The reconstruction stage of VoxDet was trained on a single Nvidia hours, while the detection training phase utilized four Nvidia V100 GPUs for a span of ~40 hours. For the sake of fairess, we trained the methods referenced [5- 14, 19, 9] mainly on the OWID dataset, adhering their official configuration. Inferences were conducted on single V100 GPU to ensure fair efficiency comparison. During testing, we supplied each model with the same set of M the top N = 500 ranking proposals for matching. In the initial reconstruction training stage, VoxDet used 98% of all 9,901 instances in the OWID dataset. For each instance, random set of K images were designated output ZS. while the remaining M - K = 6 images constituted the inputs Z7. For additional configurations of VoxDet, please refer to Appendix and our code.
Hardware and configurations:
V100 GPU over period
template images per instance, and all methods employed
Metrics: Given our assumption that only one desired instance is present in the query image, we default to selecting the Top-1 proposal as the predicted result. We report the average recall (AR) rate [44] across different IoU, such as mAR (IoU e 0.5 ~ 0.95), ARso (IoU 0.5), AR7s (IoU 0.75), and ARgs (IoU 0.95). Note that the AR is equivalent the average precision (AP) in our case.
4.2 Quantitative Results
Overall Performance Comparison: On the synthetic real datasets, we comprehensively compare with all the potential baselines, the results are detailed in Table 1, demonstrating that VoxDet consistently delivers superior performance across most settings. Notably, VoxDet surpasses the
we default the ViT-B model. DINO [9] and CLIP [19] might already be familiar with the test instances.

---

Support Images and AR
Query Images
Cb S M.
WOOO
RODO
VoxDet
OLNomoo
OLNomm Cor
Obi 150 YCD-V
GIGAR
OL.Namo OLNeor
R35
Obi Socooooool
VoxDet
Of Npnonoo OLNeerr
Figure 6: Detection qualitative results comparison between Vex and 2D baselines on the three benchmarks. VoxDet shows better robustness under pose variance (e.g. Obj. 5@ LM-O first and second columns) and occlusion (e.g. Obj. 13@ YCB-V second column and Obj. 9@ RoboTools).
next best baseline, OLNDINO. by impressive margin of up to 20% in terms of average mAR. Furthermore, due to its compact voxel representation, VoxDet observed to be markedly more efficient. On the newly built fully real dataset, RoboTools, we only compare methods trained on same synthetic dataset for fairness. As shown Table 2, VoxDet demonstrates better sim2real transfering capability compared with the 2D methods due to its 3D voxel representation. We present the results comparison with the real-image trained models in Appendix D.
Efficiency Comparison: As QVM has a lower model complexity than OLNCLIP and OLNDINo. achieves faster inference speeds, as detailed in Table 3. Compared correlation-baser matching [5], VoxDet leverages the aggregation of multi-view templates into single compact voxel, thereby eliminating the need for exhaustive 2D correlation and achieving 2 faster speed.
In addition to inference speed, VoxDet also demonstrates greater efficiency regarding the number of templates. We tested the methods the YCB-V dataset [18] using fewer templates than the default. As illustrated in Fig. 4, we found that the 2D baseline highly sensitive to the number of provided references, which may plummet by 87% when the number of templates is reduced from 10 to 2. However, such a degradation rate for VoxDet is 2 x less. We attribute this capability to the learned 2D-3D mapping. which can effectively incorporate 3D geometry with very few views.
Top-K Analysis:
Compared to the category-level method [7], VoxDet produces considerably fewer false positives among its Top-10 candidates. As depicted in Fig. 5, we considered Top- K 1, 1, 10, 20, 30, 50, 100 proposals and compared the corresponding AR between VoxDet and OS2D [7]. VoxDet's 'ss AR only declines by 5 ~ 10% when K decreases from 100 to 10, whereas OS2D's s AR suffers drop up to 38%. This suggests that over 90% of VoxDet's true positives are found among its Top- 10 candidates, whereas this ratio is only around 60% for OS2D.
Ablation Studies:
The results of our ablation studies presented in Table 4. Initially, we attempted to utilize the 3D depth-wise convolution for matching (see the fourth row). However, this proved to be inferior to our proposed instance-level voxel re- lation. Reconstruction pre- pre-training is crucial for VoxDet's ability to learn to encode the geometry of an instance (see the last row). Additionally, we conducted an ablation on the rotation mea- surement module (R) in the QVM, and also tried not supervising the predicted rotation. Both are inferior to our default settings.
Table 4: Ablation study for VoxDet in RoboTools benchmark. All the three critical modules are help- ful in our design. Supervising the estimated rota- tion achieves slightly better results. Comparison with more matching module see Appendix B.
Recon.
R
R w/
Voxel Rel.
mAR
ARoo
AR75
18.7
23.6
20.5
/
x
/
18.2
23.2
20.0
x
x
/
15.6
21.9
17.0
/
x
15.1
19.4
16.2
x
/
14.2
18.3
15.7

---

Support
Query and Voxel Activation
Support
Query and Voxel Activation
Figure 7: Visualization of the high activation grids during matching. As query instance rotates along a certain axis, the location of the high high-activated grids roughly rotates in the corresponding direction.
4.3 Qualitative Results
I
Detection Visualization The qualitative comparison is depicted in Fig. 6, where we compare VoxDet with the two most robust baselines, OLNDINO and OLNCorr.. We notice that 2D methods can easily falter if the pose of an instance is not seen in the reference, e.g., 2-nd query image in the 1-st row, while VoxDet still accurately identi- fies it. Furthermore, 2D matching exhibits less robustness under occlusion, where the instance's is appearance could significantly differ. VoxDet can effectively overcome these challenges thanks to its learned 3D geometry. More visu- alizations and qualitative comparisons see Appendix C.
78
1.5
TS
I
Deep Voxels Visualization To better validate the gcometry-awarenee of our learned voxel representation, we present the deep visualization in Fig. 7. The gradient of the matching score is backpropagated to the template voxel and we visualze the activation value of each grid. Surprisingly, discover that as the orientation of the query instance changes, the activated regions within our voxel representations accurately mirror the true rotation. This demonstrates that the voxel representation in VoxDet is aware of the orientation of the instance. Reconstruction Visualization The voxel representation in VoxDet can be decoded to synthesize novel views, even for unseen instances, which demonstrated in Fig. 8. The voxel, pre-trained on 9500 instances, is capable of approximately reconstructing the geometry of unseen instances.
Figure 8: Reconstruct results of VoxDet on unseen instances. The voxel repre- sentation in VoxDet can be decoded with a relative rotation and synthesize novel views, which demonstrate the geometry embedded our learned voxels.
Discussions
This work introduces VoxDet, a novel approach to detect novel instances using multi- view reference images. VoxDet is pioneering 3D-aware framework that exhibits robustness to occlusions and pose variations. VoxDet's crucial contribution and insight stem from its geometry- aware Template Voxel Aggregation (TVA) module and exhaustive Query Voxel Matching (QVM) specifically tailored for instances. Owing to the learned instance geometry in TVA and the meticu- lously designed matching in QVM, VoxDet significantly outperforms various 2D baselines and offers faster inference speed. Beyond methodological contributions, we also introduce the first instance detection training set, OWID, and challenging RoboTooos benchmark for future research.
Conclusion:
Despite its strengths, VoxDet has two potential limitations, Firstly, the model trained on the synthetic OWID dataset may exhibit domain gap when applied to real-world scenarios, we present details in Appendix D. Secondly, we assume that the relative rotation matrixes and instance masks (box) for the reference images are known, which may not be straightforward to calculate. However, the TVA module Vex doesn't 't require an extremely accurate rotation and 2D appearance. We present further experiments addressing these issues in Appendix E.
Limitations:
10

---

Acknowledgement
This work was sponsored by SONY Corporation of America # #1012409, This work used Bridges-2 at PSC through allocation cis220039p from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program which is supported by NSF grants #2 13825,,, #21 38288, #2138307, #2137603, and #213296. The authors would also like to express the sincere gratitute on the developers of BlenderProc2 [43].
References
[1] B. Li, C. Wang. P. Reddy, S. Kim, and S. Scherer, "AirDet: Few-Shot Detection without Fine-tuning for Autonomous Exploration," in Proceedings of the European Conference on Computer Vision (ECCV), 2022, pp. 427-444.
[2] H. Hu, S. Bai, A. Li, J. Cui, and L. Wang, "Dense Relation Distillation With Context-Aware Aggregation for Few-Shot Object Detection," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021, pp. 10 185 194.
[3] Y. Li. Mao, Girshick, and He, "Exploring Plain Vision Transformer Backbones for Object Detection," " in Proceedings of the European Conference on Computer Vision (ECCV), 2022, pp. 280-296.
[4] I. Misra, R. Girdhar, and A. Joulin, "An End-to-End Transformer Model for 3D Object Detection," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 2906-2917.
[5] Y. Liu, Wen, S. Peng, C. Lin, X. Long, T. Komura, and Wang. "Gen6D: Generalizable Model-Free DoF Object Pose Estimation from RGB Images," in Proceedings of the European Conference on Computer Vision(ECCV), 2022, pp. 298-315.
[6] J.-P. Mercier, M. Garon, Giguere, and J.-F. Lalonde. "Deep Template- based Object Instance Detection," in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2021,
pp. 1507-1516.
[7] A. Osokin, D. Sumin, and V. Lomakin, "OS2D: One-Stage One-Shot Object Detection by Matching Anchor Features," in Proceedings of the Luropean Conference Computer Vision (ECCV), 2020, pp. 635-652.
[8] P. Ammirato, C.-Y. M. Shvets, J. Kosecka, and C. Berg. "Target Driven Instance Detection," arXiv 2018.
preprint arXiv:
[9] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, "Emerging Properties in Seff-supervised Vision Transformers." in Proceedings of the IEEE/CVF international conference computer vision, 2021, pp. 9650-9660
[10] Z. Lai, S. Liu, A. Efros, and Wang, "Video Autoencoder: Self-Supervieed Disentanglement of Static 3D Structure and Motion," in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 9730-9740.
[11] H.-Y.E. Tung, R. Cheng. and K. Fragkiadaki, "Learning Spatial Common Sense with Geometry Aware Recurrent Networks." " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 2595-2603.
[12] T. Dguyen-Phuce C. Li, L. Theis, C. Richardt, and Y.-L. Yang. "HoloGAN: Unsupervised Learning of 3D Representations from Natural Images," " in Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2019, pp. 7588-7588
[13] Oquab, T. Darcet, Moutakanni, H. Vo, M. Szafranice, V. Khalidow, Fernandez, D. Haziza, F. Massa, A. El-Nouby et al., "Dinov2: Learning Robust Visual Features without Supervision," arXiv preprint arXiv:2304.07193. 2023.
[14] D. Kim, T.-Y. Lin, A. Angelova, I. S. Kweon, and W. Kuo, "Learning World Object Proposals without Learning Classify," IEEE Robotics and Automation Letters, vol. no. 2, pp. 5453-5460, 2022.
[15] A. X. Chang. T. Funkhouser. L. Guibas, P. Hanrahan, Q Huang. Z. Li, S. Savarese, M. Savva, S. Song. H. Su et al., "ShapeNet: An Information-Ric 3D Model Repository," arXiv preprint 2015.

---

[16] J. Collins, S. Goel. K. Deng, A. Luthra, L. Xu, E. Gundogdu, X. Zhang, T. F. Y. Vicente. T. Dideriksen. 1. Arora el al., "Abo: Dataset and Benchmarks for Real-World 3D Object Understanding," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 21 126- 21 136.
[17] E. Brachmann, A. Krull, F. Michel, S. Gumhold, Shotton, and C. Rother, "L.carnng 6D Object Pose Estimation using 3D Ohject Coordinates," Proceedings of the European Conference on Computer Vision (ECCV), 2014, pp. 536-551.
[18] B. Calli, A. Singh, A. Walsman, S. Srinivasa, P. Abheel, and A. M. Dollar, "The YCB Object and Model Set: Towards Common Benchmarks for Manipulation Research," in Proceedings of the International Conference on Advanced Robotics (ICAR), 2015, pp. 510-517.
[19] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., "Learning Transferable Visual Models from Natural Language Supervision." in Proceedings of the International Conference Machine Learning (ICML), 2021. pp. 8748 8763.
[20] R. Girshick, J. Donahue, T. Darrell, and J. Malik, "Rich Feature Hierarchics for Accurate Object Detection and Semantic Segmentation," in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), vol. 3, 2014, pp. 580-587.
[21] R. Girshick, "Fast R-CNN," in Proceedings of the IEEF/CVF Conference on Computer Vision ond Pattern Recognition (CVPR), 2015, pp. 1440-1448.
[22] S. Ren, K. He, R. Girshick, and J. Sun, "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks," IEEE Transactions Puttern Analysis and Machine Intelligence, vol. 39, no. pp. 1137 1949, 2016.
[23] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, "You Only Look Onee: Unified, Real-time Object Detection," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779-788.
[24] J. Redmon and A. Farhadi, "YOL00000:: Better, Faster, Stronger," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 7263-7271.
[25]
"YOLOv3: An Incremental Improvement," arXiv preprint arXiv:1804.0276; pp. pp. 1-6, 2018
[26] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. "Feature Pyramid Networks for Object Detection." " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017. pp. 2117-2125.
[27] L. Qiao, Y. Zhao, Z. Li, X. Qiu, J. Wu, and C. Zhang, "DeFRCN: Decoupled Faster R-CNN for Few- Shot Object Detection," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 8681-8690.
[28] X. Wang, Huang. Gonzalez. Darrell, and F. Yu, "Frustratingly Simple Few-Shot Object Detection," in Proceedings of the International Conference on Machine Learning (ICML), 2020, pp. 9919-9928.
[29] Y. Xiao, V. Lepetit, and R. Marlet, "Few-Shot Object Delection and Viewpoint Estimation for Objects in Wild," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 3, pp. 3090-3106, 2022.
[30] K. Joseph, S. Khan, Khan, and N. Balasubramanian. "Towards Open World Object Detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 5830-5840.
[31] A. Gupta. S. Narayan, K. Joseph, S. Khan, F. S. Khan, and M. Shah, "OW- "OW-DETR: Open- World Detection Transformer," in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 9235-9244.
[32] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo et al., "Segment Anything," arXiv preprint arXiv:2304.02643 2023.
[33] J. L. Schonberger and J.M. Frahm, memncturr-from--Moor Revisited," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 4113-4113,
[34] A. W. Harley, S. K. Lakshmikanth, F. Li, X. Zhou, H.-Y. F. Tung, and K. Fragkiadaki. "Learning from Unlabelled Videos Using Contrastive Predictive Neural 3D Mapping," in Proceedings of the International Conference on Learning Representations (ICLR), 2019, pp. 1-19.
12

---

[35] V. Sitzmann, J. Thies. F. Heide. M. NieBner, G. Wetzstein, and M. Zollhofer. "Deepvoxels: Learning Persistent 3D Feature Embeddings."" in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2019. pp. 2437-2446.
[36] B. Mildenhall, P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. "NerF: Representing Scenes as Neural Radiance Fields for View Synthesis," Communications of the ACM, vol. 65, no. 1, pp. 99-106, 2021.
[37] V. Sitzmann, M. Zollhöfer, and G. Wetzstein, "Scene Representation Networks: Continuous 3D-Structure- Aware Neural Scene Representations, in Advances in Neural Information Processing Systems (NeurlPS), vol. 32, 2019.
[38] R. Yang, G. Yang, and X. Wang, "Neural Volumetric Memory for Visual Locomotion Control," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 1-12.
[39] W. Wang. Y. Hu, and S. Scherer. "Tartanvo: A Generalizable Learning-based VO," in Proceedings of the Conference on Robot Learning (CoRL), 2021, pp. 1761-1772.
[40] Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li, "On the Continuity of Rotation Representations in Neural Networks," in Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2019, pp. 5745-5753.
[41] K. Simonyan and A. Zisserman, "Very Deep Convolutional Networks for Large-Scale Image Recognition," " arXiv preprint 14Xiv:1409. 1556, 2014.
[42] H. Yang. S. Cai, H. Sheng. B. Deng, J. Huang, X.-S. Hua, Y. Tang, and Y. Zhang, "Balanced and Hierarchi- cal Relation Learning for One-Shot Object Detection," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 7591-7600.
[43] M. Denninger, M. Sundermeyer, D. Winkelbauer, Y. Zidan, D. Olefir, M. Elbadrawy, A. Lodhi, and H. Katam, "Blenderproe," arXiv preprint arXiv: 10011110111111111111110001111111 1, 2019.
[44] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, "Microsoft COCO: Common Objects in Context," in Proceedings of the European Conference Computer Vision (ECCV), 2014, pp. 740-755.
[45] Y. Xiao and R. Marlet, "Few-Shot Object Detection and Viewpoint Estimation for Objects in the Wild," in Proceedings of the European Conference Computer Vision (ECCV), 2020, pp. 192-210.
[46] K. He, X. Zhang. S. Ren, and J. Sun, "Deep Residual Learning for Image Recognition," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770-778.
[47] D. P. Kingma and J. Ba, "Adam: A Method for Stochastic Optimization," arXiv preprint arXiv: 2014.
[48] P.-E. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich, "SuperGluue Graph Neural Networks," in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. pp. 4938-4947.
Learning Feature Matching with
[49] D. DeTone, T. Mulisiewicz, and A. Rabinovich, "Superpoint: Self-Supervised Interest Point Detection and Description." in Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshop (CVPRW), 2018, pp. 224-236.
[50] J. Henriques, R. Casciro, P. Martins, and J. Batista, "High- Speed Tracking with Kernelized Correlation Filters," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, no. 3, pp. 583 596. 2015.
[51] Li, C. Fu, F. Ding, Z. Huang, and G. Lu, "AatoTrack: Towards High- Performance Visual Tracking for UAV with Automatic Opatio-Tempone Regularization," " Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2020, pp. 11 920-11 929.
[52] B. Li, C. Fu, F. Ding, J. Ye, and F. Lin, "ADTrack: Target- Aware Dual Filter Learning for Real-Time Anti-Dark UAV Tracking," in 2021 IEEE International Conference on Robotics and Automation (ICRA), 2021, pp. 496-502.
[53] B. Li, W. Wu, Q. Wang. F. Zhang, J. Xing, and J. Yan, "SiamRPN++: Evolution of Siamese Visual Tracking With Very Deep Networks." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 4277-4286.
13

---

[54] Z. Cao, C. Ye, B. Li, and Y. Li, "HiFT: Hierarchical Feature Transformer for Aerial Tracking." in Proceedings the IELE/CVF International Conference Computer Vision (ICCV), 2021, pp. 15 457- 15 466.
[55] B. Ye, H. Chang. B. Ma. Shan, and X. Chen, "Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework," in Proceedings of the European Conference on Computer Vision (ECCV), 2022, pp. 341-357.
[56] Y. Cui, C. Jiang, Wang, and G. Wu, "Mixformer: End-to- End Tracking with Iterative Mixed Attention," " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, 13 608 138.8
[57] J. Sun, Z. Wang, S. Zhang. X. He, H. Zhao, G. Zhang, and X. Zhou. "Onepose: One-Shot Object Pose Estimation without CAD Models," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 6825-6833
[58] Y. He. Y. Wang. H. Fan, J. Sun, and Q. Chen, "FS6D: Few-Shot 6D Pose Estimation of Novel Objects." in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 6814-6824.
[59] Q Gu, B. Okom, and D. Held, "OSSID: Online Self- Supervupee Instance Detection by (and for) Posc Estimation," IEEE Robotics and Automation Letters, vol. no. 2. pp. 3022 3029, 2022.
[60] W. Chen, Y. Liu, W. Wang, E. M. Bakker, T. Georgiou, P. Fieguth, L. Liu, and M. S. Lew, "Deep Learning for Instance Retrieval: A Survey," IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[61] A. Babenko and V. Lempitsky, "Aggregating Local Deep Features for Image Retrieval," in Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV), 2015. pp. 1269-1277.
[62] R. Arandjelovic, P. Gronat, A. Torii, T. Pajdla, and J. Sivie, "NetVLAD: CNN Architccture for Weakly Supervised Place Recognition," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016. pp.
[63] T. Ng. V. Balntas, Tian, and Mikolajezyk, "Solar: Second-order loss and attention for image retrieval," in Proceedings of the European Conference on Computer Vision (ECCV), 2020, pp. 253-270.
[64] A. Gordo, Almazán, J. Revaud, Larlus, "Deep Image Reetrieval: Learning Global Representations for Image Search," in Proceedings of the European Conference on Computer Vision (ECCV), 2016, pp. 241-257.
[65] M. Yang. D. Hc, M. Fan, B. Shi, X. Xue, F. Li, E. Ding, and J. Huang, "Dolg: Single-Stage Image Retrieval with Deep Orthogonal Fusion of Local and Global Features," in Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2021, pp. 781.
14

---

Supplementary
To make our model fully reproducible, we present complete implementation details in Appendix A. Besides, our code library will be released upon acceptance. We report more comparisons between our QVM module and the 2D matching/relation techniques [1, 5, 45] in Appendix B to demonstrate the superiority of QVM in instance- level 3D matching. In Appendix C. we present more detection qualitative results. We further present some discussions about the sim2real domain gap of VoxDet in Appendix D. To test the robustness of VoxDet under interference on the voxel representation, we display results obtained from the flawed voxel in Appendix E. Finally, we provide extended related works discussions in Appendix F, where we exhaustively compare VoxDet with the existing instance-level tasks, including visual tracking, instance pose estimation, and instance retrieval.
A Implementation Details
Model Structure: We adopt ResNet50 [46] with feature pyramid network [26] as our feature
extractor v².). The default multi-scale ROIAlign in [26] is leveraged to obtain the 2D proposal features, where the dimensions are N = 500, C' = 256, w 7. In our 2D-3D mapping. we set C/d = 32, d 8. which results in the voxel feature dimension C2 = 256, D = 16, T. 14. All the 3D convolutions in TVA and QVM take kernel size as 3 and the padding equals to 1, that the dimension of the voxels remains the same throughout the two modules. For the Rot(..)) function, we have followed [10] to use worch.nn.funceee onal. (iiigdddineegid() and torch. www.nnnnnniiiiioommmm affunciiinalgrr. ona..grid [ functionalities. Though the 2D-3D mapping can learn the rotations in the physical world, it sacrifices some semantics information in the feature channels when reshaping. Therefore, in QVM, we have a global matching branch retrieve the lost semantic information. To be more specific, we apply global average pooling on the support features to get support vector k I 1 Then we adopt depth-wis, wise convolution between k and F to get a correlation map. Note that this correlation map preserved all the semantic channels from the backbone (c(ot) so that the lost information in the 2D-3D mapping. The map is added to the voxel relation output R. (V',, Rot( R)) for the final score.
Training Details: In the first reconstruction stage, we set the loss weights as Wrecon = 10.00, Wgan 0.01, Wpercop 1.0. The model trained for 16 epoch on the 9600 instances from OWID datasets. We leveraged Adam optimizer [47] with base learning rate of 5 x 10 during training. In the second detection stage, we initialize the 2D-3D mapping modules in TVA and QVM with the reconstruction pre-trained weights. VoxDet first only learns the detection task, without learning the rotation estimation, i.e., the loss weights are set as wh = w2 = W3g = W4 = w3s = 1.0, wgs = 0 in the first 10 epochs, where SGD is leveraged as an optimizer with 0.02 base learning rate. Note that in 1 this stage, the 2D-3D mapping part only takes TO of the base learning rate. Then in the final epoch, VoxDet learns the rotation estimation with the detection part fixed, i.e., w1 w2 w3 W4 W6- = 0.0, We = 1.0. However, supervising rotation is not the key requirements and is optional for VoxDet. It improves the performance slightly by 2%.
B More Matching Module Comparison
We compare QVM with more matching tech- niques in Table 5, where the averaged results onthe cluttered IM-O [17] and RoboTTools benchmark are reported. We first ablate the Voxel Relation module in QVM, which results in QVM Specifically, all the Voxel Relation in QVM are replaced by a simple depth-wise convolution, i.e., we first apply global average pooling on the template voxel to get feature vector, which is then taken as the convolution kernel calculate the correlation voxel from the queries. We can see such a naive design will result in a performance drop.
Table 5: Comparison with different types of match- ing module. We compare QVM with the correlation in [5], class-level relation proposed in [1], and the class distance defined in FSDet [45].
Method
mAR
AR50
AR75
QVM (Ours)
23.95
33.35
26.90
QVM
22.45
31.75
25.05
2D Relation [1]
20.25
29.70
22.80
FSDet [45]
20.35
29.35
22.60
Local Matching [48, 491
10.60
13.90
11.75
For all the rest methods, we used the same apen-world detector to obtain the universal proposals, which are then matched with the template images using different matching techniques. To be
more specific, 2D Corr. [5] constructs support vectors from every reference image. Then, depth- wise convolution is conducted between each support vector and the proposal patch. The resulting
15

---

VoxDet vs DTOID
VoxDet vs Gen6D
VoxDet Vs DINO
LMO
LA
YCB
RELIT
Domino
SIPAM
R33
RoboTools
Figure 9: Detection qualitative results comparison between VoxDet and 2D baselines, DTOID [6]. Gen6D [5], OLNDINO [14, 9] on the three benchmarks. Vex shows better robustness under pose variance and occlusion. These qualitative comparisons can be better visualized our supplementary video.
correlation maps sent MLP for classification score. In 2D Relation [1], substitute the simple depth-wise convolution in 2D Corr. with the spatial and channel relation proposed in [1]. In FSDet [45], the depth-wise convolution in 2D Corr is replaced by the distance defined in [45]. Since they are geometry-unawarc we find all the 2D techniques worse than our QVM module.
Additionally. we designed a Local Matching baseline [49, 48]. In Local Matching, we first extract local key points from the reference images and proposals using SuperPoint [49]. Then points descriptors are matched by SuperGlue [48]. We take the mean matching score of all the points in the proposal as their classification score. We find such an implementation, though geometry-invarian, falls short in task since it lacks semantic representation of the whole instance.
16

---

C More Detection Visualizations
We present more detection qualitative comparisons in Fig. 9. VoxDet, in red, is compared with three baselines, DTOID [6], Gen6D [5], and OLNDINO. Compared with previous instance detectors [6. 5], VoxDet is more robust under orientation variation and severe occlusion by virtue of the learned geometric knowledge. For example, in the LM-O benchmark, second column, when the duck is partially occluded and the egg box is in different orientations, VoxDet can still find them while Gen6D fails. Compared with similarity matching [9], VoxDet can better distinguish similar instances via the QVM module. For instance, in the RoboTools benchmark, the third column, the desired instance could be distracted by the motor, which has similar appearances but different geometry. Our Vox can discover such geometric differences and make correct classification, while the similarity matching falls short even if the feature from DINO [9] is stronger than ResNet50 [46].
D Sim-to-Real Comparison
VoxDet is entirely trained on synthetic dataset. OWID. We observe that the model shows some domain gap when transferred to real-world images like RoboTools. On the synthetic-real datasets, LM-O [17] and YCB-V [18), our model easily outperforms those trained on real images, while it shows limitations in fully real test set RoboTools. For example, Gen6D [5] is mainly trained on real-images, which reports 17.0 mAR, 35.5 ARso, and 14.3 AR7s. Its AR50 is higher than VoxDet (23.6) while in harder metrics like AR75. our model works better (20.5). Compared with the cutting edge foundation models that are trained on large-scale real images, our model still shows spaces for improvement. For example, OLNeLIP achieves 1 11. mAR, 20.8 AR50. and 9.2 AR75. which is worse than VoxDet. Yet, OLN [13] can outperform Vex in RoboTools with over 30 mAR. We conclude that the feature representation from the concurrent 2D foundation model [13] could be stronger backbone for VoxDet to overcome the domain gap issue. Learning geometry-awan strong voxel representation from such foundation model will be one of our future work.
E
Performance under Flawed Voxel
VoxDet assumes known instance masks and poses for the reference video, which may have some noise during realworld deployment. To quantitatively analysis the robustness of VoxDet under flawed Voxels, we present its re- sults on RoboTools when the reference video is disturbed in appearance and geometry.
Table 6: Performance of Vox on RoboTools when
the reference is disturbed. The ratio means center shift and scale noise with respect to width and height.
Mcan Shift Ratio
0
10%
20%
30%
AR50
23.6
20.1
18.9
17.1
Add noise on the reference image patches:
We tried to add random shift on the cropped area in the reference images, resulting in inaccurate instance appearance. The results on RoboTools are shown in Table 6. We conclude that even when we disturb around 65% of the voxel (30% shift on each 2D patch), the model still works, which means VoxDet is robust to appearance noise.
Add noise on the relative poses We tried to add random error the pose of the reference images, resulting in inaccurate instance geometry. When we add as large as 15 degree angular error, the performance (ARso) decreased from 23.6 to 20.4. We conclude that Vox is not very sensitive to the geometry noise.
F Extended Related Works
Visual Object Tracking aims localize general target instance in video, given its initial state in the first frame. Early methods adopt discriminative correlation filters [50-52], where the calculation in the frequency domain is efficient that real-time speed can be achieved on a single CPU. More recently, methods are developed on Siamese Network [53] and Transformers [54-56]. Unlike detection, object tracking has strong temporal consistency assumption, iee,, the location and appearance of the instance in the next frame do not largely vary from the previous frame. So that they only conduct detection/matchingg in the small search region with single 2D template, which can't work for our whole image detection setting.
Instance Pose Estimation is developed to estimate the 6 DoF' pose of an unseen instance. Some of them [57, 58) match the local point features and resort to RANSAC to optimize the relative pose. While others [5, 59] first selects the closest template frame and then conducts pose refinement on the
17

---

known template poses. Most of these methods usually assume the instance detection is perfect. i.e. they crop the instance out of the query image with the ground truth box and estimate the pose on the small object-centered patch. Our VoxDet can serve as their front-end, which is robust to cluttered environments, thus making the detection- pooo estimation framework more reliable.
Instance Retrieval hopes to retrieve a specific instance from large database with a single reference image [60-65]. Some early work extracts local point features from template and query patch for image matching [61, 49], which may suffer from poor discriminative capability. More recent work resorts to the deep neural network for global representation of the instance [62-65], which compared with the features from query images. However, most them construct 2D template features from the reference, that their representation is unaware of the 31) geometry of the instance, which may not be robust under severe pose variation. Besides, instance retrieval methods usually require high-resolution query images for the discriminative features, while the instance in cluttered query image could be in low-resolution, which sets additional barriers to these approaches.
18


Reviews:
[{"review_id": "lxGa7CZ9kA", "reviewer": "Reviewer_7Hv8", "summary": "The paper presents a new 3D geometry-aware instance detection framework, VoxDet, which effectively transforms multi-view 2D images into 3D voxel features using a Template Voxel Aggregation (TVA) module. This representation demonstrates improved resilience to occlusions and pose variations. VoxDet also uses a Query Voxel Matching (QVM) module to estimate relative rotation and compare aligned voxels for instance detection. The proposed framework demonstrates superior performance on various benchmarks including LineMod-Occlusion, YCB-video, and the newly constructed RoboTools.", "questions": "1. It is unclear how effectively VoxDet can detect texture-less objects or objects that have similar appearances but different geometric structures\n2. How easily can VoxDet be adapted to detect instances from different object classes? Generalizability is key in practical applications where objects to be detected may span a wide variety.", "limitations": "A limitation is the model's assumption of the availability of a known relative rotation matrix for reference images. This could be hard to procure in many real-world scenarios.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ VoxDet represents a significant shift from 2D to 3D instance detection, which could potentially handle more complex detection scenarios, particularly in cases of occlusions and varying object poses.\n+ The creation of the OWID dataset and RoboTools benchmark contributes to the research community, providing valuable resources for future work in instance detection.", "weaknesses": "1. The training of VoxDet on synthetic data raises concerns about a potential domain gap when deployed in real-world situations. Real-world conditions such as varied lighting, noise, and complex backgrounds might not be adequately represented in synthetic datasets.\n2. VoxDet's performance is contingent on the accurate synthesis of the 3D voxel by the TVA module. Any inaccuracies in the voxel representation could impact the detection process, potentially leading to incorrect identification or missed detection of instances.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 7Hv8,\n\nWe would like to express our sincere gratitude for your careful inspection and constructive feedback on our paper.\nWe address your specific concerns as follows:\n\n1. **Synthetic training data raises concerns about a potential domain gap**\n   \n   Thanks for this concern. \n   \n   * ***We have compared with the models trained with much larger real-world images*** (which is acknowledged by Reviewer W6QG), *i.e.*, $\\mathrm{OLN}_\\mathrm{DINO, CLIP}$. Though the 2D representations from CLIP and DINO are learned from large-scale high-quality real-world images, VoxDet outperforms them in instance detection due to our innovative and carefully designed voxel learning paradigm.\n   * Besides, ***We have built one real-world dataset*** (RoboTools) with complicated background, realistic lighting, and noise. Our experiments show that among all the methods that are trained with synthetic data, our approach is the least affected by the sim-to-real gap. ***We have added more visualization results under real-world challenges in the Rebuttal PDF for better understanding***.\n   \n   * In addition to VoxDet, we also found that ***other 2D models trained with OWID actually outperforms the ones trained with real-world dataset***. We tried to train Gen6D with COCO and OWID, and the results on YCB-v is listed below.\n   \n       | Training set\\Metric |  mAR   |   AR50   |  AR75   |\n       | :-----------------: | :----: | :------: | :-----: |\n       |        COCO         |  7.5   |   20.8   |   3.4   |\n       |      **OWID**       | **12** | **26.8** | **4.8** |\n   \n     For fairness, we keep the scale of both training set similar (both have ~200k training samples). We conclude that OWID provides instance-tailored data, which compensates for its synthetic nature and finally brings better results in instance detection task. ***We also display some rendering examples in the Rebuttal PDF, where we believe OWID included common real-world challenges.***\n   \n2. **Any inaccurate template voxel  may cause detection failure**\n   \n   We sincerely appreciate this insightful concern.\n   \n   To test the robustness of VoxDet under inaccurate template voxel, we further conducted the following experiments:\n   \n   - ***Add noise on the reference image patches***\n   \n     We tried to add random shift on the cropped area in the reference images, resulting in inaccurate instance appearance. The results on RoboTools are shown below:\n   \n     | Mean Shift Ratio |  0   | 10%  | 20%  | 30%  |\n     | :--------------: | :--: | :--: | :--: | :--: |\n     |       AR50       | 14.0 | 13.7 | 12.8 | 11.5 |\n   \n     The ratio means center shift and scale noise with respect to width and height. We conclude that ***even when we disturb around 65% of the voxel (30% shift on each 2D patch), the model still works***, which means VoxDet is robust to appearance noise.\n   \n   - ***Add noise on relative poses***\n   \n     We tried to add random error on the pose of the reference images, resulting in inaccurate instance geometry. The results on RoboTools are shown below:\n   \n     | Mean Angle Error (degree) |  0   |  5   |  10  |  15  |\n     | :-----------------------: | :--: | :--: | :--: | :--: |\n     |           AR50            | 14.0 | 13.6 | 13.1 | 12.7 |\n   \n     0 means without noise, from 1 to 3, we introduce larger and larger angular error (5 to 15 mean error). We conclude that ***when we disturb 15 degree of the reference poses (which is a very large error in pose estimation), the model drops by only 9%***, which means VoxDet is robust to geometry noise.\n   \n   We will add these experiments and analysis in the final version of our appendix. Thanks again for your constructive feedback.\n   \n3. **Effectiveness on textureless and similar objects**\n\n   Thanks for this question.\n\n   * ***We have included LM-O as one of our test sets, which includes 8 textureless objects (see L240-241)***. VoxDet outperforms other 2D baselines and shows better effectiveness in these textureless objects. See Table 1 for quantitive results and Figure 5, supplementary video, and Rebuttal PDF Figure 2 for qualitative visualizations.\n\n   * ***We tried to introduce similar objects distractions for tes.***, We have displayed some samples in Rebuttal PDF Figure 2, the first row. Under this distraction, our model still works well.\n\n4. **How easily can VoxDet be adapted to detect instances in different classes?**\n\n   We sincerely appreciate this question.\n\n   * In the OWID training set ***we have excluded all the possible overlapping classes during test***, so that the test instances are all unseen classes.\n\n   * Besides, ***VoxDet doesn't need any further tuning or optimization when deployed on a novel instance***.\n\n   Therefore, we believe VoxDet can generalize to unseen instances (classes) effectively and efficiently.\n\n5. **VoxDet has the assumption that the relative poses of template images are known**\n\n   Thanks for pointing this out.\n   \n   * VoxDet is robust to inaccurate poses of the reference images. We tried introducing a 15-degree error on the rotation angle, which only results in a 9% performance drop as shown above.\n\n   * In practice, we can resort to any off-the-shelf SLAM/SfM external modules, making the achievement of the poses easy."}, {"review_id": "ZtOdySS9iK", "reviewer": "Reviewer_6hBZ", "summary": "The paper introduces a method to address the problem of novel instance detection. The key challenge of this problem over object detection is that the network needs to detect novel types of objects that were not in the training data. In contrast to prior art, this paper presents  a 3D geometry-aware framework that is built upon a 3D volume representation. Given a few images of the target instance, It first uses a template voxel aggregation module to transform multi-view information into 3D voxel features. The Query Voxel Matching module then aligns 2D queries with the template voxel. The proposed method outperforms various 2D baselines on LineMod-Occlusion, YCB-video, and RoboTools benchmarks.", "questions": "Please refer to the weaknesses section", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed 3D geometry aware representation has cleared outperform the 2D counterpart by a large margin.\n\n- This paper introduces a synthetic and a real-world datasets or training and evaluation for novel instance detection respectively. The research community can benefit from this new benchmark.", "weaknesses": "- When aligning 3D features of the support templates, why not consider both translation and rotation?\n\n- The authors should make it clear that the proposed method requires posed images, where the camera poses are estimated by an external module, while previous methods use unposed images.\n\n- What is the minimum number of support templates required by the proposed method in the inference stage? It would help readers to understand the performance of the proposed method better if the authors can provide an analysis on how the performance change as the number of support templates increases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 6hBZ,\n\nThanks a lot for providing valuable suggestions and raising insightful questions about our work.\nWe address and answer your questions below:\n\n1. **Why not consider both translation and rotation in the alignment?**\n   \n   Thanks for raising up this issue.\n\n   ***In the TVA module, we assume the cropped image patches are object-centered with little translation***. The reference video is assumed to be rotation-dominant by task setting. Small translations between reference frames can be compensated by the provided rotation. We thus intuitively didn't consider the translation in the template voxel alignment.\n   \n   ***In the QVM module, we have tried to estimate both translation and rotation before alignment***. However, we observe that additionally estimating query translation may not benefit detection tasks. The results on the YCB-v benchmark are shown below:\n   \n   | Translation Estimation |   mAR    |   AR50   |   AR75   |\n   | :--------------------: | :------: | :------: | :------: |\n   |    **No (default)**    | **33.3** | **55.0** | **33.4** |\n   |          Yes           |   29.4   |   47.6   |   32.1   |\n   \n   This could be because:\n    * Training set has different translation scales, which makes learning the translation hard.\n    * Translation scales in training may be different from testing, which sets a barrier to a generalizable translation estimation module.\n\n    We will include more intuitive explanations in our final version supplementary. Thank you again for this constructive feedback.\n\n2. **VoxDet requires posed images, while previous methods use unposed images**\n    \n    Thanks for this concern.\n    \n    * We agree that VoxDet introduced explicit geometry for the detection task, which is different from the previous 2D methods.\n\n    * ***VoxDet doesn't require perfect reference poses***. We tried to introduce noisy estimation on the rotation angle, which only results in a 3-9% performance drop:\n       | Mean Angle Error (degree) |  0   |  5   |  10  |  15  |\n       | :-----------------------: | :--: | :--: | :--: | :--: |\n       |           AR50            | 14.0 | 13.6 | 13.1 | 12.7 |\n       |           Drop            |  0   | 2.8% | 6.4% | 9.3% |\n    \n    * In practice, with the current advancement of SLAM/SfM systems, having posed images become more feasible.\n    \n    We will carefully polish the writing in the paper to avoid confusion.\n\n\n3. **What's the minimum number of support templates required by VoxDet?**\n\n   Thanks for this question.\n\n   * Please kindly refer to L264 in the original submission. ***By default, we uniformly sampled 10 images*** from the 360-degree object-centric video.\n\n   * We also conducted additional experiments, proving fewer images for VoxDet. The results on RoboTools are shown below:\n\n       | Num. Ref | 10 (default) |  8   |  6   |  4   |  3   |  2   |  1   |\n       | :------: | :----------: | :--: | :--: | :--: | :--: | :--: | :--: |\n       |   mAR    |   **16.1**   | 16.2 | 16.0 | 15.9 | 16.0 | 14.9 | 12.3 |\n       |   AR50   |   **14.0**   | 13.9 | 13.9 | 13.7 | 13.7 | 12.9 | 11.2 |\n       |   AR75   |   **11.8**   | 11.9 | 11.7 | 11.7 | 11.8 | 11.0 | 9.3  |\n\n       We conclude that ***given only 3 (uniformly sampled) reference images, VoxDet can still work without obvious performance drop***. We attribute this *template efficiency* to the incorporated explicit geometric information and the implicitly learned 2D-3D mapping. We will add these additional results in the supplementary.\n   * Moreover, it is worth mentioning that, ***unlike VoxDet, the 2D baselines may suffer from fewer reference***. Please see Figure 3 in the paper for more details.\n\n   We will provide these additional experiments and analyses in our final version."}, {"review_id": "iC6H1VKhsw", "reviewer": "Reviewer_W6QG", "summary": "The manuscript introduces VoxDet, a system for novel instance detection. The method takes in a set of query images of an object instance and aims to detect this instance in a target image. The core novelty of the approach lies in the lifting of the query instance and all potential target instances (detected via an open world 2d detector) into a 3D voxel feature grid. Matching between the query voxels and the proposal voxels is carried in 3D instead of 2D (which is the current standard approach). The manuscript that listing the comparison into 3D yields superior novel instance detection performance to staying in 2D.\n", "questions": "see weaknesses", "limitations": "Some limitations are addressed by the authors (sim-to-real gap and rotation for query images). Another important one that I was wondering about is the fact that the training dataset contains 10k object instances. From the writing its not clear how diverse those instances are and hence how much of the space of all possible objects we can expect this to cover?\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper approaches the hard task of novel instance detection from the interesting novel perspective of lifting the instance observations into 3D before the scoring and retrieval. The key intuition is that a fused 3d representation (than also has learned 3D object priors via the reconstruction objective) should be better at detection than pure 2D approaches which are current state of the art. The paper convincingly shows that this intuition holds using ablation studies and comparison to various 2D baselines (including using some of the cutting edge foundational models DINO and CLIP).\n\nThe paper uses high quality visuals to convey the core ideas and qualitative results.\nThe authors promise to release a novel dataset (OWID) and benchmark (RoboTools).\n             \n      \n", "weaknesses": "To me the main weakness is the clarity of the paper in the writing and the figures showing the system. I would not trust myself to be able to reimplement this syste:\nWhile the figures and visuals (1 and 2) are high quality, I did have a hard time following them - a lot is going on!\n- Fig 1: I was initially confused as to what is being Reconstructed or aggregated (its not clear from the figure)\n- Fig 2: Even after reading the text and (hopefully having understood the system) this figure is hard to parse for me.\nThe writing was also in parts hard to follow for me. For example the Voxel Relation paragraphs - It seems voxel relation is about rotating voxels into the same coordinate system to be able to compare and score them?\n\nPart of the contribution of the paper is a novel dataset of rendered objects, as far as I can tell. I would want to see some example renderings of this dataset to be able to judge the quality and potential impact of it.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer W6QG,\n\nWe sincerely appreciate your feedback and concerns.\nWe provided detailed explanations and responses as follows:\n\n1. **Concerns to reimplement this system due to complex framework figures**\n     \n     We sincerely appreciate this concern.\n     \n     - To ensure reproducibility and help readers better understand our system, ***we have provided our code library in the supplementary material during initial submission.***\n     \n     - We promise to release all the code, pre-trained models, and data upon acceptance.\n     \n2. **Figures 1 and 2 are not very clear**\n     \n     Thank you for pointing this out.\n     \n     * Figure 1: We first resort to the reconstruction objective to pre-train the 2D-3D mapping in the Voxel Aggregation module. The aggregation module is then partially initialized from the reconstruction of pre-trained weights.\n     * Figure 2: The first voxel relation is used to estimate query rotation, which is then utilized to rotate voxels into the same coordinate system. The second voxel relation is to compare and score the rotated voxels.\n     \n     Following your valuable advice, we will carefully polish Figures 1 and 2 with more detailed explanations to avoid confusion. In the final version, we'd like to polish our presentation with any further suggestions to help the readers understand our system.\n\n\n3. **Better to display some example renderings from OWID**\n\n   We express our sincere gratitude for this constructive suggestion.\n\n   ***We present a figure in the Rebuttal PDF, displaying representative reference images and test scenes in the OWID dataset.*** We will add this figure to the final version of our supplementary and include a more detailed explanation of the synthetic dataset.\n   \n4. **How diverse is OWID?**\n   \n   Thanks for this question.\n   \n   * The 10k instances in OWID are sampled from the combination of ShapeNet and ABO datasets. Specifically, for each class in ShapeNet (47 classes excluding the classes appearing in the test), we randomly sampled 60 CAD models, resulting in a total of around 3k instances. We used all of the 8k available instances in the ABO dataset. These data together contribute to the exact number of instances in OWID, 10,860. We will follow your kind advice and provide these details in our final supplementary.\n   \n   * ***From our qualitative and quantitative experiments, VoxDet can generalize well to unseen instances (w/ unseen class label) using these data.***\n   \n   * In the future, we are also planning to incorporate the most recent large-scale 3D CAD model dataset, *e.g.*, objaverse or OmniObject3D, for a larger instance-level training set."}, {"review_id": "QRb0DViTUh", "reviewer": "Reviewer_dtGe", "summary": "This paper tackles the task of novel instance detection when given multi-view reference images.\nThe key is the utilization of a 3D voxel representation, contrary to 2D representations used by prior work that fair under occlusion and pose variation.\nThe proposed detector, VoxDet, combines reference features into a 3D template voxel through the Template Voxel Aggregation module and compares that to voxels similar generated from object proposals generated from a RPN through the Query Voxel Matching module.\nBy aligning and comparing the resulting voxels, the detections are more robust to pose variations than existing methods.\nExtensive experimental results are shown on existing benchmarks as well as the newly proposed RoboTools benchmark, which contains much more clustered scenes to test existing methods.\nFurthermore, a synthetic training dataset, Open-World Instance Detection, is also introduced, which contains numerous training examples sourced from object datasets.", "questions": "See weaknesses section.\n\nWriting:\n- Line 197: Typo \"haed\" -> \"head\".\n- Line 246: Typo \"og\" -> \"of\".", "limitations": "The paper discusses several limitations of the proposed method.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The underlying motivation of the method makes intuitive sense. By leveraging a 3D representation of every object, it is much more robust to changes in viewpoint, which 2D methods struggle with.\n- The proposed method achieves significantly better performance compared to previous methods across all benchmarks, even when compared to their own strong baselines.\n  - Qualitatively, the provided visualizations and the video in the supplementary clearly demonstrates the improvement in robustness in detection.\n- The paper also introduces both a large-scale training dataset and a new evaluation benchmark, both of which are good contributions to the field of instance detection.\n- The paper is very well written and easy to understand.", "weaknesses": "- The training process requires rotation to be known for the given reference images, which may not be available and are also not used by other methods.\n  - The results in Table 3 seems to suggest that without supervising rotation, the performance drops by 1 point in mAR, which is quite significant (though still higher than other methods).\n  - Can the authors explain the \"R\" and \"R w/ sup\" column in Table 3? This is not well explained.\n- Comparison to 2D baselines:\n  - Are the previous 2D methods trained with the open-world detector used by the proposed method?\n  - What is the pre-training procedure of these baselines on the OWID dataset? Comparison can be made without pre-training to compare the performance without the OWID dataset.\n  - The paper lacks comparisons to more recent few-shot/one-shot detection methods, such as [1, 2, 3].\n\n[1] Yang, Hanqing, et al. \"Balanced and hierarchical relation learning for one-shot object detection.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n[2] Qiao, Limeng, et al. \"Defrcn: Decoupled faster r-cnn for few-shot object detection.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n[3] Zhang, Gongjie, et al. \"Meta-DETR: Image-level few-shot object detection with inter-class correlation exploitation.\" arXiv preprint arXiv:2103.11731 (2021).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer dtGe,\n\nThank you very much for the questions and comments. We address them as follows:\n\n1. **VoxDet requires known rotations of the references during training.**\n\n     Thanks for raising this concern.\n\n     * It is true that most 2D methods didn't consider the geometry information during training. Our intuition is that humans understand/remember one specific instance not only through its appearance but also via its geometry. The geometry and shape can help the search task under occlusion or pose variation challenges. Motivated by this, we have incorporated explicit geometry to construct a compact instance voxel, which enables more effective and efficient detection.\n     \n     * We would like to mention that ***in OWID synthetic dataset, the geometry information is readily available to make the system better***, while other 2D methods can't cooperate with such information due to their network design.\n     \n     * ***Durining inference, VoxDet doesn't need perfect reference poses***. We tried to introduce errors in rotation angle, which only results in a 3-9% performance drop:\n       | Mean Angle Error (degree) |  0   |  5   |  10  |  15  |\n       | :-----------------------: | :--: | :--: | :--: | :--: |\n       |           AR50            | 14.0 | 13.6 | 13.1 | 12.7 |\n       |           Drop            |  0   | 2.8% | 6.4% | 9.3% |\n       \n     * Moreover, we could resort to any off-the-shelf SLAM/SfM system to obtain the reference poses during inference.\n     \n2. **More Explanations to Table 3**\n\n     Thanks for raising this up.\n\n     - In Table 3, \"R\" means \"measuring query rotation in QVM w/o supervision\", and \"R w/ supervision\" means \"measuring query rotation in QVM w/ GT supervision\".\n     \n     - From Table 3, we have the following conclusions\n       1. ***Though w/o supervision, the structure design \"Measuring R\" in QVM can improve the performance of VoxDet, see last and second last row, AR50 and AR75***. Intuitively, when we match the instance in 3D, we would first roughly estimate its relative pose to better compare the appearance. While such improvements are not very robust in simpler metrics (mAR, which means IoU thresh from 0.25 to 0.95).\n       1. ***\"Rotation supervision\" can further improve the robustness of QVM, see the last and the first row.*** We by default used supervision since the ground truth is readily available in our synthetic dataset. And the result suggests that supervising this intermediate value (rotation) is better since we give the model more guidance for performing the appropriate transform. Yet, without supervision, our system still achieves SOTA performance.\n     \n\n      We will carefully polish our writing and table captions in our final version to avoid confusion. We'd also like to answer any further questions or discussions.\n     \n3. **Details about the 2D baselines**\n\n     - $\\mathrm{OLN}_\\mathrm{CLIP, DINO, Corr.}$ are our self-crafted baselines, ***they all used the same open-world detection model***. For the other baselines, we have used the official code library and model structure without modification.\n     \n     - For the baselines $\\mathrm{OLN}_\\mathrm{CLIP, DINO}$, we pre-train the open-world detection module and directly used the off-the-shelf foundation models' feature for cosine-similarity-based matching. We believe that the large-scale real-image base training has enabled strong representation from the foundation models. So that they are not tuned on OWID synthetic dataset in our experiments.\n     \n     - The others are entirely trained on OWID for fairness, we have followed their official code and tried our best to reproduce the highest score. We also tried directly using their official pre-training, but the results are worse. For example, the results for Gen6D are shown below:\n     \n       | Training set\\Metric |  mAR   |   AR50   |  AR75   |\n       | :-----------------: | :----: | :------: | :-----: |\n       |        COCO         |  7.5   |   20.8   |   3.4   |\n       |      **OWID**       | **12** | **26.8** | **4.8** |\n     \n     We will provide this detailed information in the final version of our supplementary.\n     \n4. **Comparison to more recent few/one-shot detectors**\n   \n   Thank you for this constructive suggestion.\n   \n   - We agree to include BHRL and Meta-DETR in the experimental comparison since they share a similar pipeline (base training + inference) and siamese network structure with VoxDet.\n   \n     The comparison on the LM-O dataset is shown below, where ***VoxDet outperforms these class-level detectors by a larger margin***.\n   \n   \n     | Model |  VoxDet  | BHRL | Meta-DETR |\n     | :---: | :------: | :--: | :-------: |\n     |  mAR  | **27.4** | 14.1 |   12.0    |\n     | AR50  | **36.9** | 22.8 |   15.3    |\n     | AR75  | **26.4** | 14.6 |   11.6    |\n   \n   DeFRCN is a transfer-learning-based few-shot detector, it requires exhaustive fine-tuning on the few reference images, which is inefficient for instance-search tasks. Besides, we find it hard (and unfair) to adjust the fine-tuning hyper-parameters on our new dataset. We thus didn't consider it in our experiments.\n   \n   We will include BHRL and Meta-DETR in our final version."}, {"review_id": "TaobXajKPW", "reviewer": "Reviewer_NFHY", "summary": "The paper introduces a new 3D geometry-aware framework, VoxDet, to address the challenge of detecting unseen instances based on multi-view templates. VoxDet includes three main modules: a Template Voxel Aggregation (TVA) module, an open-world detection module, and a Query Voxel Matching (QVM) module. TVA transforms multi-view 2D features to 3D template voxel feature and then accumulated according to pre-computed relative rotation. During detection, the detected 2D proposals are converted to query voxels through the learned 2D-3D mapping. QVM module estimates the relative rotation between the query and template, aligns them and performs a comparison.\n\nAlong with the introduction of this method, the authors also build a large-scale synthetic training dataset, Open-World Instance Detection (OWID), made from the ShapeNet and some objects datasets. Experiments on LineMod-Occlusion, YCB-video dataset. ", "questions": "The manuscript presents several intriguing concepts and experimental results. However, I have a few queries that need addressing for a more comprehensive understanding of the study.\n\n1. While the paper extensively compares methods developed by the authors (as mentioned in lines 261-262), it raises the question of whether this is an entirely novel experimental setup. Would it be feasible to compare other open-world methods without necessarily using the same training data? This query stems from the desire to assess the actual impact of the proposed method across various techniques and its potential real-world applications.\n2. I noticed that the supplementary material does not appear to include experiments testing robustness against accurately estimated rotation, as claimed in lines 341-343. Could you provide more details or point me towards this information?\n3. I am curious whether the OWID and RoboTools benchmarks will be made publicly available to aid future research.\n4. In line 192, the text mentions the experiment that benefits from the group kernel design. I am keen to understand why and how this group gets the advantage.", "limitations": "Yes.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. As I am not directly working on this research topic, the main text give enough context knowledge to understand the overall method. The overall writing are smooth and easy to understand.\n2. As claimed by the author, VoxDet is the first to employ the explicit 3D knowledge in the multi-view templates to perform open-world detection in the object-level household datasets.\n3. In my view, the OWID dataset effectively complements existing household object datasets. It specifically addresses areas typically overlooked such as accommodating pose variations and occlusions of household objects, thereby enhancing our understanding and performance in these complex scenarios.\n4. Experimental results show that the 3D geometry-aware VoxDet outperforms various previous works and different 2D baselines while achieving faster inference speed.\n", "weaknesses": "No obvious weakness in my view. See question section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer NFHY,\n\nThanks for your insightful questions and feedback. We provide specific answers below:\n\n1. **Is this a novel experiment setup?**\n     \n     Thanks for this question.\n     \n     Yes, this is a novel setup, we have developed new training data (OWID) and test benchmarks (authoritative data from LM-O, YCB-V, and newly built RoboTools) for the instance detection task. Gen6D, OS2D, DTOID, and $\\mathrm{OLN}\\_\\mathrm{Corr.}$ are entirely trained on our OWID dataset. \n    For $\\mathrm{OLN}\\_\\mathrm{CLIP, DINO}$, we use OWID to train the open-world detector and resort to an off-the-shelf pre-trained foundation model for matching.\n     \n2. **Would it be feasible to compare other open-world methods without necessarily using the same training data?**\n     \n     Thank you for this question.\n     \n     It is feasible. \n     * For Gen6D, OS2D, $\\mathrm{OLN}\\_\\mathrm{Corr.}$, and DTOID, we have tried directly using their official model trained on their official 2D dataset, while it performs worse than re-training with our new OWID dataset, e.g., see the Gen6D results on YCB-v dataset as follows:\n     \n          | Training set\\Metric |  mAR   |   AR50   |  AR75   |\n          | :-----------------: | :----: | :------: | :-----: |\n          |        COCO         |  7.5   |   20.8   |   3.4   |\n          |      **OWID**       | **12** | **26.8** | **4.8** |\n\n        We believe OWID is a novel dataset focused on instance detection. We have tried our best to re-train the models to reported their highest score in the comparison.\n     \n     * For $\\mathrm{OLN}\\_\\mathrm{DINO}$ and $\\mathrm{OLN}\\_\\mathrm{CLIP}$, we try our best to be fair by using the same open-world detector. Since we believe their pre-trained 2D representation from the large real-world dataset is good enough, we only used synthetic OWID to pre-train the open-world detection module and utilized the off-the-shelf CLIP and DINO models to match the reference images and proposals.\n     \n     To summarize, we have tried our best to ensure fairness in all experiments. We conclude that VoxDet outperforms these baselines by virtue of the innovative incorporation of explicit 3D geometry. We will provide these details in the final version supplementary.\n\n\n3. **More details about the robustness against inaccurate rotation estimation**\n     \n     Thanks very much for raising this issue.\n     \n     We tried to introduce random errors in the rotation angle of the reference poses. The results on RoboTools are shown below:\n     \n     | Mean Angle Error (degree) |  0   |  5   |  10  |  15  |\n     | :-----------------------: | :--: | :--: | :--: | :--: |\n     |           AR50            | 14.0 | 13.6 | 13.1 | 12.7 |\n     \n     We conclude that ***adding as large as a 15-degree error will not cause a significant performance drop***.\n     \n     We will include these results in the final version supplementary. We'd like to thank you again for the careful inspection of our work.\n     \n4. **Will the dataset be publicly available?**\n    \n    Yes, we promise to open source all the data, code, and models upon acceptance. (We have appended our code in the supplementary to ensure reproducibility)\n\n\n5. **Detailed explanation of the group convolution**\n    \n    Thanks for this technical question.\t\t\n    \n    Please check L187-193 in our original submission, the combination of channel interleave and grouped convolution kernel is the key innovation of our QVM module, which effectively compares two voxels on the instance level.\n    \n    Our key finding is that, when the model learns each corresponding channel independently, it better discovers the similarity and differences between two instance voxels.\n    \n    To achieve this, we first interleave the feature channels of two voxels $\\mathbf{v}_1,\\mathbf{v}_2\\in\\mathbb{R}^{c\\times a\\times a\\times a}$ so that the corresponding channels are \"grouped\" together as $\\mathrm{In}(\\mathbf{v}_1, \\mathbf{v}_2)=[\\mathbf{v}^1_1, \\mathbf{v}^1_2, \\mathbf{v}^2_1, \\mathbf{v}^2_2, \\cdots, \\mathbf{v}^c_1, \\mathbf{v}^c_2]\\in \\mathbb{R}^{2c\\times a\\times a\\times a}$, where $\\mathbf{v}^k_1, \\mathbf{v}^k_2$ is the voxel feature in the $k$-th channel. \n    Then in the grouped convolution $\\mathrm{Conv3D}(\\mathrm{In}(\\mathbf{v}_1, \\mathbf{v}_2), \\mathrm{group}=c)$, each kernel only learns to distinguish the same corresponding channels from the two voxels. \n    \n    To validate this, we have tried using the simple standard 3D convolution to replace voxel relation, the averaged results on LM-O and YCB-v benchmarks are shown below:\n    \n    |        Impementation         |   mAR    |   AR50   |   AR75   |\n    | :--------------------------: | :------: | :------: | :------: |\n    | **Voxel Relation (default)** | **21.7** | **25.4** | **19.1** |\n    |      Standard 3D Conv.       |   16.5   |   19.2   |   14.3   |\n    \n    We will include more explanations and results in our final paper."}]

Author rebuttal:
Dear AC, Reviewers,

We would like to express our sincere gratitude for the careful inspection and constructive feedback from all the reviewers and the area chair. We are glad to see that **all of the reviewers in general hold a positive attitude** towards our paper in the pre-rebuttal period.

For positive comments, ***significant shift from pure 2D to 3D-aware detection*** (7Hv8, W6QG, NFHY), ***reasonable motivation and intuition*** (dtGe, W6QG), ***significantly better performance*** (6hBZ, W6QG, dtGe, NFHY), ***well-written pape***r (dtGe, NFHY), and ***favorable OWID and RoboTools datasets*** (All), we appreciate them and will carry them forward in our future work.

For negative comments, we first summarize and address the most general concern here:

1. **VoxDet assumes posed reference images**

    * We agree that different from previous 2D methods, VoxDet incorporates explicit 3D geometric information for instance representation and matching. We believe this is one of the innovations of our system.

    * Yet, we also hope to mention that ***VoxDet is robust to inaccurate poses of the reference images***. We tried to add as large as a 15-degree mean error on the rotation angle, which only results in a 9% performance drop as shown below:

       | Mean Angle Error (degree) |  0   |  5   |  10  |  15  |
       | :-----------------------: | :--: | :--: | :--: | :--: |
       |           AR50            | 14.0 | 13.6 | 13.1 | 12.7 |
       |           Drop            |  0   | 2.8% | 6.4% | 9.3% |

    * In practice, we can resort to SLAM/SfM systems (which are already embedded in many smart phones), making the availability of the poses easy.

    We will include the additional experiments in our final version and better polish our writing to avoid confusion.
    
1. **Details and fairness about the 2D baselines**

    - For Gen6D, OS2D, $\mathrm{OLN}_\mathrm{Corr.}$, and DTOID, we tried our best to re-train them using our OWID dataset for fairness. We have also tried directly using their official model trained on their official 2D dataset, while it performs worse than re-training with our new OWID dataset, e.g., see the Gen6D results on LM-O dataset as follows:
      
      | Training set\Metric |  mAR   |   AR50   |  AR75   |
    	| :-----------------: | :----: | :------: | :-----: |
      |        COCO         |  7.5   |   20.8   |   3.4   |
      |      **OWID**       | **12** | **26.8** | **4.8** |
      
    - For $\mathrm{OLN}_{\mathrm{DINO, CLIP}}$  we believe their pre-trained 2D representation from the large real-world dataset is good enough. So that we only used synthetic OWID to pre-train the open-world detection module and utilized the off-the-shelf CLIP and DINO models to match the reference images and proposals.
      
    - Following Reviewer deGE's valuable suggestion, ***we also included additional SOTA one/few-shot detectors, BHRL and Meta-DETR***, in our experiments. The comparison on the LM-O dataset is shown below, where VoxDet outperforms these class-level detectors by a large margin.
      
       | Model |  VoxDet  | BHRL | Meta-DETR |
       | :---: | :------: | :--: | :-------: |
       |  mAR  | **27.4** | 14.1 |   12.0    |
       | AR50  | **36.9** | 22.8 |   15.3    |
       | AR75  | **26.4** | 14.6 |   11.6    |
    
    To summarize, we have tried our best to ensure fairness in all experiments. We conclude that VoxDet outperforms these baselines by virtue of the innovative incorporation of explicit 3D geometry and reconstruction primitives. We will provide these details and experiments in our final version.

For the other specific concerns, we have tried to address them with detailed explanations, quantitative results, and qualitative visualizations (**in the Rebuttal PDF**) under each reviewer's comments area.

The authors are also open and ready for any further questions/suggestions in the discussion period.

Meta-review: The paper received positive ratings from all reviewers. It introduces an interesting 3D-aware instance detection method that is novel and demonstrates the promising properties of 3D-aware representations for detection, particularly in the context of novel instance detection
Decision: Accept (spotlight)