SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Bowen Li
Jiashun Wang
Yaoyu Hu
Chen Wang2
Sebastian Scherer
1 Carnegie Mellon University 2 State University of New York at Buffalo
Detecting unseen instances based on multi-view templates is a challenging problem due to its open-world nature. Traditional methodologies, which primarily rely on 2D representations and matching techniques, are often inadequate in handling pose variations and occlusions. To solve this, we introduce VoxDet, a pioneer 3D geometry-aware framework that fully utilizes the strong 3D voxel representation and reliable voxel matching mechanism. VoxDet first ingeniously proposes template voxel aggregation (TVA) module, effectively transforming multi-view 2D images into 3D voxel features. By leveraging associated camera poses, these features are aggregated into a compact 3D template voxel. In novel instance detection, this voxel representation demonstrates heightened resilience to occlusion and pose variations. We also discover that a 3D reconstruction objective helps to pre-train the 2D-3D mapping in TVA. Second, to quickly align with the template voxel, VoxDet incorporates a Query Voxel Matching (QVM) module. The 2D queries are first converted into their voxel representation with the learned 2D-3D mapping. We find that since the 3D voxel representations encode the geometry, we can first estimate the relative rotation and then compare the aligned voxels, leading to improved accuracy and efficiency. In addition to method, we also introduce the first instance detection benchmark, RoboTools, where 20 unique instances are video-recorded with camera extrinsic. RoboTools also provides 24 challenging cluttered scenarios with more than 9k box annotations. Exhaustive experiments are conducted on the demanding LineMod-Occlusion, YCB-video, and RoboTools benchmarks, where VoxDet outperforms various 2D baselines remarkably with faster speed. To the best of our knowledge, VoxDet is the first to incorporate implicit 3D knowledge for 2D novel instance detection tasks. Our code, data, raw results, and pre-trained models are public at https://github.com/Jaraxxus-Me/VoxDet.
Consider the common scenarios of locating the second sock of a pair in a pile of laundry or identifying luggage amid hundreds of similar suitcases at an airport. These activities illustrate the remarkable capability of human cognition to swiftly and accurately identify a specific instance among other similar objects. Humans can rapidly create a mental picture of a novel instance with a few glances even if they see such an instance for the first time or have never seen instances of the same type. Searching for instances using mental pictures is a fundamental ability for humans, however, even the latest object detectors [1-7] still cannot achieve this task.
We formulate the above tasks as novel instance detection, that is identification of an unseen instance in a cluttered query image, utilizing its multi-view support references. Previous attempts mainly work in 2D space, such as correlation [8, 5], attention mechanisms [6], or similarity matching [9], thereby localizing and categorizing the desired instance, as depicted in Fig. 1 gray part. However, these techniques struggle to maintain their robustness when faced with significant disparities between the query and templates. In comparison to novel instance detection, there is a vast amount of work centered around few-shot category-level object detection [7, 1, 2]. Yet, these class-level matching techniques prove insufficient when it comes to discerning specific instance-level features.
Figure 1: Architecture comparison between previous 2D methods (gray) and proposed VoxDet (black). Previous methods resorts to pure 2D correlation/attention/matching for novel instance detection. In contrast, VoxDet is 3D-inspired, leveraging reconstruction objective to learn the geometry-aware voxel representation, which enables more effective and accurate voxel-based instance detection. In the challenging newly built RoboTools benchmark shown on the right, VoxDet exhibits surprising robustness to severe occlusion and orientation variation.
Humans exhibit the remarkable capability to swiftly formulate a mental model of an unfamiliar instance, facilitated by a rapid comprehension of its 3D geometric structure [10-12]. Leveraging such a mental representation, once presented with a single query image, a human can probably search and identify the same instance despite alterations in distance, occlusion, and even approximate the instance's orientation. Motivated by this, we propose VoxDet, a pioneer 3D geometry-aware instance detection framework as shown in Fig. 1 bottom. In contrast to state-of-the-art methods [7, 5, 6, 9, 13], VoxDet adapts two novel designs: (1) a compact 3D voxel representation that is robust to occlusion and pose variations and (2) an effective voxel matching algorithm for identifying instances.
VoxDet consists of three main modules: a template voxel aggregation (TVA) module, an open-world detection module, and a query voxel matching (QVM) module. Initially, the TVA module transforms multi-view 2D features of an instance into individual 3D template voxels [10]. These template voxels are then accumulated using relative rotations, thus incorporating both geometry and appearance into a condensed template voxel. As VoxDet learns this 2D-3D mapping via a reconstruction objective, TVA effectively encapsulates both the geometry and appearance of any instance into a compact template voxel. When presented with a query image, VoxDet employs an open-world detector [14] that universally identifies potential objects within the image as 2D proposals. These proposals are then converted to query voxels via the learned 2D-3D mapping and compared with the template voxel by the QVM module. QVM initiates this comparison process by first estimating the relative rotation between a query voxel and the template, which is then used to align the two voxels. Finally, the comparison between aligned voxels is delivered by a carefully designed voxel relation module.
Besides methodology, we also construct a large-scale synthetic training dataset, Open-World Instance Detection (OWID). OWID comprises 10k instances sourced from the ShapeNet [15] and Amazon Berkeley Objects [16] datasets, culminating in 55k scenes and 180k query bounding boxes. Trained on OWID, VoxDet demonstrates strong generalization ability on novel instances, which we attribute to the meticulously designed voxel-based framework and the large-scale OWID training set.
To validate VoxDet, we further build RoboTools, a new instance detection benchmark compiled from a diverse range of real-world cluttered environments. RoboTools consists of 20 unique instances, 24 test scenes, and over 9,000 annotated bounding boxes. As shown in Fig. 1 right, in the demanding RoboTools benchmark, VoxDet can robustly detect the novel instances under severe occlusion or varied orientation. Evaluations are also performed on the authoritative Linemod-Occlusion [17] and YCB-video [18] for more compelling results. The exhaustive experiments on these three benchmarks demonstrate that our 3D geometry-aware VoxDet not only outperforms various previous works [5-7] and different 2D baselines [19, 9] but also achieves faster inference speed.
Typical object detection [20-26] thrive in category-level tasks, where all the instances belonging to a pre-defined class are detected. Typical object detection can be divided into two-stage approaches and
one-stage approaches. For the former one, RCNN [20] and its variants [21, 22] serves as foundations, where the regions of interest (ROI) are first obtained by the region proposal network. Then the detection heads classify the labels of each ROI and regress the box coordinates. On the other hand, theYOLO series [23-25] and recent transformer-based methods [4, 3] are developing promisingly as the latter stream, where the detection task is tackled as an end-to-end regression problem.
Few-shot/One-shot object detection [1, 27, 28, 2, 29, 7] can work for unseen classes with only a few labeled support samples, which are closer to our task. One stream focuses on transfer-learning techniques [28, 27], where the fine-tuning stage is carefully designed to make the model quickly generalize to unseen classes. While the other resorts to meta-learning strategies [1, 7, 2, 29], where various kinds of relations between supports and queries are discovered and leveraged. Since the above methods are category-level, they assume more than one desired instances exist in an image, so the classification/matching designs are usually tailored for Top-100 precision, which is not a very strict metric. However, they can easily fail in our problem, where the Top-1 accuracy is more important.
Open-world/Zero-shot object detection [30-32, 14] finds any objects on an image, which is class-agnostic and universal. Some of them learn objectiveness [30, 14] and others [32] rely on large-scale high-quality training sets. These methods can serve as the first module in our pipeline, which generates object proposals for comparison with the templates. Among them, we adopt [14] with its simple structure and promising performance.
Instance detection requires the algorithm to find an unseen instance in the test image with some corresponding templates. Previous methods [6, 5, 8] usually utilize pure 2D representations and 2D matching/relation techniques. For example, DTOID [6] proposed global object attention and a local pose-specific branch to predict the template-guided heatmap for detection. However, they easily fall short when the 2D appearance varies due to occlusion or pose variation. Differently, VoxDet leverages the explicit 3D knowledge in the multi-view templates to represent and match instances, which is geometry-invariant.
Multi-view 3D representations Representing 3D scenes/instances from multi-view images is a long-standing problem in computer vision. Traditional methods resort to multi-view geometry, where structure from motion (SfM) [33] pipeline has enabled joint optimization of the camera pose and 3D structure. Modern methods usually adopts neural 3D representations [34, 11, 35-37, 12, 10], including deep voxels [35, 12, 10, 38] and implicit functions [36, 37], which have yielded great success in 3D reconstruction or novel view synthesis. Our framework is mainly inspired by Video Autoencoder [10], which encodes a video by separately learning the deep implicit 3D structure and the camera trajectory. One biggest advantage of [10] is that the learned Autoencoder can encode and synthesize test scenes without further tuning or optimization, which greatly satisfies the efficiency requirement of our instance detection task.
Given a training instance set  $\mathcal{O}_{\mathrm{base}}$  and an unseen test instance set  $\mathcal{O}_{\mathrm{novel}}$ , where  $\mathcal{O}_{\mathrm{base}} \cap \mathcal{O}_{\mathrm{novel}} = \phi$ , the task of novel instance detection (open-world detection) is to find an instance detector trained on  $\mathcal{O}_{\mathrm{base}}$  and then detect new instances in  $\mathcal{O}_{\mathrm{novel}}$  with no further training or finetuning. Specifically, for each instance, the input to the detector is a query image  $\mathcal{I}^{\mathrm{Q}} \in \mathbb{R}^{3 \times W \times H}$  and a group of  $M$  support templates  $\mathcal{I}^{\mathrm{S}} \in \mathbb{R}^{M \times 3 \times W \times H}$  of the target instance. The detector is expected to output the bounding box  $\mathbf{b} \in \mathbb{R}^4$  of an instance on the query image. We assume there exists exactly one such instance in the query image and the instance is located near the center of the support images.
The architecture of VoxDet is shown in Fig. 2, which consists of an open-world detector, a template voxel aggregation (TVA) module, and a query voxel matching (QVM) module. Given the query image, the open-world detector aims to generate universal proposals covering all possible objects. TVA aggregates multi-view supports into a compact template voxel via the relative camera pose between frames. QVM lifts 2D proposal features onto 3D voxel space, which is then aligned and matched with the template voxel. In order to empower the voxel representation with 3D geometry, we first resort to a reconstruction objective in the first stage. The pre-trained models serve as the initial weights for the second instance detection training stage.
Figure 2: Architecture of VoxDet. VoxDet mainly consists of three modules, namely, open-world detection, template voxel aggregation (TVA), and query voxel matching (QVM). We first train TVA via the reconstruction stage, where the 2D-3D mapping learns to encode instance geometry. Then the pre-trained mapping serves as initial weights in the TVA and QVM modules for detection training.
3.2.1 Open-World Detection
Since the desired instance is unseen during training, directly regressing its location and scale is non-trivial. To solve this, we first use an open-world detector [14] to generate the most possible candidates. Different from standard detection that only finds out pre-defined classes, an open-world detector locates all possible objects in an image, which is class-agnostic.
As shown in Fig. 2, given a query image  $\mathcal{I}^{\mathrm{Q}}$ , a 2D feature map  $\mathbf{f}^{\mathrm{Q}}$  is extracted by a backbone network  $\psi(\cdot)$ . To classify each pre-defined anchor as foreground (objects) or background, the region proposal network (RPN) [22] is adopted. Concurrently, the boundaries of each anchor are also roughly regressed. The resulting anchors with high classification scores are termed region proposals  $\mathbf{P} = [\mathbf{p}_1, \mathbf{p}_2, \dots, \mathbf{p}_N] \in \mathbb{R}^{N \times 4}$ , where  $N$  is the number of proposals. Next, to obtain the features  $\mathbf{F}^{\mathrm{Q}}$  for these candidates, we use region of interest pooling (ROIAlign) [22],  $\mathbf{F}^{\mathrm{Q}} = \mathrm{ROIAlign}(\mathbf{P}, \mathbf{f}^{\mathrm{Q}}) \in \mathbb{R}^{N \times C \times w \times w}$ , where  $C$  denotes channel dimensions and  $w$  is the spatial size of proposal features. Finally, we obtain the final classification result and bounding box by two parallel multi-layer perceptrons (MLP), known as the detection head, which takes the proposal features  $\mathbf{F}^{\mathrm{Q}}$  as input, and outputs the binary classification scores and the box regression targets. The training loss is comprised of RPN classification loss  $\mathcal{L}_{\mathrm{cls}}^{\mathrm{RPN}}$ , RPN regression loss  $\mathcal{L}_{\mathrm{reg}}^{\mathrm{RPN}}$ , head classification loss  $\mathcal{L}_{\mathrm{cls}}^{\mathrm{Head}}$ , and head regression loss  $\mathcal{L}_{\mathrm{reg}}^{\mathrm{Head}}$ .
To make the detector work for open-world objects, the classification branches (in RPN and head) are guided by objectiveness regression [14]. Specifically, the classification score is defined (supervised) by Intersection over Union (IoU), which showed a high recall rate over the objects in test images, even those unseen during training. Since they have learned the class-agnostic "objectiveness", we assume the open-world proposals probably cover the desired novel instance. Therefore, we take the top-ranking candidates and their features as the input of the subsequent matching module.
3.2.2 Template Voxel Aggregation
To learn geometry-invariant representations, the Template Voxel Aggregation (TVA) module compresses multi-view 2D templates into a compact deep voxel. Inspired by previous technique [10] developed for unsupervised video encoding, we propose to encode our instance templates via their relative orientation in the physical 3D world. To this end, we first generate the 2D feature maps  $\mathbf{F}^{\mathrm{S}} = \psi (\mathcal{I}^{\mathrm{S}})\in \mathbb{R}^{M\times C\times w\times w}$  using a shared backbone network  $\psi (\cdot)$  used in the query branch and then map the 2D features to 3D voxels for multi-view aggregation.
2D-3D mapping: To map these 2D features onto a shared 3D space for subsequent orientation-based aggregation, we utilize an implicit mapping function  $\mathcal{M}(\cdot)$ . This function translates the 2D features to 3D voxel features, denoted by  $\mathbf{V} = \mathcal{M}(\mathbf{F}^{\mathrm{S}}) \in \mathbb{R}^{M \times C_{\mathrm{v}} \times D \times L \times L}$ , where  $\mathbf{V}$  is the 3D voxel feature
from the 2D feature,  $C_{\mathrm{v}}$  is the feature dimension, and  $D,L$  indicate voxel spatial size. Specifically, we first reshape the feature maps to  $\mathbf{F}^{\prime \mathrm{S}}\in \mathbb{R}^{M\times (C / d)\times d\times w\times w}$ , where  $d$  is the pre-defined implicit depth, then we apply 3D inverse convolution to obtain the feature voxel.
Note that with multi-view images, we can calculate the relative camera rotation easily via Structure from Motion (SfM) [33] or visual odometry [39]. Given that the images are object-centered and the object stays static in the scene, these relative rotations in fact represent the relative rotations between the object orientations defined in the same camera coordination system. Different from previous work [10] that implicitly learns the camera extrinsic for unsupervised encoding, we aim to explicitly embed such geometric information. Specifically, our goal is to first transform every template into the same coordinate system using their relative rotation, which is then aggregated:
$$
\mathbf {v} ^ {\mathrm {S}} = \frac {1}{M} \sum_ {i = 1} ^ {M} \operatorname {C o n v 3 D} \left(\operatorname {R o t} \left(\mathbf {V} _ {i}, \mathbf {R} _ {i} ^ {\top}\right)\right), \tag {1}
$$
where  $\mathbf{V}_i\in \mathbb{R}^{C_{\mathrm{v}}\times D\times L\times L}$  is the previously mapped  $i$  -th independent voxel feature,  $\mathbf{R}_i^\top$  denotes the relative camera rotation between the  $i$  -th support frame and the first frame.  $\mathrm{Rot}(\cdot ,\cdot)$  is the 3D transform used in [10], which first wraps a unit voxel to the new coordination system using  $\mathbf{R}_i^\top$  and then samples from the feature voxel  $\mathbf{V}_i$  with the transformed unit voxel grid. Therefore, all the  $M$  voxels are transformed into the same coordinate system defined in the first camera frame. These are then aggregated through average pooling to produce the compact template voxel  $\mathbf{v}^{\mathrm{S}}$ .
By explicitly embedding the 3D rotations into individual reference features, TVA achieves a geometry-aware compact representation, which is more robust to occlusion and pose variation.
3.2.3 Query Voxel Matching
Given the proposal features  $\mathbf{F}^{\mathrm{Q}}$  from query image  $\mathcal{I}^{\mathrm{Q}}$  and the template voxel  $\mathbf{C}^{\mathrm{S}}$  from supports  $\mathcal{I}^{\mathrm{S}}$ , the task of the query voxel matching (QVM) module is to classify each proposal as foreground (the reference instance) or background. As shown in Fig. 2, in order to empower the 2D features with 3D geometry, we first use the same mapping to get query voxels,  $\mathbf{V}^{\mathrm{Q}} = \mathcal{M}(\mathbf{F}^{\mathrm{Q}}) \in \mathbb{R}^{N \times C_{\mathrm{v}} \times D \times L \times L}$ . VoxDet next accomplishes matching  $\mathbf{v}^{\mathrm{S}}$  and  $\mathbf{V}^{\mathrm{Q}}$  through two steps. First, we need to estimate the relative rotation between query and support, so that  $\mathbf{V}^{\mathrm{Q}}$  can be aligned in the same coordinate system as  $\mathbf{v}^{\mathrm{S}}$ . Second, we need to learn a function that measures the distance between the aligned two voxels. To achieve this, we define a voxel relation operator  $\mathcal{R}_{\mathrm{v}}(\cdot, \cdot)$ :
Voxel Relation Given two voxels  $\mathbf{v}_1, \mathbf{v}_2 \in \mathbb{R}^{c \times a \times a \times a}$ , where  $c$  is the channel and  $a$  is the spatial dimension, this function seeks to discover their relations in every semantic channel. To achieve this, we first interleave the voxels along channels as  $\operatorname{In}(\mathbf{v}_1, \mathbf{v}_2) = [\mathbf{v}_1^1, \mathbf{v}_2^1, \mathbf{v}_1^2, \mathbf{v}_2^2, \dots, \mathbf{v}_1^c, \mathbf{v}_2^c] \in \mathbb{R}^{2c \times a \times a \times a}$ , where  $\mathbf{v}_1^k, \mathbf{v}_2^k$  is the voxel feature in the  $k$ -th channel. Then, we apply grouped convolution as  $\mathcal{R}_{\mathrm{v}}(\mathbf{v}_1, \mathbf{v}_2) = \operatorname{Conv3D}(\operatorname{In}(\mathbf{v}_1, \mathbf{v}_2), \operatorname{group} = c)$ . In the experiments, we found that such a design makes relation learning easier since each convolution kernel is forced to learn the two feature voxels from the same channel. With this voxel relation, we can then roughly estimate the rotation matrix  $\hat{\mathbf{R}}^Q \in \mathbb{R}^{N \times 3 \times 3}$  of each query voxel relative to the template as:
$$
\hat {\mathbf {R}} ^ {Q} = \operatorname {M L P} \left(\mathcal {R} _ {\mathrm {v}} \left(\mathbf {V} ^ {\mathrm {S}}, \mathbf {V} ^ {\mathrm {Q}}\right)\right), \tag {2}
$$
where  $\mathbf{v}^{\mathrm{S}}$  is copied  $N$  times to get  $\mathbf{V}^{\mathrm{S}}$ . In practice, we first predict 6D continuous vector [40] as the network outputs and then convert the vector to a rotation matrix. Next, we can define the classification haed with the Voxel Relation as:
$$
\hat {s} = \operatorname {M L P} \left(\mathcal {R} _ {\mathrm {v}} \left(\mathbf {V} ^ {\mathrm {S}}, \operatorname {R o t} \left(\mathbf {V} ^ {\mathrm {Q}}, \hat {\mathbf {R}} ^ {Q}\right)\right)\right), \tag {3}
$$
where  $\mathrm{Rot}(\mathbf{V}^{\mathsf{Q}},\hat{\mathbf{R}}^{Q})$  rotates the queries to the support coordination system to allow for reasonable matching. In practice, we additionally introduced a global relation branch for the final score, so that the lost semantic information in implicit mapping can be retrieved. More details are available in the supplementary material. During inference, we rank the proposals  $\mathbf{P}$  according to their matching score and take the Top-k candidates as the predicted box  $\hat{\mathbf{b}}$ .
3.3 Training Objectives
As illustrated in Fig. 2, VoxDet contains two training stages: reconstruction and instance detection.
Reconstruction To learn the 3D geometry relationships, specifically 3D rotation between instance templates, we pre-train the implicit mapping function  $\mathcal{M}(\cdot)$  using a reconstruction objective. We
Figure 3: The instances and test scenes in the newly built RoboTools benchmark. The 20 unique instances are recorded as multi-view videos, where the relative camera poses between frames are provided. RoboTools consists of various challenging scenarios, where the desired instance could be under severe occlusion or in different orientation.
divide  $M$  multi-view templates  $\mathcal{I}^{\mathrm{S}}$  into input images  $\mathcal{I}_{\mathrm{i}}^{\mathrm{S}} \in \mathbb{R}^{(M - K) \times 3 \times W \times H}$  and outputs  $\mathcal{I}_{\mathrm{o}}^{\mathrm{S}} \in \mathbb{R}^{K \times 3 \times W \times H}$ . Next, we construct the voxel representation  $\mathbf{V}^{\mathrm{S}}$  using  $\mathcal{I}_{\mathrm{i}}^{\mathrm{S}}$  via the TVA module and adopt a decoder network Dec to reconstruct the output images through the relative rotations:
$$
\hat {\mathcal {I}} _ {\mathrm {o}, j} ^ {\mathrm {S}} = \operatorname {D e c} \left(\operatorname {R o t} \left(\mathbf {V} ^ {\mathrm {S}}, \mathbf {R} _ {j} ^ {\top}\right)\right), j \in \{1, 2, \dots , K \}, \tag {4}
$$
where  $\hat{\mathcal{I}}_{\mathrm{o},j}^{\mathrm{S}}$  denotes the  $j$ -th reconstructed (fake) output images and  $\mathbf{R}_j$  is the relative rotation matrix between the 1-st to  $j$ -th camera frame. We finally define the reconstruction loss as:
$$
\mathcal {L} _ {\mathrm {r}} = w _ {\text {r e c o n}} \mathcal {L} _ {\text {r e c o n}} + w _ {\text {g a n}} \mathcal {L} _ {\text {g a n}} + w _ {\text {p e r c e p}} \mathcal {L} _ {\text {p e r c e p}}, \tag {5}
$$