SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We compare QVM with more matching techniques in Table 5, where the averaged results onthe cluttered LM-O [17] and RoboTools benchmark are reported. We first ablate the Voxel Relation module in QVM, which results in  $\mathrm{QVM}^{\dagger}$ . Specifically, all the Voxel Relation in  $\mathrm{QVM}^{\dagger}$  are replaced by a simple depth-wise convolution, i.e., we first apply global average pooling on the template voxel to get a feature vector, which is then taken as the convolution kernel to calculate the correlation voxel from the queries. We can see such a naive design will result in a performance drop.
Table 5: Comparison with different types of matching module. We compare QVM with the correlation in [5], class-level relation proposed in [1], and the class distance defined in FSDet [45].
Method	mAR	\( \mathrm{AR}_{50} \)	\( \mathrm{AR}_{75} \)
QVM (Ours)	23.95	33.35	26.90
\( QVM^† \)	22.45	31.75	25.05
2D Relation [1]	20.25	29.70	22.80
FSDet [45]	20.35	29.35	22.60
Local Matching [48, 49]	10.60	13.90	11.75
For all the rest methods, we used the same open-world detector to obtain the universal proposals, which are then matched with the template images using different matching techniques. To be more specific, 2D Corr. [5] constructs support vectors from every reference image. Then, depth-wise convolution is conducted between each support vector and the proposal patch. The resulting
VoxDet vsDTOID
VoxDet vs Gen6D
VoxDet vs OLNDINO
LM-O
YCB-V
RoboTools
Figure 9: Detection qualitative results comparison between VoxDet and 2D baselines, DTOID [6], Gen6D [5],  $\mathrm{OLN_{DINO}}$  [14, 9] on the three benchmarks. VoxDet shows better robustness under pose variance and occlusion. These qualitative comparisons can be better visualized in our supplementary video.
correlation maps are sent to an MLP for classification score. In 2D Relation [1], we substitute the simple depth-wise convolution in 2D Corr. with the spatial and channel relation proposed in [1]. In FSDet [45], the depth-wise convolution in 2D Corr is replaced by the distance defined in [45]. Since they are geometry-unaware, we find all the 2D techniques worse than our QVM module.
Additionally, we designed a Local Matching baseline [49, 48]. In Local Matching, we first extract local key points from the reference images and proposals using SuperPoint [49]. Then the points descriptors are matched by SuperGlue [48]. We take the mean matching score of all the points in the proposal as their classification score. We find such an implementation, though geometry-invariant, falls short in our task since it lacks semantic representation of the whole instance.
C More Detection Visualizations
We present more detection qualitative comparisons in Fig. 9. VoxDet, in red, is compared with three baselines, DTOID [6], Gen6D [5], and  $\mathrm{OLN}_{\mathrm{DINO}}$ . Compared with previous instance detectors [6, 5], VoxDet is more robust under orientation variation and severe occlusion by virtue of the learned geometric knowledge. For example, in the LM-O benchmark, second column, when the duck is partially occluded and the egg box is in different orientations, VoxDet can still find them while Gen6D fails. Compared with similarity matching [9], VoxDet can better distinguish similar instances via the QVM module. For instance, in the RoboTools benchmark, the third column, the desired instance could be distracted by the motor, which has similar appearances but different geometry. Our VoxDet can discover such geometric differences and make correct classification, while the similarity matching falls short even if the feature from DINO [9] is stronger than ResNet50 [46].
D Sim-to-Real Comparison
VoxDet is entirely trained on synthetic dataset, OWID. We observe that the model shows some domain gap when transferred to real-world images like RoboTools. On the synthetic-real datasets, LM-O [17] and YCB-V [18], our model easily outperforms those trained on real images, while it shows limitations in fully real test set RoboTools. For example, Gen6D [5] is mainly trained on real-images, which reports  $17.0\mathrm{mAR}$ ,  $35.5\mathrm{AR}_{50}$ , and  $14.3\mathrm{AR}_{75}$ . Its  $\mathrm{AR}_{50}$  is higher than VoxDet (23.6) while in harder metrics like  $\mathrm{AR}_{75}$ , our model works better (20.5). Compared with the cutting edge foundation models that are trained on large-scale real images, our model still shows spaces for improvement. For example,  $\mathrm{OLN}_{CLIP}$  achieves  $11.0\mathrm{mAR}$ ,  $20.8\mathrm{AR}_{50}$ , and  $9.2\mathrm{AR}_{75}$ , which is worse than VoxDet. Yet,  $\mathrm{OLN}_{DINO}$  [13] can outperform VoxDet in RoboTools with over  $30\mathrm{mAR}$ . We conclude that the feature representation from the concurrent 2D foundation model [13] could be a stronger backbone for VoxDet to overcome the domain gap issue. Learning a geometry-aware strong voxel representation from such foundation model will be one of our future work.
E Performance under Flawed Voxel
VoxDet assumes known instance masks and poses for the reference video, which may have some noise during realworld deployment. To quantitatively analysis the robustness of VoxDet under flawed Voxels, we present its results on RoboTools when the reference video is disturbed in appearance and geometry.
Table 6: Performance of VoxDet on RoboTools when the reference is disturbed. The ratio means center shift and scale noise with respect to width and height.
Mean Shift Ratio	0	10%	20%	30%
AR50	23.6	20.1	18.9	17.1
Add noise on the reference image patches :
We tried to add random shift on the cropped area in the reference images, resulting in inaccurate instance appearance. The results on RoboTools are shown in Table 6. We conclude that even when we disturb around  $65\%$  of the voxel (30% shift on each 2D patch), the model still works, which means VoxDet is robust to appearance noise.
Add noise on the relative poses : We tried to add random error on the pose of the reference images, resulting in inaccurate instance geometry. When we add as large as 15 degree angular error, the performance  $(\mathrm{AR}_{50})$  decreased from 23.6 to 20.4. We conclude that VoxDet is not very sensitive to the geometry noise.
F Extended Related Works
Visual Object Tracking aims to localize a general target instance in a video, given its initial state in the first frame. Early methods adopt discriminative correlation filters [50-52], where the calculation in the frequency domain is so efficient that real-time speed can be achieved on a single CPU. More recently, methods are developed on Siamese Network [53] and Transformers [54-56]. Unlike detection, object tracking has a strong temporal consistency assumption, i.e., the location and appearance of the instance in the next frame do not largely vary from the previous frame. So that they only conduct detection/matching in the small search region with a single 2D template, which can't work for our whole image detection setting.
Instance Pose Estimation is developed to estimate the 6 DoF pose of an unseen instance. Some of them [57, 58] match the local point features and resort to RANSAC to optimize the relative pose. While others [5, 59] first selects the closest template frame and then conducts pose refinement on the
known template poses. Most of these methods usually assume the instance detection is perfect, i.e., they crop the instance out of the query image with the ground truth box and estimate the pose on the small object-centered patch. Our VoxDet can serve as their front-end, which is robust to cluttered environments, thus making the detection-pose estimation framework more reliable.
Instance Retrieval hopes to retrieve a specific instance from a large database with a single reference image [60-65]. Some early work extracts local point features from template and query patch for image matching [61, 49], which may suffer from poor discriminative capability. More recent work resorts to the deep neural network for a global representation of the instance [62-65], which is compared with the features from query images. However, most of them construct 2D template features from the reference, so that their representation is unaware of the 3D geometry of the instance, which may not be robust under severe pose variation. Besides, instance retrieval methods usually require high-resolution query images for the discriminative features, while the instance in our cluttered query image could be in low-resolution, which sets additional barriers to these approaches.