SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 3: Benchmarking results on our dataset. We summarize three salient conclusions. (1) End-to-end trained detectors perform better with stronger detector architectures, e.g., the transformer DINO (27.99 AP) outperforms FasterRCNN (19.54 AP). (2) Interestingly, the non-learned method SAM+DINOv2f performs the best (41.61 AP), significantly better than end-to-end learned detectors including DINO (27.99 AP). (3) All methods have much lower AP on hard testing images or smal1 objects (e.g., SAM+DINOv2f yields 28.03 AP on hard vs. 47.57 AP on easy), showing that future work should focus on hard situations or smal1 instances.
AP	\( \mathbf{A}\mathbf{P}_{50} \)	\( \mathbf{A}\mathbf{P}_{75} \)
avg	hard	easy	small	medium	large
FasterRCNN [37]	19.54	10.26	23.75	5.03	22.20	37.97	29.21	23.26
RetinaNet [29]	22.22	14.92	26.49	5.48	25.80	42.71	31.19	24.98
CenterNet [49]	21.12	11.85	25.70	5.90	24.15	40.38	32.72	23.60
FCOS [45]	22.40	13.22	28.68	6.17	26.46	38.13	32.80	25.47
DINO [48]	27.99	17.89	32.65	11.51	31.60	48.35	39.62	32.19
SAM + DINOf	36.97	22.38	43.88	11.93	40.85	62.67	44.13	40.42
SAM + DINOv2f	41.61	28.03	47.57	14.58	45.83	69.14	49.10	45.95
Table 4: Benchmarking results w.r.t average recall (AR) for small, medium and large instances. "AR@max10" means AR within the top-10 ranked detections. In computing AR, we rank detections by using the detection confidence scores of the learning-based methods (e.g., FasterRCNN) or similarity scores in the non-learned methods (e.g., SAM+DINO $_f$ ). AR $_s$ , AR $_m$ , and AR $_l$  are breakdowns of AR for small, medium and large testing object instances. Results show that (1) the non-learned methods that use SAM generally recall more instances than others, and (2) all methods suffer from small instances. In sum, results show that methods yielding higher recall achieve higher AP metrics (cf. Table 3).
AR@max10	AR@max100	ARs@max100	ARm@max100	ARl@max100
FasterRCNN [37]	26.24	39.24	14.83	44.87	60.05
RetinaNet [29]	26.33	49.38	22.04	56.76	69.69
CenterNet [49]	23.55	44.72	17.84	52.03	64.58
FCOS [45]	25.82	46.28	22.09	52.85	64.11
DINO [48]	29.84	54.22	32.00	59.43	72.92
SAM + DINOf	31.25	63.05	31.65	70.01	90.63
SAM + DINov2f	40.02	63.06	31.11	70.40	90.36
learning rate of 1e-3 and a batch size of 16. We fine-tune all the models for 5 epochs (which are enough for training to converge) and evaluate checkpoints after each epoch for model selection. The models are trained on a single Tesla V100 GPU with 32G memory.
If applied, we preprocess object instance profile images and proposals. Specifically, for a profile image, we remove the background pixels (e.g., pixels of QR code) using foreground segmentation (i.e., GrabCut). For each proposal, we crop its minimum bounding square box. We also study whether removing background pixels by using SAM's mask output performs better. We use  $\mathrm{DINO}_f$  and  $\mathrm{DINOv2}_f$  to compute feature representations.
5.1 Benchmarking Results
Quantitative results. To evaluate the proposed InsDet protocol and dataset, we first train detectors from a COCO-pretrained backbone following the cut-past-learn baseline. Table 3 lists detailed comparisons and Fig. 6 plots the precision-recall curves for the compared methods. We can see that detectors with stronger architectures perform better, e.g. DINO (27.99% AP) vs. FasterRCNN (19.54% AP). Second, non-learned methods outperform end-to-end trained models, e.g., SAM+DINOv2f (41.61% AP) vs. DINO (27.99% AP). Third, all the methods perform poorly on hard scenes and small instances, suggesting future work focusing on such cases.
Table 4 compares methods w.r.t the average recall (AR) metric. "AR@max10" means AR within the top-10 ranked detections. In computing AR, we rank detections by using the detection confidence scores of the learning-based methods (e.g., FasterRCNN) or similarity scores in the non-learned methods (e.g.,  $\mathrm{SAM + DINO}_f$ ).  $\mathrm{AR}_s$ ,  $\mathrm{AR}_m$ , and  $\mathrm{AR}_l$  are breakdowns of AR for small, medium, and large testing object instances. Results show that (1) the non-learned methods that use SAM generally recall more instances than others, and (2) all methods suffer from small instances. In sum, results show that methods yielding higher recall achieve higher AP metrics (cf. Table 3). Table 5 further studies AR in hard and easy scenes. We can observe that: (1) the non-learned methods that use SAM
Table 5: Benchmarking results w.r.t average recall (AR) for hard and easy scenes. We add a breakdown analysis of testing images on hard and easy scenes. Results show that (1) the non-learned methods that use SAM generally recall more instances than others, and (2) all methods suffer from hard scenes.
AR@max10	AR@max100
avg	hard	easy	avg	hard	easy
FasterRCNN [37]	26.24	12.92	32.33	39.24	16.91	49.43
RetinaNet [29]	26.33	15.38	31.33	49.38	29.00	58.69
CenterNet [49]	23.55	11.87	28.87	44.72	24.88	53.76
FCOS [45]	25.82	12.81	31.74	46.28	26.55	55.27
DINO [48]	29.84	16.63	35.84	54.22	36.46	62.30
SAM + DINOf	31.25	16.96	37.73	63.05	42.46	72.41
SAM + DINOv2f	40.02	27.64	45.36	63.06	43.47	71.96
Figure 6: Precision-recall curves with IoU=0.5 (AP50 in the legend) on our InsDet dataset. Stronger detectors perform better, e.g., DINO, a transformer-based detector significantly outperforms FasterRCNN. Furthermore, even with a simple non-learned method, leveraging pretrained models, e.g., SAM+DINOv2f, outperforms end-to-end learned methods.
recall more proposals than other competitors in both hard and easy scenes; (2) all methods basically suffer from hard scenes.
Qualitative results. Fig. 7 visualizes qualitative results on two testing examples from the InsDet dataset. Stronger detectors, e.g., the non-learned method SAM+DINOv2 $_f$ , produce fewer false negatives. Even so, all detectors still struggle to detect instances with presented barriers such as heavy occlusion, instance size being too small, etc. As shown in Fig. 6, the non-learned method SAM+DINOv2 $_f$  outperforms end-to-end learned methods in a wide range of recall thresholds.
5.2 Ablation Study
Due to the space limit, we ablate the instance crop and stable matching in the main paper and put more (including ablation studies for the cut-paste-learn methods) in the supplement.
Proposal feature extraction in the non-learned method. Given a box crop (encapsulating the proposal) generated by SAM in the non-learned method, we study how to process the crop to improve InsDet performance. Here, we can either crop and feed its minimum bounding box to compute DINOv2f features, or we can use the mask to remove the background in the box. Table 6 shows the comparison. Clearly, the latter performs remarkably better in both hard and easy scenarios.
Proposal-instance match in the non-learned method. After generating proposals by SAM, we need to compare them with instance profile images to get the final detection results. We study the InsDet performance of the two matching algorithms. Rank & Select is a greedy algorithm that iteratively finds the best match between any proposals and instances until no instances/proposals are left unmatched; stable matching produces an optimal list of matched proposals and instances such that there does not exist a pair in which both prefer other proposals/instances to their current correspondence under the matching. Table 7 compares these two methods, clearly showing that stable matching works better.
Impact of different image/proposal resolutions. We study the InsDet performance when using images of different sizes for SAM, and using different resolutions of crops fed into DINOv2  $f$ . For example, we can use SAM on images of size  $3072 \times 4096$  to generate proposals, we can resize crops of proposals to  $224 \times 224$  to feed into DINOv2  $f$  if they are larger than  $224 \times 224$  (otherwise, keep them unchanged). Table 8 lists detailed comparisons. We have two observations. (1) The InsDet performance generally increases with the image resolution but starts to drop when the input image is too large, i.e.,  $6144 \times 8192$ . This is because SAM tends to produce object parts as individual instances,
Figure 7: Visual results of FasterRCNN, DINO, and SAM+DINOv2 on our InsDet dataset. The top row illustrates the sparse placement of instances (i.e., easy scenario), while the bottom contains more cluttered instances (i.e., hard scenario). We drop predicted instance names for brevity. SAM helps localize instances with more precise bounding boxes, e.g., as arrows labeled in the upper row. DINOv2 provides more precise recognition of localized instances, e.g., five instances in the right of the bottom row. Compared with DINO, SAM+DINOv2 is better at locating occluded instances.
Table 6: Ablation study: whether to remove background in crops for feature computation. Based on a proposal given by SAM, we can crop and feed its minimum bounding square to compute DINOv2  $f$  feature, or we can use the mask to remove the background in the square before computing the feature. Clearly, the latter performs remarkably better.
strategy	AP	AP50	AP75
avg	hard	easy	avg	hard	easy	avg	hard	easy
w/o background removal	36.04	23.04	42.37	43.84	29.12	51.00	39.59	25.74	46.13
w/ background removal	39.12	24.00	47.17	46.72	30.81	54.66	42.86	26.40	51.58
Table 7: Ablation study: whether to generate unique proposal-instance match. In contrast to Rank&Select, Stable Matching produces a unique match to proposal/instance for each instance/proposal, yielding better performance than Rank&Select.
strategy	AP	AP50	AP75
avg	hard	easy	avg	hard	easy	avg	hard	easy
Rank & Select	38.62	23.95	46.31	46.04	30.77	53.64	42.37	26.39	50.61
Stable Matching	39.12	24.00	47.17	46.72	30.81	54.66	42.86	26.40	51.58
Table 8: Ablation study: which image/proposal size to use for SAM and DINOv2. We notice that InsDet performance generally increases with the input image resolution, but starts to drop when the image is too large. When using larger proposals for DINOv2, InsDet performance also gets better.
image resolution for SAM	input size for DINOv2f	AP	AP50	AP75
avg	hard	easy	avg	hard	easy	avg	hard	easy
768×1024	36.46	21.67	43.88	46.22	29.52	54.11	41.53	24.95	49.99
1536×2048	39.12	24.00	47.17	46.72	30.81	54.66	42.86	26.40	51.58
3072×4096	224×224	39.17	24.08	46.60	45.71	30.34	53.07	41.90	26.12	49.71
6144×8192	38.74	23.39	46.29	45.24	29.24	52.78	40.81	25.14	48.65
1536×2048	112×112	26.46	16.52	31.32	30.83	20.94	36.26	28.89	18.81	33.73
224×224	39.12	24.00	47.17	46.72	30.81	54.66	42.86	26.40	51.58
448×448	41.61	28.03	47.57	49.10	36.64	54.84	45.95	31.41	52.03
method	time (sec)	AP (%)
FasterRCNN [37]	0.00399	19.54
RetinaNet [29]	0.00412	22.22
CenterNet [49]	0.00376	21.12
FCOS [45]	0.00271	22.40
DINO [48]	1.90625	27.99
SAM + DINOf	15.10	36.97
SAM + DINOv2f	14.70	41.61
Table 9: We compare the inference runtime (second/image) of different methods, along with their InsDet performance in AP (\%). Clearly, there is a trade-off between runtime and detection precision. For example, among the methods studied in our work, SAM+DINOv2f achieves the highest AP (41.61) but is four orders of magnitude slower than FasterRCNN (19.54 AP). Developing faster and better InsDet methods is apparently future work.
resulting in more false positives. (2) When using larger proposals for DINo  $2_{f}$ , InsDet performance gets better, e.g.,  $41.61\%$  ( $448 \times 448$ ) vs.  $39.12\%$  ( $224 \times 224$ ).
5.3 Runtime Comparison
Table 9 compares the runtime of different methods, along with their InsDet performance. There is a trade-off between runtime and detection precision. For example, among the methods studied in our work, SAM+DINOv2f achieves the highest AP (41.61) but is four orders of magnitude slower than FasterRCNN (19.54 AP). Developing faster and better InsDet methods is future work.
5.4 Discussions
Societal impact. InsDet is a crucial component in various robotic applications such as elderly-assistive agents. Hence, releasing a unified benchmarking protocol contributes to broader communities. While our dataset enables InsDet research to move forward, similar to other works, directly applying algorithms brought by our dataset is risky in real-world applications.
Limitations. We note several limitations in our current work. First, while our work uses normal cameras to collect datasets, we expect to use better and cheaper hardware (e.g., depth camera and IMU) for data collection. Second, while the cut-paste-learn method we adopt does not consider geometric cues when synthesizing training images, we hope to incorporate such information to generate better and more realistic training images, e.g., pasting instances only on up-surfaces like tables, desks, and floors. Third, while SAM+DINOv2  $f$  performs the best, this method is time-consuming (see a run-time study in the supplement); real-world applications should consider real-time requirements.
Future work. In view of the above limitations, the future work includes: (1) Exploring high-resolution images for more precise detection on hard situations, e.g., one can combine proposals generated from multi-scale and multi-resolution images. (2) Developing faster algorithms, e.g., one can use multi-scale detectors to attend to regions of interest for progressive detection. (3) Bridging end-to-end fast models and powerful yet slow pretrained models, e.g., one can train lightweight adaptors atop pretrained models for better InsDet.
We explore the problem of Instance Detection (InsDet) by introducing a new dataset consisting of high-resolution images and formulating a realistic unified protocol. We revisit representative InsDet methods in the cut-paste-learn framework and design a non-learned method by leveraging publicly-available pretrained models. Extensive experiments show that the non-learned method significantly outperforms end-to-end InsDet models. Yet, the non-learned method is slow because running large pretrained models takes more time than end-to-end trained models. Moreover, all methods struggle in hard situations (e.g., in front of heavy occlusions and a high level of clutter in the scene). This shows that our dataset serves as a challenging venue for the community to study InsDet.
This work is supported by NSFC (No.62206256), and University of Macau (SRG2023-00044-FST). Shu Kong acknowledges Dr. Bin Liu for the initial support via compute resource.
[39] Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. "grabcut" interactive foreground extraction using iterated graph cuts. ACM transactions on graphics (TOG), 23(3):309-314, 2004.
[40] Neil Savage et al. Robots rise to meet the challenge of caring for old people. Nature, 601(7893): 8-10, 2022.
[41] Johannes L Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In CVPR, 2016.
[42] Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Cnn features off-the-shelf: an astounding baseline for recognition. In CVPR Workshops, 2014.
[43] Arjun Singh, James Sha, Karthik S. Narayan, Tudor Achim, and P. Abbeel. Bigbird: A large-scale 3d database of object instances. IEEE International Conference on Robotics and Automation (ICRA), 2014.
[44] Zhi Tian, Hao Chen, Xinlong Wang, Yuliang Liu, and Chunhua Shen. AdelaiDet: A toolbox for instance-level recognition tasks. https://git.io/adelaidet, 2019.
[45] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: Fully convolutional one-stage object detection. In ICCV, 2019.
[46] Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. Yolov7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. arXiv:2207.02696, 2022.
[47] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detector2. https://github.com/facebookresearch/detectron2, 2019.
[48] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Harry Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. In International Conference on Learning Representations, 2022.
[49] Xingyi Zhou, Dequan Wang, and Philipp Krahenbuhl. Objects as points. In arXiv:1904.07850, 2019.
[50] Xingyi Zhou, Vladlen Koltun, and Philipp Krahenbuhl. Probabilistic two-stage detection. In arXiv:2103.07461, 2021.
[51] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv:2010.04159, 2020.