SlowGuess/NeurIPS_2023_OCR huggingface dataset:
show applications of our method on entirely unseen data. In running these experiments, we aim to validate the usefulness of our hand-object detector to the broader community.
We evaluate primarily on the new Hands23, using splits defined in §3. We now define the metrics used for our output, covering box detection, segmentation, hand state and interaction classification.
Detection. Following past work [44, 43, 53], we use Average Precision with an IOU threshold of 0.5. This lower threshold is used because the precise hand/forearm boundary is often unclear in images. We report the COCO AP (averaged over IOU thresholds) in the supplement.
Segmentation. We evaluate as in instance segmentation; for consistency, we use a threshold of 0.5. Our ground-truth comes by prompting SAM [29] with our ground-truth boxes. We evaluated the SAM masks compared to crowd annotations. We obtained annotations for segments of 1000 hands and 1000 in-contact objects and evaluated on segments that workers came to consensus on. Median IoUs were high (hands: 0.84, objects: 0.89) and most objects had IoU  $\geq 0.75$  (hands:  $72\%$ , objects:  $77\%$ ). IoUs are close to those for COCO reported in [4]. We hypothesize that the slight drop in precision for hands is due to differences in where hands end.
Hand State. Past work [53] evaluated hand state questions (e.g., side) by calculating AP, but treating hands as correct only if both the box and state are correct. This metric mixes detection and state and is hard to interpret. Since hand detection performs well ( $\approx 85\%$  mAP), we instead evaluate accuracy on the true positives, or the chance that correctly detected hands have the correct hand state.
Hand Association. Following the approach for hand state, we evaluate hand detection at a fixed operating point. For each correctly detected hand, we analyze whether there is a correctly associated object box. We define true positives as hands for which there is an object box and the predicted object box has IOU  $\geq 0.1$ ; we set a low IoU to account for ambiguities in the object's extent and to avoid also implicitly evaluating object detection. We define false positives as false detections, and false negatives as hands with ground-truth object boxes and no predicted boxes.
Table 3: The proposed model's performance on detection and segmentation (AP) and states (accuracy)
Detection (AP)	Segmentation (AP)	State (Acc)
Hand	Obj.	2nd Obj.	Hand	Obj.	2nd Obj.	Side	Cont.	Fine	Grasp
84.3	51.6	44.0	67.0	45.0	33.2	95.0	84.5	67.0	56.0
Table 4: Hand/Object Association Evaluation. On most data, proposed association mechanism and of [53] are similar, but [53] performs poorly with many hands. On both subsets of the data, the proposed model produces more hand detections (34.9K vs 31.6K on all, 2.3K vs 1.1K on  $\geq 6$  hands).
(All)	Prec.	Recall	F1	(≥ 6 hands)	Prec.	Recall	F1
Proposed	84.6	96.5	89.9	70.2	92.3	79.1
Shan et al. [53]	78.0	93.9	85.3	45.9	71.4	55.9
We next analyze the performance of the system. We evaluate multiple outputs on a variety of datasets.
Performance on Hands23. We show qualitative results in Figure 4, including random results and quantitative performance in Table 3. More analysis appears in the supplement. Our method produces strong performance on this data: hands are detected accurately, and objects and second objects show strong performance. Performance does drop when hands and objects are small, and in cluttered images. Common grasp confusions include precision-vs-power for prismatic grasps, and precision prismatic and non-prehensile fingers. Contact is recognized well. For fine-grained contact, the most common confusions are about the object type or use vs hold: tool held is most confused with tool use and neither held, for instance. Interaction association performs well but struggles if there are heavy occlusions or shadows. Mask predictions do not perform well when it is a large scene object (e.g., the sea the boat is surfing on or the snow the ski pole is sticking into) and the edges are not sharp on thin or crowded objects. Common failure modes are visualized in the supplement.
Interaction Prediction Performance. One key methodological difference between the proposed approach and past work [53] is the mechanism for predicting interaction: [53] produces a single vector per hand that points to the center of the object box the hand is interacting with. This works well on images with relatively few hands, but does not work well on many images. We compare the approaches quantitatively in Table 4 and qualitatively in Figure 5: our approach has a slight degradation when there are many  $(\geq 6)$  hands while the performance of [53] plummets.
Blurring. We now analyze the impact of facial blurring on hand recognition by analyzing the performance of systems trained and tested with either blurred or unblurred faces. We aim to address the two questions also asked by [63]: does face blurring lead to missed detections, and does a model trained on blurred faces work when tested with unblurred faces, as will be common in practice?
We report the four options in Table 5 for detection, segmentation, and one representative hand state question, hand side. Full state is in the supplement. Working entirely with blurred data  $(\sqrt{}/\sqrt{})$  causes a slight dip compared to working entirely with unblurred data  $(X/X)$ , but performance is close. Training on blurred data and testing on unblurred data  $(\sqrt{}/X)$  is virtually identical – a tiny (0.1) drop in AP. The model has few false positives on faces, typically when they are large and in the upper half of the image. This may impact deployment and could be mitigated by either pre-blurring all input to the model or adding additional hard negatives of faces.
Generalization to Ego4D. Having trained an effective model, we now use it on the Ego4D [21] dataset. We show zero-shot generalization results in Figure 4. Quantitatively, our model yields a 86.9 Box AP / 60.0 Segmentation AP on the on the validation set.
We were unable to annotate Ego4D due to strict restrictions on data redistribution, but can use self-training to integrate Ego4D and our data. For each hand box, we add the output generated by our model plus SAM masks and retrain. We show qualitative results in the supplement, but quantitatively fine-tuning for 110K iterations raises Ego4D performance to 90.9 Box AP / 66.3 Segmentation AP. Continuing to train on Hands23 does not increase for the same number of iterations performance.
Table 5: Cross-facial blurring performance for hand+object detection, and hand state. For each category, we compare training with  $(\checkmark)$  and without  $(X)$  blurring and test in each column with  $(\checkmark)$  and without  $(X)$  blurring. Performance is similar. Full hand states are similar and in the supplement.
Hand Det.	Obj. Det.	2nd Obj. Det.	Side
✓	X	✓	X	✓	X	✓	X
Train	✓	84.3	84.2	51.6	51.5	44.0	42.3	95.0
Train X	82.4	84.3	50.1	52.6	43.3	43.6	94.0	95.1
Figure 6: Applications Enabled. Top row: searching for the bi-manual manipulation  $\text{Hand} \rightarrow \text{Object} \rightarrow 2nd \text{Object} \leftarrow \text{Hand}$ . We show 5 examples selected from a set of 7 automatically found ones; the full set is in the supplement. Bottom: top precision prismatic grasps from each subset plus Ego4D.
Applications: Finding Configurations of Objects and Grasps. The ability to recognize grasps, hands, objects, and second objects enables us to find particular types of interactions in our data. These may help robotics research find, for instance, demonstrations of bi-manual manipulations in video data or examples of people using a precision prismatic grasp. We show some examples in Figure 6 for a particular bimanual manipulation involving a tool as well as a grasp.
6 Discussion, Impacts, and Limitations
We have introduced a model that produces a rich output of hands engaged with interaction, as well as a dataset for supporting this model. We hope that our model will enable autonomous systems to better understand interaction, thereby enabling applications ranging from parsing robotics demonstrations [68] to helping stroke rehabilitation [58]. Nonetheless, surveillance systems may benefit from the increased ability to recognize hands and held object. We expect that the domain shift [57] between our data and many surveillance use-cases [9] will make use less likely, but note that there are other sorts of surveillance (e.g., via interactive home systems) that may stand to benefit more directly.
Our model is imperfect. While detection often works well, small hands and hands undergoing occlusion are often missing, and occlusions and shadows interfere with interaction association. Our grasp categorization adds richness to our understanding of the diversity of hands grasps, but may fail for grasps that fall between our categories. Additionally, predictions on masks are not good when facing large scene objects and it is hard to get accurate sharp boundaries for very thin objects.
Our data is reflective of its sources, which may not be representative of the data expected in many situations. Beyond the usual vision domain shifts (the real world does not look like the Internet, for instance), this non-representation can be in terms of demographics. For example, since we removed minors, our model may perform poorly on children. Likewise, while we did not find orders of magnitude difference, it is likely that there are some performance differences across demographics; we thus would extensive testing before use, and discourage use in high-stakes settings. Finally, while facial blurs had limited impact on performance, they did have impact, such as on-face predictions of hands that downstream systems may need to account for.
Acknowledgments. This material is based upon work supported by the National Science Foundation under Grant No. 2006619. We thank the University of Michigan DCO for their tireless continued support.
[64] Lixin Yang, Kailin Li, Xinyu Zhan, Fei Wu, Anran Xu, Liu Liu, and Cewu Lu. Oakink: A large-scale knowledge repository for understanding hand-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20953-20962, 2022.
[65] Lingzhi Zhang, Shenghao Zhou, Simon Stent, and Jianbo Shi. Fine-grained egocentric hand-object segmentation: Dataset, model, and applications. In European Conference on Computer Vision, pages 127-145. Springer, 2022.
[66] Yang Zheng, Yanchao Yang, Kaichun Mo, Jiaman Li, Tao Yu, Yebin Liu, Karen Liu, and Leonidas J Guibas. Gimo: Gaze-informed human motion prediction in context. arXiv preprint arXiv:2204.09443, 2022.
[67] Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. Freihand: A dataset for markerless capture of hand pose and shape from single rgb images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 813-822, 2019.
[68] Kateryna Zorina, David Kovar, Florent Lamiraux, Nicolas Mansard, Justin Carpentier, Josef Sivic, and Vladimir Petrik. Multi-contact task and motion planning guided by video demonstration. In ICRA 2023-International Conference on Robotics and Automation, 2023.