Towards A Richer 2D Understanding of Hands at Scale
Authors: Tianyi Cheng
Dandan Shan
Ayda Sultan Hassen
Richard Ely Locke Higgins
David Fouhey
Venue: NeurIPS 2023 poster (2023)
Keywords: human-object interaction; hand object detection; hand detection
TL;DR: We introduce a new model that provides more information about hand interaction from images powered by a new dataset.

Abstract: As humans, we learn a lot about how to interact with the world by observing others interacting with their hands. To help AI systems obtain a better understanding of hand interactions, we introduce a new model that produces a rich understanding of hand interaction. Our system produces a richer output than past systems at a larger scale. Our outputs include boxes and segments for hands, in-contact objects, and second objects touched by tools as well as contact and grasp type. Supporting this method are annotations of 257K images, 401K hands, 288K objects, and 19K second objects spanning four datasets. We show that our method provides rich information and performs and generalizes well.

Towards A Richer 2D Understanding of Hands at Scale
Tianyi Cheng"! Dandan Shan" Ayda Sultan Richard E. L. Higgins' David F. Fouhey!.
University of Michigan fevacheng, dandans, ayhassen, relh, fouhey)
Addis Ababa University "New York University
Abstract
As humans, learn lot about how interact with the world by observing others interacting with their hands. To help Al systems obtain a better understanding of hand interactions, of hand interaction. Our system produces richer output than past systems at a larger scale. Our outputs include boxes and segments for hands, in-contact objects, and second objects touched by tools as well as contact and grasp type. Supporting this method are annotations of 257K images, 401K hands, 288K objects, and 19K second objects spanning four datasets. We show that our method provides rich information and performs and generalizes well.
introduce new model that produces rich understanding
1 Introduction
From the moment we wake up to the moment we go to sleep, most of us actively use our hands to shape and interact with the world. As infants, own ability to interact with the world is accelerated and guided by our observations of others using their hands (55) 37]. These interactions involve not just the objects we directly hold, but also tools that enable to interact with second objects, such as a spatula that enables us to push an egg in a pan. As adults, as demonstrated by the endless supply of instructional videos on the Internet [39], we can parse demonstrations of activities to help guide own efforts; we can follow hands, the objects they hold, the grasps they use, and how they use tools to interact with the world. Accordingly, there have been many efforts 2D hand understanding, each focusing on a different aspect. These include large-scale bounding box datasets [53], segmentation datasets (65 [4], and in-lab grasp and contact efforts (50) 5) Each contributes to the literature, but obtaining unified understanding of hands in action, from box to segment to grasp to tool use, requires stitching together many models and datasets.
We contribute a model that predicts a rich, unified 2D output hands interacting with objects that effective variety of first and third person data. Our model predicts boxes for hands and in-contact/active objects like past efforts [7], as well as second objects, or objects contacted by an in-use tool, which in turn enables understanding how humans use tools to manipulate their world. Like past work [53] 44,, our model predicts when hands contact objects; we extend this to a richer vocabulary that distinguishes touching, holding and using, and includes grasps. Finally, by converting boxes to segments [29), our system produces segments for hands, objects, and second objects.
These outputs are built on two key components model and a dataset. The model is straightforward, easily-extensible deep network for parsing interaction from an image. We build on standard detection machinery 24 with standard losses and use simple inference method. Powering this model is large dataset, Hands23, consisting of 257K images, 401K hands, 288K objects, and 19K second objects. Hands23 provides unified annotations for four datasets: EPIC-KITCHEN the 2017 train set of COCO 35). Internet Articulation [46), as well as on our introduced dataset of interaction-rich videos, New Days. All Internet data has Creative Commons licenses and has had faces obfuscated after automatic and manual detection as well as minors removed from the dataset.
[3] VISOR (14)
Equal contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Tool
Right, Object Contact, Power Prissattc
Auticulation
Now Vidoos
COCO 2017
EPIC-KITCH. VISOR
Used
(Left) The rich output of our model, consisting of boxes and segments for hands (left hand blue, right hand red), objects in contact, as well as second objects that are contacted by a tool. Output includes contact, making distinctions between touching and holding as well as grasp type. (Right) The method is powered by new annotations on new and existing data, covering many scenarios. Results shown are selected outputs on our test set.
Figure 1:
We analyze our approach and dataset through series of experiments. Across datasets, our results show that hand detection models trained on our dataset have strong zero-shot performance compared demonstrating the expansiveness of Hands23. Within our new dataset, our experiments demonstrate that our model can detect our detailed hand- object state both well and better than past efforts such as [53]. Past work in this space [53] found adoption in a variety of communities interested in learning from hand-object interaction, ranging from parsing robotics demonstrations [3I 60 68 to guiding video understanding 120 [5] and more (58) 40 62]. To this end, we conclude by showing applications of our model to finding grasps in action as well external datasets.
against past hand detection datasets [53 16 44
2 Related Work
Our work aims to provide a model and dataset for obtaining rich information about hands in a wide variety of images. Our key contribution lies in enabling richer output, which includes second objects that are interacted with by tools, more fine- anee-graind contact, grasps, and segments. As such our work spans a few directions in computer vision, machine learning, and robotics.
2D Hand Understanding. There have been extensive efforts to understand hands interacting with objects. One direction has been generating increasingly rich datasets for egocentric 1 12 14 65
48 21) and third person data [1 44 43 [33] 22 42), with each dataset covering a particular use case and perspective. Our new dataset, H3 provides a unified dataset which builds on and annotates four data sources: EPIC-KITCHENS VISOR 14, COCO (35], Internet Articulation (46), and a new video dataset. Hands23 contributes to this literature on three fronts: it introduces a richer contact 14 that includes fine-grained contact in terms of touching-vs-holl and tools, grasps building on the vocabulary of [II]. second objects, and segments automatically obtained from SAM [29]: second, it spans both first and third person data; finally, to the best of our knowledge, it is the largest with 250K images and 400K hands. There are specialized datasets still richer than ours: (50) has a fine-grained grasp vocabulary and (65) has human-annotated segments; however, these datasets are more limited in scope and are complementary to our efforts.
vocabulary than 43 53 21
To take advantage of this new data, we introduce new detection- model that can predict our new rich output state. We build on standard detection machinery [24] to accomplish our goal. With the advent of 2D understanding datasets, there has been a flurry of works proposing new mechanisms to recognize active objects [17 61 and improve detection and segmentation performance 65] We see this work as complimentary and likely to assist our model as well.
Applications of 2D Hand Understanding, The ability to reliably obtain basic 2D information about hands has unlocked many use cases. Naturally, some have focused on hands themselves, such as making more diverse and challenging data accessible to 3D hand reconstruction systems 7 23 51]. However, since hands are the key to many interactions. many other domains stand to benefit. These include ML areas like vision, where hands can guide video understanding 22
[5], as well as robotics,
where hand-object detection helps parse natural human demonstrations [31 60 68 and video data
is used for pretraining 47 54 Excitingly, reliable detection enables applications like post-stroke
rehabilitation (58). understanding the human visual experience for neuroscience [40). and finding
lost objects (62). Our work enhances these efforts by providing richer and more accurate outputs.

---

Table 1: Our Hands23 and its subsets compared with the largest similar dataset, 100DOH [33
Hands23 is 2.5x larger annotated images and has 2nd objects. Each subset adds a different photo- graphic bias [7 and challenge. New Days and VISOR [1] have many objects and 2nd objects. COCO 33 has images with many hands, but fewer second objects. Due to a focus on articulated objects in Internet Articulation [46, there are no 2nd objects.
Source
#Img
>2 Hand
#Hands
w/Obj
w/2nd
#Obj
#2nd
100DOH [53
Video
100K
9.5%
190K
74.1% 1%
0.0%
140K
0K
Hands23
Both
257K
8.6%
401K
71.7%
4.9%
288K
19K
New Days
Video
96K
4.4%
121K
77.0%
8.2%
93K
10K
VISOR 14
Video
38K
0.0%
58K
83.8%
10.7%
49K
6K
COCO 35
Image
45K
33.4%
123K
64.4%
1.9%
79K
2K
Artic. 46
Video
76K
3.6%
97K
67.0%
0.9%
65K
OK
Understanding Whole-Body and 3D Interactions. Our focus is specifically hands and objects in 2D, but there is wide literature understanding interactions with different goals. The most related threads of research are whole body interaction and 3D hand understanding. Whole-bode body
interactions [33 T0 26
56
66
38 27
45 28 usually evaluated
different styles of data and
pose orthogonal challenges. Similarly, full 3D understanding of hands in action (64 67 5 52 42 22
36 [2] usually provide richer 3I output space the cost of working in constrained settings.
Our work may help bring these 3D techniques to more unconstrained settings.
Dataset
We contribute a new dataset images of hands engaged in interaction with the world. All data Creative Commons or was user-collected [12] and has had faces blurred and minors removed for privacy. The data covers four datasets: new video dataset that we refer to as New Days, COCO (35). EPIC-KITCHENS [13 VISOR [14), and Internet Articulation vidcos [46]. These disparate datasets are unified by a new set of annotations including: (1) boxes for hands, held objects, and second objects; (2) contact information that distinguishes touching, holding, and using: (3) grasp contact labels for subset of the data; and (4) automatically extracted segments for boxes. We include a datasheet [18] for our proposed dataset in the supplement.
3.1 Underlying Image Data
Our underlying data is obtained from four different datasets. These datasets cover still images, third person video, and egocentric video.
New Video Dataset. We gather fresh set of video data of people engaged in interaction with the world, following the general strategy outlined in [16, 53]. The supplemental contains the full details, but in short, we follow three steps: we generate a vast pool of potential videos via combinatorial set of text queries; we automatically select videos as likely to contain interactions (and unlikely to contain children, lecturing, have cartoons) using a classifier trained on video thumbnails; finally, we sample frames from the selected videos to contain a mix of up-close and far-away scenes.
Past Data. We three public datasets: the 2017 training of COCO 335 EPIC-KITCHENS VISOR 14 13, and Internet Articulation [46] We use the full dataset from Internet Articulation
and VISOR, and use COCO images containing non-crowd people with area 2 10 pixels.
Data Splits. We provide 80 10/10/1100 10% train/val/test splits that split video data by channel and are backwards compatible with existing datasets. These are documented in the supplement.
Copyright & Privacy. All underlying image data used are public; when posted by users, they were posted publicly with a Creative Commons license selected. Obtaining explicit permission from each user for inclusion the data is impractical due to the scale of the data. However, for privacy, we have: (1) blurred faces of people via automatic methods combined with multiple rounds of manual correction; (2) found and removed images containing minors; and (3) will provide means for people to remove their data from the dataset. These procedures are detailed in the supplemental.

---

Hands23
New
Honds23 100p0et
101
Donsity
New Vidwos
0000
02 03
Normalizod Hand Size
Figure 2: Dataset statistics. (Left) Density of normalized hand size (box diagonal over image diagonal) for four datasets. COCO and VISOR have a strong preferential sizes; 100DOH and New Days have wider distribution. Hands23 covers all the sizes. (Right) Spatial density of hand bounding boxes, showing left and right hand blue and red respectively. VISOR has a typical egocentric side and location bias and COCO has few hands in the upper image.
3.2 Annotations
We provide unified annotations for four datasets. Annotation needed, even for a dataset like VISOR [4 that already has annotations, VISOR "hands" have varied definition that includes the forearm.
Hand and Side. All annotations are built on top of hand. We obtain annotations for left/right hands as bounding boxes, defining hands as terminating at the wrist following [33 43 To avoid tiny hands in COCO 35,, we only annotate non-crowd instances with at least 10 pixels of area.
Auxiliary Labels. For each identified hand, obtain annotations for rich set of labels, including boxes for contacted objects and secondary objects.
Contuct. We label contact following the contact vocabulary of 43 [no, object, self, other-person).
Fine-Grained Contact: Bodding-vs-Tocching and Tools and Containers. Past datasets [53 43
have not distinguished touching and holding objects. For more nuance, we obtain annotations for {use tool) and the Cartesian product of [touch, hold) and {tool, container, neither). We define tools as objects that are used like hand to physically interact (e.g., spoons, tennis rackets) and containers objects that can hold something (e.g.. bins, trays, bottles). We distinguish use from hold by requiring physical contact with a second object, and distinguish hold from touch by whether the hand expected to drop the object. We provide these annotations only to provide potentially useful information if downstream task aligns with these definitions. Tasks that do not distinguish tools can, for instance, marginalize over the probabilities obtain touch/hold/use.
In Contact Object, Secondary Objects. For hands in contact with objects, we also obtain annotations of the box of the contacted object following (53). For self and other-person contact, this box the full person. Furthermore, for in-contact objects that are tools, we obtain annotations of the box of the second object (e.g., lettuce if a hand is holding a knife that cutting lettuce).
Hand Grasp. For a 65K subset of hands in contact, we obtain seven grasp types. The first distinction annotated is Prehensile grasps (holding) and Non-Prehensile grasps (non-holding). Prehensile grasps are then divided into five categories from the Cutkosky [11] taxonomy, producing the Cartesian product of (Power, Precision) and (Prismatic, Circular) as well Lateral grasps. Non-Prehensile grasps are also divided into Finger Only and Palm grasps.
Segmentation. We automatically generate segments for the hands, first objects, and second objects by converting boxes to segments. We use the Segment Anything [29] Model (SAM) and prompt with box. For hands, we accept the SAM output unmodified, while for first objects and second object, we take the output with the hand mask subtracted.
3.3 Dataset Analysis
We next analyze the data. Examples of the proposed data appear throughout the paper, but we report some summary statisties, well comparison with other datasets.
Data Analysis. We analyze the subsets of the data that have been annotated Table compare with the most comparable dataset, 100DOH (53). An extended table with more datasets appears in

---

Table 2: Zero-shot generalization to/from Hands23. Training on Hands23 produces good performance and sometimes better results compared to within-dataset performance; no other dataset produces good results on Hands23.
Hands23
100DOH [33
VLOG [16]
Ego[2]
VGG 41
TV
44
Train/Test Same
85.6
93.6
80.9
97.9
84.4
71.0
0-Shot from Hands23
85.6
93.6
85.5
96.4
64.4
60.9
1 Generalization from Hands23 to other f
+ Generalization from other to Hands23 +
0-Shot to Hands23
85.6
78.5
64.9
44.5
56.0
51.7
the Supplement. Our new video dataset New Days has similar statistics to 100DOH and the full Hands23 is 2.5x the size of 100DOH.
Due to photographic bias (57 each subset of Hands23 contributes a unique challenge: VISOR shows many tools in use and thus second objects, and COCO has many images with many people but few second objects (since still images rarely capture actions-in -proogis--rrrrsss..... We spatially analyze the hands in Figure 2. Hands23 shows good coverage across sizes and locations. On the other hand, COCO and VISOR have a strong preferential size due to photographic biases (unlike internet datasets like 100DOH and New Days). VISOR similarly shows strong left/right bias for locations of left/right hands, and COCO has few hands in the top of the image. Together, however, the datasets provide good coverage of wide variety of scenarios.
Cross-Dataset Performance. To analyze Hands23's difficulty, we do cross-datased analysis for hand detection. We train a ResNet-101 25 Faster-RCNN 4 to follow [53] We report zero-shot
generalization performance compared to within train/test performance Table [ Training on
Hands23 is often close to within-dataset performance. On the other hand, training on other datasets and testing on Hands23 produces poor performance. 100DOH (53) is the best performing dataset and produces 7% AP drop (in addition to having a less rich set of labels).
Demographies and Realism. Hands23 has the demographies of its underlying data and is thus not representative of the world's people. We did an initial check for the orders of magnitude differences seen in other models 6 We compared results across binarized perceived gender presentation and Fitzpatrick scale (1-3 vs 4-6) following [6]; we note the same caveats about binarization.
We obtained 100 samples of each category and compared hand/object false positive/negative rates using the model described in subsequent sections. VISOR was both substantially easier for detection and heavily skewed light-skinned in our analysis. Controlling for all such difficulty factors relating to demographics (e.g., video styles) is challenging, but to control for this known factor, we excluded VISOR. A full evaluation is in the supplement, but most error rates were close and within I standard error of the mean: e.g., hand FPRs across skin tone were 3.2 + 1.8 (darker) vs 3.0 + 2.1 (lighter) and across perceived gender were 4.3 + 2.1 (female) vs 2.2 1.6 (male). We did observe slight increase in object FPR across perceived gender (24.5 + 4.9 female vs 16.9 + 4.3 male) and skin tone (16.0 + 3.8 darker vs 12.1 4.0 lighter), although neither was significant according Fisher's Exact Test [59]. We nonetheless would recommend that users be cautious and not deploy the system in high stakes settings.
Regardless demographics, the content of the data is not necessarily representative of everyday life. For instance, many people who are eating are doing performative or competitive eating. Thus, while the data depicts actual interactions of people with objects (e.g.. how one picks up an egg), we urge caution in learning priors (e.g., will someone eat 50 hard boiled eggs).
Method
For simplicity, we build our approach on top of standard RCNN object detection machinery 49 19
Our goal is to provide a simple and easily extended model for the community to build upon. In addition to predicting the object class, bounding box regression, and segment from a ROI pooling layer, we also predict set of auxiliary labels. Our experiments are done with a standard MaskRCNN [19] model, but our approach can be generalized to other frameworks like DETR [8). Conceptually, the
5

---

Input
Detection Per ROI
Interaction Per ROI Pair
Output
Delection
Side
Contact
Touch
Grasp
Side:
Contact:
Touch:
Grasp, Prehensile:
Lon-Prehensilee
dbject, athee-person)
circule)
Hinger palm)
Figure 3: Our approach predicts: (1) rich information per-region of interest (ROI) that is done with MLPs on top of standard instance segmentation machinery; and (2) an interaction between pairs of ROIs (here, a hand plus a held object). The final results merge these outputs with a simple algorithm.
underlying detection architecture needs to provide feature fü for the ith hypothesized box. Given this feature, predict the labels annotated in our dataset follows (illustrated in Figure 3
Standard Detection. The model predicts the standard targets, consisting of the object class and bounding box regression parameters. In setting, in addition to the implicit background class, there are three foreground classes: hand (regardless of side), object, and second object.
Segmentation. We predict segment using a PointRend 30 head that is set up identically to 30
Auxiliary Hand Information: Hand Side, Contact, Fine-Grained Contact, Grasp. The model predicts auxiliary hand task as independent multiclass outputs using independent two-layer MLPs.
Hand/Object, Object/Second-Ojjer Interaction. We train the network to classify whether pairs of objects are interacting. This is distinct from past work on hand-object association [53 19 that
regresses offset vector for the hand that is compared with detected object centers.
Given features f/ and f, for boxes i and j. the network classifies the interaction between boxes and The network inputs the concatenation f. fj. and relational feature Φij and as output, predicts multiclass output of no contact, object contact, self contact, other-person, or bhjecc-second-objee Hand/Object pairs are only trained on the first four classes; Object/Second- trained on the first and last category; impossible combinations such as Hand/Hand are not trained. Because the interaction MLP is evaluated on the many pairs of boxes, it is small, two layer network.
pairs are only
The relation feature di aims help the network quickly reject unlikely interactions and includes: the vector v between the box centers and its normalized version v/l v/2. help recognize boxes that overlap, Φij contains the minimum and maximum values of the signed distance function for box i, evaluated at box j's corners, and scaled proportional to box i's diagonal: these are negative if box j's corners are inside box and represent the distance box measured in hand-units.
Inference. For inference, we apply simple greedy method. We start with all hand, object, and second object boxes with detection scores 2 ts. 2 to and 2 ts respectively. Then, for every hand, we find the object with the highest predicted interaction score; if the hand-object interaction score is 2 A, or the tool- -ol-interaction score is 2 t, then we accept the interaction. We associate second objects to objects using an identical algorithm and threshold. Objects without a hand, and second objects without an object are removed. We set tH 0.7, To 0.5, Ts 0.3, Ta 0.1 and Ty = 0.7 via grid search on the validation set for thresholds that achieve best evaluation results. The performance is relatively insensitive to the hand threshold, but low thresholds for hands led to lead to poor performance on interaction prediction.
Training and Loss Balancingg We train using standard Mask-RRCNN recipe. All of our additional losses are standard multinomial classification problems, and we use a standard cross-entropy loss. We sum all the losses together and weight all but one equally: we scale the association loss by 0.1. which we found to stabilize training. For the backbone, we use a standard FPN [3 ResNet101 25 backbone. Full details are in the supplement.
5 Experiments
We evaluate hand, object, and second-object detection. We also evaluate our auxiliary heads, including hand side, contact, and grasp. We demonstrate results new Hands23 dataset, and

---

Fgo4D
Figure 4: Results Hands23 and zero-shot generalization Ego4D 12D (columns: 1-4 selected; column 5,6: random). Our model performs well on different scenarios including indoor and outdoor scenes and does well random images, including data from different dataset.
more
more
moor
Qurs
100DOH
Figure 5: Comparison with the 100DOH model (53 trained on Hands23. Results are similar with few hands, but with more hands, 53 finds incorrect distant interactions. Our method more accurate.
show applications of our method on entirely unseen data. In running these experiments, aim to validate the usefulness of our hand-object detector to the broader community.
5.1 Experimental Setup
We evaluate primarily on the new Hands23, using splits defined in S We now define the metrics used for our output, covering box detection, segmentation, hand state and interaction classification.
we use Average Precision with an IOU threshold of 0.5. This lower threshold is used because the precise hand/forearm boundary is often unclear in images. We report the COCO AP (averaged over IOU thresholds) in the supplement.
Detection. Following past work 44 43 53
Segmentation. We evaluate as in instance segmentation; for consistency, Our ground-truth comes by prompting SAM [29] with our ground-truth boxes. We evaluated the SAM masks compared to crowd annotations. We obtained annotations for segments of 1000 hands and 1000 in-contact objects and evaluated on segments that workers came to consensus on. Median IoUs were high (hands: 0.84, objects: 0.89) and most objects had IoU 2 0.75 (hands: 72%, objects: 77%). IoUs are close to those for COCO reported in [4]. We hypothesize that the slight drop in precision for hands is due differences in where hands end.
a threshold of 0.5.
Hand State. Past work [53] evaluated hand state questions (c.g., side) by calculating AP, but treating hands as correct only if both the box and state are correct. This metric mixes detection and state and hard to interpret. Since hand detection performs well (s:85% mAP), we instead evaluate accuracy on the true positives, the chance that correctly detected hands have the correct hand state.
Hand Association. Following the approach for hand state, we evaluate hand detection at a fixed operating point. For each correctly detected hand, we analyze whether there is correctly associated object box. We define true positives as hands for which there there is an object box and the predicted object box has IOU > 0.1; set low IoU to account for ambiguities in the object's extent and to avoid also implicitly evaluating object detection. We define false positives as false detections, and false negatives hands with ground-truth object boxes and predicted boxcs.
7

---

Table 3: The proposed model's performance on detection and segmentation (AP) and states (accuracy)
Detection (AP)
Segmentation (AP)
State (Acc)
Hand Obj. 2nd Obj.
Hand Obj. 2nd Obj.
Side Cont. Fine Grasp
84.3 51.6
44.0
67.0
45.0
33.2
95.0 84.5 67.0 56.0
Table 4: Hand/Object Association Evaluation. On most data, proposed association mechanism and of [53] are similar, but [53] performs poorly with many hands. On both subsets of the data, the proposed model produces more hand detections (34.9K vs 31.6K on all, 2.3K vs 1.1K on 2 6 hands).
(All) Prec.
Recall F1
(2 hands) Prec.
Recall F1
Proposed
84.6
96.5
89.9
70.2
92.3
79.1
Shan al. 53
78.0
93.9
85.3
45.9
71.4
55.9
5.2 Results
We next analyze the performance of the system. We evaluate multiple outputs on a variety of datasets.
Performance on Hands23. We show qualitative results in Figure 4 including random results and quantitative performance in Table More analysis appears in the supplement. Our method produces strong performance on this data: hands are detected accurately, and objects and second objects show strong performance. Performance does drop when hands and objects are small, and in cluttered images. Common grasp confusions include precision-vs-pon prismatic and non- preheeeeensie fingers. Contact is recognized well. For fine- ane-grained contact, the most common confusions are about the object type or use vs hold: tool held is most confused with tool use and neither held, for instance. Interaction association performs well but struggles if there are heavy occlusions or shadows. Mask predictions do not perform well when it is large scene object (e.g.. the sea the boat is surfing on or the snow the ski pole is sticking into) and the edges are not sharp on thin crowded objects. Common failure modes are visualized in the supplement.
for prismatic grasps, and precision
Interaction Prediction Performance. One key methodological difference between the proposed approach and past work 533 is the mechanism for predicting interaction: (53] produces a single vector per hand that points the center of the object box the hand is interacting with. This works well on images with relatively few hands, but does not work well on many images. We compare the approaches quantitatively in Table 4 and qualitatively in Figure 5 our approach has a slight degradation when there are many (2 6) hands while the performance of (33 plummets.
Blurring. We now analyze the impact of facial blurring hand recognition by analyzing the performance of systems trained and tested with either blurred or unblurred faces. We aim to address the two questions also asked by [63] does face blurring lead to missed detections, and does a model trained on blurred faces work when tested with unblurred faces, as will be common in practice?
We report the four options in Table 5 for detection, segmentation, and representative hand state question, hand side. Full state is in the supplement. Working entirely with blurred data (VIV) causes a slight dip compared working entirely with unblurred data (X/X), but performance is close. Training on blurred data and testing on unblurred data (VIX) is virtually identical - a tiny (0.1) drop in AP. The model has few false positives on faces, typically when they are large and in the upper half of the image. This may impact deployment and could be mitigated by either pre-blurring all input to the model or adding additional hard negatives of faces.
Generalization to Ego4D. Having trained an effective model, we now use it on the Ego4D 21] dataset. We show zero-shot generalization results in Figure 4 Quantitatively, our model yields 86.9 Box AP/ / 60.0 Segmentation AP on the on the validation set.
We were unable to annotate Ego4D due to strict restrictions on data redistribution, but can use self-training to integrate Ego4D and our data. For each hand box, we add the output generated by our model plus SAM masks and retrain. We show qualitative results in the supplement, but quantitatively fine-tuning for I 110K iterations raises Ego4D performance to 90.9 Box AP/ / 66.3 Segmentation AP. Continuing to train on Hands23 does not increase for the same number of iterations performance.
8

---

Table 5: Cross-facial blurring performance for hand+object detection, and hand state. For each category, we compare training with (V) and without (X) blurring and test in each column with (V) and without (X) blurring, Performance is similar. Full hand states are similar and in the supplement.
Hand Det. X
Obj.
Det.
2nd Obj. Det.
Side X
X
Train /
84.3
84.2
51.6
51.5
44.0
42.3
95.0
95.3
Train x
82.4
84.3
50.1
52.6
43.3
43.6
94.0
95.1
Config
Pre-Pric
oct
Figure 6: Applications Enabled. Top row: searching for the bi-manual manipulation Hand Object _ 2nd Object <- Hand. We show examples selected from set of 7 automatically found ones; the full set is in the supplement. Bottom: top precision prismatic grasps from each subset plus Ego4D.
Applications: Finding Configurations of Objects and Grasps. The ability recognize grasps, hands, objects, and second objects enables us to find particular types of interactions in our data. These may help robotics research find, for instance, demonstrations of bi-manual manipulations in video data or examples of people using a precision prismatic grasp. We show some examples in Figure for particular bimanual manipulation involving tool as well as a grasp.
6 Discussion, Impacts, and Limitations
We have introduced a model that produces a rich output of hands engaged with interaction, as well dataset for supporting this model. We hope that model will enable autonomous systems to better understand interaction, thereby enabling applications ranging from parsing robotics demonstrations [68] to helping stroke rehabilitation [58]. Nonetheless, surveillance systems may benefit from the increased ability to recognize hands and held object. We expect that the domain shift (57) between data and many surveillance use-cases will make less likely, but note that there are other sorts of surveillance (e.g., via interactive home systems) that may stand to benefit
more directly.
Our model is imperfect. While detection often works well, small hands and hands undergoing occlusion often missing, and occlusions and shadows interfere with interaction association. Our grasp categorization adds richness to our understanding of the diversity of hands grasps, but may fail for grasps that fall between categories. Additionally, predictions masks are not good when facing large scene objects and it is hard to get accurate sharp boundaries for very thin objects.
Our data is reflective of its sources, which may not be representative of the data expected in many situations. Beyond the usual vision domain shifts (the real world does not look like the Internet, for instance), this non-representation can be in terms of demographics. For example, since removed minors, our model may perform poorly on children. Likewise, while we did not find orders of magnitude difference, it is likely that there are some performance differences across demographics; we thus would extensive testing before use, and discourage use in high-stakes settings. Finally, while facial blurs had limited impact performance, they did have impact, such on-face predictions of hands that downstream systems may need to account for.
Acknowledgments. This material is based upon work supported by the National Science Foundation under Grant No. 2006619. We thank the University Michigan DCO for their tireless continued support.

---

References
[1] The vision for intelligent vehicles and applications (VIVA) challenge, laboratory for intelligent and safe automobiles, UCSD. http://cerr.uesd.cduu http://r.uusd...uuiiivaiiteeeeee wwu/vvivacaalllnnge..
[2] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views, arXiv preprint arXiv: 00910, 2019.
[3] Sven Bambach, Stefan Lee, David J Crandall, and Chen Yu. Lending a hand: Detecting hands and recognizing activities in complex egocentric interactions. In Proceedings of the IEEE international conference on computer vision, pages 1949-1957, 2015.
[4] Rodrigo Benenson. Stefan Popov, and Vittorio Ferrari. Large-scale interactive object segmentation with human annotators. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 11700-11709 1709, 2019.
[5] Samarth Brahmbhatt, Chengcheng Tang. Christopher D. Twigg, Charles C. Kemp, and James Hays. ContactPose: A dataset of grasps with object contact and hand pose. In The European Conference on Computer Vision (ECCV), August 2020.
[6] Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities commercial gender classification. In Conference on faimess, accountability and transparency, pages 77-91. PMLR, 2018.
[7] Zhe Cao, 1iiaa Radosavovic, Angjoo Kanazawa, and Jitendra Malik. Reconstructing hand-object interactions in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12417- 12426, 2021.
[8] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Port 16, pages 213-229. Springer, 2020.
[9] Tatjana Chavdarova, Pierre Baqué, Stéphane Bouquet, Andrii Maksai, Cijo Jose, Timur Bagautdinov, Louis Lettry. Pascal Fua, Luc Van Gool, and François Fleuret. Wildtrack: A multi-camera dataset for dense unscripted pedestrian detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5030-5039, 2018.
[10] Yixin Chen, Sai Kumar Dwivedi, Michael Black, and Dimitrios Tzionas. Detecting human-object contact in images. arXiv preprint ar 233v:23333033375 2023.
[11] Mark R Cutkosky al. On grasp choice, grasp models, and the design of hands for manufacturing tasks. IEFE Transactions on robotics and automation, 5(3):269-279, 1989.
[12] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision. CoRR, abs/2006. 1000006..225. 13256, 2020.
[13] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision:
Collection, pipeline and challenges for ppie-kitchens-1e 1000 International Journal of Computer Vision (IJCV), 130:33-55, 2022.
[14] Ahmad Darkhalil, Dandan Shan, Bin Zhu, Jian Ma, Amlan Kar, Richard Higgins, Sanja Fidler, David Fouhey, and Dima Damen. Epic-kitchens visor benchmark: Video segmentations and object relations. In Proceedings of the Neural Information Processing Systems (NeurlPS) Track on Datasets and Benchmarks, 2022.
[15] Victor Escorcia, Ricardo Guerrero, Xiatian Zhu, and Brais Martinez. Sos! self self-supervised learning over sets of handled objects in egocentric action recognition. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23 27, 2022, Proceedings. Part XIII, pages 604 620. Springer, 2022. [16] David F. Fouhcy, Weicheng Kuo, Alexci A. Efros, and Jitendra Malik. From lifestyle vlogs to everyday interactions. In CVPR, 2018.
[17] Qichen Fu, Xingyu Liu, and Kris Kitani. Sequential voting with relational box fields for active object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2374-2383, June 2022.
[18] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé lii, and Kate Crawford. Datashects for datascts. Communications of the ACM, 64 82-22, 2021.
[19] Georgia Gkioxari, Ross Girshick, Piotr Dollar, and Kaiming He. Detecting and recognizing human-ohject interactions. In CVPR, 2018.
[20] Mohit Goyal, Sahil Modi, Rishabh Goyal, and Saurabh Gupta. Human hands as probes for interactive object understanding, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3293- 00933-303,,10 2022.
[21] Kristen Grauman, Andrew Westbury, Eugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic.
10

---

Santhosh Kumar Ramakrishnan. Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhong- cong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do. Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer. Adriano Fragomeni, Qichen Fu, Christian Fuegen, Abrham Gebreselasie, Cristina Gonzalez, James Hillis, Xuhua Huang, Yifei Huang. Wenqi Jia, Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li. Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullic Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang. Xindi Wu, Takuma Yagi, Yunyi Zhu, Pablo Arbelaez, David Crandall, Dima Damen, Giovanni Maria Farinella, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li. Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg. Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba. Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. Ego4D: Around the World in 3,000 Hours of Egocentric Video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[22] Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent I Lepetit. Honnotate: A method for 3d annotation of hand and object poses. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 3196-3206, 2020.
[23] Yana Hasson, Gül Varol, Cordelia Schmid, and Ivan Laptev. Towards unconstrained joint hand-object reconstruction from rgb videos. In 202/ International Conference on 3D Vision (3DV), pages 659-668.
JEEE, 2021.
[24] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969 2017.
[25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016. [26] Chun-Hao P Huang, Hongwei Yi. Markus Höschle, Matvey Safroshkin, Tsvetelina Alexiadis, Senya Polikovsky, Daniel Scharstein, and Michacl J Black. Capturing and inferring dense full-body human-scene contact. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13274-13222 2022.
[27] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6m 66m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Transactions on Puttern Analysis and Machine Intelligence, 36(7):1325 1339, 2014.
[28] Wei Jiang. Kwang Moo Yi, Golnoosh Samei, Oncel Tuzel, and Anurag Ranjan. Neuman: Neural human radiance field from single video. In Proceedings of the Eumpean conference on computer vision (ECCV), 2022.
[29] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete
Xiao, Spencer Whitehead, Alexander C Berg. Wan Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.000 026433.000333.... 2023.
[30] Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick. Pointrend: Image segmentation as rendering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9799-9808, 2020.
[31] Satcesh Kumar, Jonathan Zamora, Nicklas Hansen, Rishabh Jangir, and Xiaolong Wang. Graph inverse reinforcement learning from diverse videos. In Conference on Robot Learning, pages 55-66. PMIR, 2023.
[32] Taein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Marc Pollefeys. H2o: Two hands manipulating objects for first person interaction recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 10138-101 10148, 2021.
[33] Yong-Lu Li, Hongwei Fan, Zuoyu Qiu, Yiming Dou, Liang Xu, Hao-Shu Fang, Poiyang Guo, Haisheng Su, Dongliang Wang, Wei Wu, et al. Discovering a variety of objects in spatio-temporal human-object interactions. arXiv preprint 1rX1v:2211.07501, 2022.
[34] Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117-2125, 2017.
[35] Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[36] Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujic Fu, He Wang, and Li Yi. Hoi4d: A 4d egocentric dataset for category-level human-object interaction. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21013- 21022, 2022.
[37] Jeffrey Lockman and Bjöm A Kahrs. New insights into the development of human tool use. Current directions in psychological science, 26(4):330-33 2017.
[38] Zhengyi Luo, Ryo Hachiuma, Ye Yuan, and Kris Kitani. Dynamics-regulated kinematic policy for egocentric pose estimation. In Advances in Neural Information Processing Systems, 2021.
[39] Antoine Miech, Dimitri Zhukov, Jean -an--Bttttte Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. HowTo100M: Learning Text- Video Embedding by Watching Hundred Million Narrated Video Clips. In ICCV, 2019.
11

---

[40] Joao Mineiro and Gavin Buckingham. O hand, where art thou? mapping hand location across the visual field during common activities. Experimental Brain Research, pages 1-13, 2023.
[41] A. Mittal, A. Zisserman, and P. H. S. Tomr. Hand detection using multiple proposals. In BMVC, 2011.
[42]
Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. Interhand2. 6m: A dataset and haseline for 3d interacting hand pose estimation from single rgb image. In Computer VcCi 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16, pages
548-564, Springer. 2020.
[43] Supreeth Narasimhaswamy, Trung Nguyen, and Minh Hoai. Detecting hands and recognizing physical contact in the wild. 2020.
[44] Supreeth parasinhaswamy. Zhengwei Wei, Yang Wang, Justin Zhang, and Minh Hoai. Contextual attention for hand detection the wild. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9567-9577 2019.
[45] Sida Peng. Yuanqing Zhang, Yinghao Xu, Qianqian Wang. Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In CVPR, 2021.
[46] Shengyi Qian, Linyi Jim, Chris Rockwell, Siyi Chen, and David F. Fouhey, Understanding 3d object articulation in internet vidcos. In CVPR, 2022.
[47] Iijaa Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. In Conference on Robot Learning, pages 416-426. PMLR, 2023.
[48] Francesco Ragusa, Antonino Furnari. Salvatore Livatino, and Giovanni Maria Farinella. The meccano dataset: Understanding human-object interactions from egocentric videos in an industrial-tike domain. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1569-1578, January 2021.
[49] Shaoqing Ren, Kaiming Hc, Ross B. Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. NIPS, 2015.
[50] Grégory Rogez, James S Supancic, and Deva Ramanan. Understanding everyday hands in action from rgh-d images. In Proceedings of the IEFE international conference on computer vision, pages 3889-3897, 2015.
[51] Yu Rong, Takaaki Shiratori, and Hanbyul Joo. Frankmocap: Fast monocular 3d hand and body motion capture by regression and integration. arXiv preprint arXiv:2008.0823 2020.
[52] Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela
Yao. Assembly101 101: A large-scale multi-view video dataset for understanding procedural activities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21096-
21106, 2022.
[53] Dandan Shan, Jiaqi Geng, Michelle Shu, and David F Fouhey. Understanding human hands in contact at internet scale. In Proceedings of the IEEE/CVF Conference on Computer Viston and Puttern Recognition, pages 9869 9878, 2020.
[54] Kenneth Shaw, Shikhar Bahl, and Deepak Pathak. Videodex: Learning dexterity from internet videos. In Conference on Robot Learning, pages 654-663. PMLR, 2023.
[55] Linda Smith, Swapnaa Jayaraman, Elizabeth Clerkin, and Chen Yu. The developing infant creates curriculum for statistical learning. Trends in cognitive sciences, 22(4):325-336, 2018
[56] Omid Taheri. Nima Ghorbani, Michael J Black, and Dimitrios Tzionas. Grab: A dataset of whole-body human grasping objects. In Computer EisionEECY' 2020: 16th Earopean Conference. Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pages 581-600 Springer, 2020.
[57] Antonio Torralba and Alexei Efros. Unbiased look dataset bias. In CVPR 2011, pages 1521- IEEE, 2011.
[58]
Meng-Fen Tsai, Rosalie Wang, and Jose Zariffa. Recognizing hand use and hand role at home after stroke from egocentric video. arXiv preprint (tt1:2207.0092200 2022.
[59] Graham JG Upton. Fisher's exact test. Journal of the Royal Statistical Society: Series A (Statistics in Society), 155(3):395-405 1992
[60] Chen Wang. Linxi Fan, Jiankai Sun, Ruohan Zhang. Li Fei-Fei, Danfei Xu, Yuke Zhu, and Anima Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play. arXiv preprint arXiv:2302.1242: 2023.
[61] Takuma Yagi. Md Tasnimul Hasan, and Yoichi Sato. Hand-object contact prediction via motion-based pscudo-labeling and guided progressive label correction. arXiv preprint arXiv:2/ 10174, 2021.
[62] Takuma Yagi, Takumi Nishiyasu, Kunimasa Kawasaki, Moe Matsuki, and Yoichi Sato. Go-finder: a registration-free free wearable system for assisting users in finding lost objects via hand-held object discovery. In 26th International Conference Intelligent User Interfaces, pages 139-149,, 2021.
[63] Kaiyu Yang. Jacqueline Yau, Li Fei, Jia Deng. and Olga Russakovsky. A study of face obfuscation in ImageNet. In International Conference Machine Learning (ICML), 2022.
12

---

[64]
Lixin Yang. Kailin Li. Xinyu Zhan, Fei Wu, Anran Xu, Liu Liu, and Cewu Lu. Oakink: A large- scale knowledge repository for understanding hand-object interaction. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 20953-20962. 2022.
[65] Lingzhi Zhang, Shenghao Zhou, Simon Stent, and Jianbo Shi. Fine-grained egucentric hand-object
segmentation: Dataset, model, and applications. In Luropean Conference Computer Vision, pages 127-145. Springer, 2022.
[66] Yang Zheng, Yanchao Yang, Kaichun Mo, Jiaman Li. Tao Yu, Yebin Liu, Karen Liu, and Leonidas J Guibas. Gimo: Gaze-informed human motion prediction in context. arXiv preprint arXiv:2204.094433 2022.
[67] Christian Zimmermann, Duygu Ceylan, Jimei Yang. Bryan Russell, Max Argus, and Thomas Brox. Freihand: A datasct for markerless capture of hand pose and shape from single rgb images. In Proceedings of IEEE/CVF International Conference on Computer Vision, pages 813- 2019.
[68] Kateryna Zorina, David Kovar, Florent Lamiraux, Nicolas Mansard, Justin Carpentier, Josef Sivic, and Vladimir Petrik. Multi-contact task and motion planning guided by vidco demonstration. In ICRA Inte-nnnernaioonal Conference on Robotics and Automation, 2023.
13


Reviews:
[{"review_id": "m5J35VigjT", "reviewer": "Reviewer_ufdR", "summary": "This work presents a framework that outputs rich interaction information for hands that are in an interactive state on a daily basis, and a large dataset that supports the model. The framework is based on the standard RCNN object detection mechanism and has a simple structure that can be easily extended to meet subsequent information additions for interactive hand understanding.", "questions": "The narrative of the article is clear and there is little doubt. ", "limitations": "A large number of 3D hand-object interaction state estimation methods already exist in the community, which provide a more fine-grained analysis of contact. The prominent problem with current 2D-level interactions exists in the robustness to small and multiple targets, but these are subsumed into the Limitation of this work.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "- A large dataset of human hand interactions is proposed, which provides rich and high-quality data annotation that facilitates the understanding and prediction of human hand behavior.\n- A framework for manual interaction understanding, with a simple structure that is easy to extend and conducive to directly supplement more interactive information.\n- Additional categories of how human hands interact with objects are classified and understood.", "weaknesses": "- The understanding of human-object interaction is still in the form of classification mapping, which is a large and tedious classification and lacks simplicity.\n- The framework is based on the object detection approach, which may be less applicable for large scenes or scenes that are messy.\n- It could be ragarded as an incremental supplement for the work of Shan et al, not innovative enough.（ Dandan Shan, Jiaqi Geng, Michelle Shu, and David F Fouhey. Understanding human hands in contact at internet scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 482 pages 9869–9878, 2020.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their review. We respectfully disagree on a number of key points that we will describe below.\nWe believe that the reviewer is suggesting that 3D hand-object reconstruction is sufficient to solve the task we tackle, and so we will respond to this first before responding to the three weaknesses.\n\n**Q1: Can 3D hand-object reconstruction solve hand-object interaction understanding?**\n\n3D hand-object reconstruction does provide an alternate fine-grained characterization but often does not generalize well to realistic data such as internet videos, and often only works on a fixed set of objects with given 3D object models or a limited set of (often hand-held) objects. In the real world, many objects that can be in contact with hands. Some objects like large fridges and tables may be larger than the full frame and be occluded and impossible to reconstruct. Others may be tiny and too numerous to have reliable reconstructions, such as the many small pieces of hardware while assembling furniture. Sometimes hands have self-contact with the body. Further, tools interact with objects as well, which is not captured by existing hand-object reconstruction methods, to the best of our knowledge. \n\nOur method finds this interaction in realistic data, possibly with many people and without needing a fixed set of objects in advance. In fact, 2D models such as the ones we use have been used to localize hands and objects for 3D methods, such as Frankmocap which uses the Shan detector to work on isolated hands. This is, of course, in addition to a wide variety of other tasks including action recognition, interactive object understanding, and robotics manipulation learning. \n\nAs a demonstration of the difficulty of the data, we show two existing interaction models on the hands23 test set in Figure 1 in the figure response. As shown, none of the methods generalize well.\n\n**W1: The interaction formulation is large and tedious and lacks simplicity**\n\n**A1:** We respectfully disagree with this characterization. The method is simple enough to fit into one page, and we provide a few extra classes that are directly usable, as shown by the application examples such as Figure 6 and supplement section A.3. These analyses let us understand the diverse/rich types of hand/object/second-object interaction that exist. We genuinely do not understand what is meant by tedious. If the reviewer is referring to the pairwise interaction classification, we note that this is simple and can likely be further accelerated with various tricks.\n\n**W2: Are detection-based frameworks less applicable to large and messy scenes?**\n\n**A2:** We respectfully disagree. We precisely chose to formulate the problem as object detection/instance segmentation because we want to tackle large, messy scenes with unknown numbers of objects. Indeed, we show results on COCO, which has many scenes with multiple people and in fact, this is where we show a large improvement over Shan et al. in hand association (74.2 vs 55.9 F1 for 6+ hands). Detection is the standard approach for handling an unknown number of objects, and without a concrete alternative, we’re at a loss to respond.\n\n**W3: “It could be ragarded as an incremental supplement for the work of Shan et al, not innovative enough.”**\n\n**A3**: We strongly disagree with this characterization. The paper introduces (a) a richer set space of labels, including first and second object contact which is not provided at this scale in any other dataset, as well as grasps and fine-grained contact annotations; (b) annotations for 250K images with clear copyright status and data protections; (c) a method to produce the rich set of labels that improves over Shan et al., especially for complex scenes; (d) various smaller contributions such as SAM-enabled segmentations, analysis of potential biases in detection performance. It is true that there are similarities to Shan et al., which is because the core idea works well and has seen substantial use in many downstream tasks. Building on this past work is a strength, not a weakness.\n"}, {"review_id": "ZSOFzjlbcr", "reviewer": "Reviewer_oZYn", "summary": "The paper presents a method and dataset for hand-object understanding. Specifically, the dataset provides bounding boxes and segmentation annotations for (1) hands, (2) objects that are in contact with hands, and (3) second objects touched by tools. The annotation also includes contact and grasp type for second objects touched by tools. Overall, there are annotations for ~260K images spanning four different datasets. The segmentations masks are automatically extracted using the off-the-shelf method, Segment Anything.", "questions": "Please see the weakness section.", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposed dataset has images from both first-person and third-person views. This can help develop hand interaction detection methods that generalize well on different domains. \n- The dataset has images from other popular benchmarks like EPIC-KITCHENS and COCO. This could be beneficial to the community to develop and train unified methods for human understanding.\n- The authors demonstrate the benefits of data through cross-dataset experiments. \n- The proposed dataset annotates richer contact vocabulary, such as distinguishing between touching vs holding.", "weaknesses": "The proposed architecture does not convince me fully. There are a few major concerns. \n\n- Currently, the method first detects three 'object' classes: hand, object, and second object. These classes are detected independently. However, there is a clear correlation between these three classes. For instance, the object's location in contact with the hand depends on the location of the hand. Also, the location of the second object depends on the location of both the hand and the first object. However, in the current method, this is not reflected; the three classes are detected independently.\n- In L217-L218, every hand is matched to an object with the highest interaction score. This excludes the possibility that a hand can contact more than one object. For example, a hand can hold a knife and the chopping board at the same time. The current model design does not reflect this.\n- When detecting object and second-object interaction, does the model also consider hand-object interaction? For example, consider object A (first object) and object B (second object). When inferring the interaction score between the two objects, will the model consider if object A was in contact with the hand? If object A was not in contact with the hand, then object B cannot be the second interaction object. Is there any such explicit modeling? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review of the paper and respond to their three questions and weaknesses below. \n\n**Q1: Does the method model the correlation between three classes?**\n\n**A1:** Thank you for the great question. It is true that our model treats the three classes independently in terms of formulation, but by using a shared backbone, we believe that the model is capable of internally and implicitly modeling the relationships between the three classes. We choose to frame the problem as independent to make the problem amenable to existing object detection machinery. Moreover, the relationship is detected based on features from both objects as well as the relative location of paired bounding boxes (more details at line 211).\n\nTo verify our hypothesis of the implicit modeling of object relationships, we provide qualitative results from videos in Figure 2 showing all detected objects above a low threshold (0.8 for hands, 0.3 for first and second objects) without association inference. The *before* image shows the object without the hand and the *after* image shows the hand in contact. In the before example, the object is not detected or detected with a lower confidence score. In the after example, when the hand appears, the object is detected or its score increases. This suggests that the model is implicitly using the other object categories during its detection.\n\n**Q2: Can the model capture cases of one hand in contact with multiple objects?**\n\n**A2:** Thank you for the thoughtful comment about cases of a hand in contact with multiple objects. In data annotation, we annotated a single hand/object contact to make data annotation scalable. However, our model is, in principle, able to handle the multiple contact case because it predicts contact state for each pair of hand and object as an independent classification (i.e., the objects do not compete in the prediction). This is unlike the Shan et al. formulation, which explicitly precludes multiple contacts by using a single offset vector.\n\nIf accepted, we will include a more thorough analysis and investigate additional annotations. In the meantime, we did a small experiment to see if the model has this capacity (since during training, it is identifying which object is annotated as in contact). Instead of keeping only the object with the highest in-contact score, we modify the inference algorithm to keep all predicted in-contact objects with a high contact score (threshold=0.8) and low IoU with other in-contact objects (IoU threshold = 0.2). In Fig 3, we show examples of multiple-object contact: (a) a hand holding a brush and touching the surface of a phone case, (b) a hand grabbing food and putting them into the bowl, (c) a hand holding two beans, and (d) a hand holding the comb and a strand of hair. In other cases like (e), the model will find the object by two bounding boxes because the object is separated by occlusion. There are some failure cases too. \n\n**Q3: Does the model consider hand and first-object interaction before deciding on first-object and second-object interaction?**\n\n**A3:** The inference algorithm does indeed consider interactions between the hand and first-object when detecting interactions between the first and second object. In our inference, the contact association is always starting from a hand. The inference algorithm will only consider first-object and second-object contact if the first object is in contact with a hand.\n"}, {"review_id": "NcO0BFY2ND", "reviewer": "Reviewer_87MP", "summary": "This paper introduces a hand object interaction-related dataset, Hands23, that providing rich labels for hand images (segemantion and boundingbox of left hand, right hand, object and second object, types of contact, grasp and touch). It labels the existing three datasets (COCO, EPIC-KITCHENS VISOR, and Internet Articulation), as well as self-collected videos. Based on the new dataset with rich labels, the paper uses RCNN to train a multi-task model, which performs and generalizes well. ", "questions": "- [Action] As Hands23 consists of a large number of videos, what do you think about action labels?\n\n- [SAM] The segmentation labels are generated with the help of Segment Anything (Line.142). I am wondering if the paper could provide some observations or insights when working with Segment Anything or even other large models to generate labels.", "limitations": "Yeah. The authors have addressed the limitations in the conclusion section.", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- [Size and rich labels] The presented dataset is much larger than existing datasets and provide segemantion and boundingbox labels of hands and objects.\n\n- [Experiments] The model trained on Hands23 for hand or object detection shows good generalization performance. It outperforms the models trainend on other datasets.\n\n- [Annotation] The annotation procedure is elaborated in great detail in the supplementary.", "weaknesses": "- [Multi-task] The paper highlights the importance of rich labels for hands and introduces a multi-task framework (Fig.3). It would be interesting to provide more insights on the interaction of different sub-tasks. For example, do the auxiliary labels affect hand segmentation or bounding boxes? what is the benefits of richer labels?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review and answer the questions below. \n\n**Q1: What are the benefits of multi-task training? Does it hurt the performance on segmentation?**\n\n**A1:** The benefits of the multiple tasks is primarily the richer predictions, which we envision will enable various applications such as robotics, hand-grasp analysis, and parsing video datasets for understanding hands or finding demonstrations.\n\nOnce losses are scaled properly, we think that multitask training does not hurt performance much. We also trained a version consisting only of 3-class (hand, first, and second object) instance segmentation without extra heads. The performance was similar. Hands were slightly better (+3.7 detection, +3.5 segmentation), first objects were slightly worse (-1.2 detection and segmentation), and second objects were slightly worse (-1.7 detection, -0.4 segmentation). We believe this variance is within the range of random seeds and the intrinsic minor differences when comparing two setups, but will provide a more detailed analysis if accepted. \n\n**Q2: What do you think about action labels?**\n\n**A2:** We hope that our model will help with action understanding. We do not have action labels for the new videos subset of Hands23, but we chose VISOR precisely since it is connected to the excellent EPIC-KITCHENS action labels. Beyond labeling the dataset, we do believe that the output of our system could help serve for action recognition since the state of hands, first objects and second objects often are informative for action recognition.\n\n**Q3: How does SAM perform on predicting masks?**\n\n**A3:** We will include additional information about SAM if accepted, since we did find a number of interesting observations while both using SAM and trying to develop our SAM-like system. \n\nBriefly, we found that SAM generally performs well and robustly when provided with bounding box prompts. There are some limitations and cases where more failures emerge. One is that SAM has difficulty deciding the end of the wrist for hand bounding boxes and can produce jaggy masks as the wrist. Another is that SAM can have trouble with thin objects where multiple objects appear in the same box. SAM occasionally will segment another object rather than the intended one. However, overall, as shown by our analysis at L238, SAM produces quite good results. These are where failures tend to be more frequent, rather than places where SAM systematically does not work.\n\nWe also want to point out Section D in the appendix, where we discuss SAM compared to our own similarly structured system. We compare the proposed model’s performance between using masks from SAM and our own internal mask prediction system. The bounding box detection performance is largely identical, but segmentation is much better using SAM.\n"}, {"review_id": "m9JTNIb4Ic", "reviewer": "Reviewer_N4EC", "summary": "This paper builds a large-scale hand-object interaction dataset named Hands23 that contains 257K images, 401K hands, 288K objects, and 19K second 33 objects and combines different image sources (Visor, Epic-kitchen, COCO, Internet articulation and novel videos). Compared to the previous hand-object interaction datasets, it includes fine-grained contacts, detailed hand-grasp labels, second objects, and object segmentation. The resulting dataset covers a large amount of hand-object interactions in both ego and exo views. Based on the dataset, they show that a baseline model (Mask-RCNN) can perform well on Hands23 and also generalizes better to other datasets (than previous work). Thus, Hands23 provides a useful resource for the community to study hand-object interactions with applications from computer vision to robotics.\n", "questions": "Overall the paper is clear,  but I do have a few suggestions/questions: \n-\tObject labels in the figures are not too self-evident (e.g., ‘ObjC, NP-Fin’ in Figure 1, etc.)\n-\tThe caption for Table 2 does not mention the metric (the text does, but would be good to add to caption) \n-\tIn the caption of Table 2 you state: “no other dataset produces good results on Hands23.” Yet, 100DOH gives a reasonable 78.5. Likely the only difference is, thus, scale? What if you train on a fraction of Hands23 that has as many images as 100DOH? \n- Last but not least, I would find it really useful to get an assessment of the accuracy of the human labelers.\n\nOverall, this is a strong paper, and indeed I was wondering why they did not include Ego4D in the dataset. This comment clarified it: “We were unable to annotate Ego4D due to strict restrictions on data redistribution, but can use self-training to integrate Ego4D and our data” That’s great, but could already be mentioned earlier and I think Ego4D and a few other datasets (e.g. Grab, Discovering Human Interactions with Large-Vocabulary Objects via Query and Multi-Scale Detection) should be in Table 1. \n\nIn the Supp Mat it is stated that the data format for the release is not clear yet. I think it would be ideal to follow, e.g., COCO or Epic kitchen for consistency in the literature. ", "limitations": "\n", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1) The proposed dataset is large and covers a wide distribution of hand-object interaction scales in both first and third-person views. \n\n2) Data were richly annotated: hand bounding boxes and segmentation (standard), contact (follows [43], in-contact object (following [53], secondary object (novel), hand grasps (following Cutkosky taxonomy). Thus, the authors innovatively combined to provide a more comprehensive annotation than previous work at scale. Thus, the proposed dataset contains more fine-grained annotations, e.g., second object, detailed contacts and hand-grasp labels.\n\n3). Ethics, privacy and demographics were particularly well addressed. Faces were obfuscated for privacy and Demographics and Realism of the data was analyzed. \n\n4). The proposed baseline model can predict all the rich interaction information and shows good generalization ability to Ego4D dataset.\n\n5) This is a comprehensive paper with excellent Supp. Materials. ", "weaknesses": "1) One of the main advantages of the proposed dataset is the “second object” defined as an object in contact via a handheld tool. However, the motivation for introducing the second object and the corresponding possible applications are not as well discussed. \n\n2) This paper shows the zero-shot generalization to Ego4D dataset. However, Ego4D only contains the first-person view, which is less challenging due to rel. fixed hand scale and few (hand) occlusions. Therefore, it would be good if the authors could also show the zero-shot generalization to third-person data.\n\n3) While I do think the paper makes significant advances (thus my score), it is not higher, as one would hope for even richer annotations. \n\n4) One more minor comment, the authors state that “We only annotate non-crowd instances with at least 10^3 pixels of area.” While that makes sense it would also be good to annotate crowded areas (just like it is done for pose estimation data in COCO).\n\nRegarding relations to prior work I have two minor suggestions:\n- This paper gives fine-grained contacts annotation ({touch, hold} and {tool, container, neither}). However, some previous works in human object interaction (e.g., Discovering Human Interactions with Large-Vocabulary Objects via Query and Multi-Scale Detection) provide even more detailed object and contact classes (e.g., unlocking door). Therefore, the advantage of the current annotation type could be further discussed.\n- Line 29: “we extend this to a richer vocabulary that distinguishes touching, holding and using, and includes grasps” Here, GRAB [56] should also be mentioned as it does something similar. Perhaps one should also state that EPIC kitchen has a much broader vocabulary?\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed feedback which we believe will improve the paper.\n\n**Q1: Could you provide more motivation for second-objects?**\n\n**A1:** We will include a more explicit and centrally organized discussion of second objects if accepted. Since humans use tools to accomplish tasks, second objects are needed for a complete and rich understanding of hand interactions. Consider, for instance, a hand holding a knife. Without the second object, one doesn’t know whether the action is cutting bread, transferring chopped garlic to a pan, scraping something off a cutting board, or even brandishing the knife. Indeed, knowing this distinction is important for connecting to narrations: if one is drilling into a table, the second object is the part that is described as the direct object, rather than the drill. \n\n**Q2 Does the model generalize well on 3rd person data?**\n\n**A2:** Indeed, our motivation for the diversity of data used is so that the model has a good chance of generalizing well to 3rd person data. We have some partial generalization experiments while testing on other datasets as has been noted. These, however, test only hand detection. We will add generalization to other 3rd person datasets if accepted. In the meantime, we show some results on the SAM dataset in the figure pdf. We selected SAM because it is diverse, recent, and high quality. Figure 4 shows two rows of results on SAM; the top is selected and the bottom is random. Our method generalizes fairly well on unseen 3rd person data. \n\n**Q3: What about even richer annotations**\n\n**A3:** We highly welcome suggestions for more labels to further enrich Hands23. We additionally will ensure that is possible for the community to add more annotations. \n\n**Q4: In COCO, why only annotate non-crowded instances with at least $10^3$ pixel area?**\n\n**A4:** We originally did this because we thought that the crowds of COCO (often next to baseball fields or tennis courts) were too small for hands to be reliably understood and parsed. We note that $10^3$ pixels are under $32^2$, which likely leaves each hand being just a pixel. However, we will re-examine this design decision to see if there are many hands amongst crowd instances that are visible.\n\n**Q5: Suggestions about prior work, discussion of Ego4D**\n\n**A5:** Thank you. We will add a discussion about these works and add them to the table. Briefly, we think that our less fine-grained annotations can be obtained very easily and provide a great first initial parse of the interaction. We see these works as complementary, and think the community benefits from having data at different levels of granularity and scale. We will additionally move the discussion of Ego4D and difficulty annotating it earlier in the paper.\n\n**Q6: Paper suggestions**\n\n**A6:** Thank you. Will will fix these if accepted.\n\n**Q7: Is the performance gain of Hands23 primarily due to scale?**\n\n**A7:** Thanks for this great question. To check if scale really matters, we trained the hand detection model using the same amount of data (90K) as the 100DOH trainval set. The resulting model obtains 83.4 AP (Hands23 with 90K images), which is close to the original model’s 85.6 (Hands23 full), and higher than 100DOH’s 78.5 AP. This experiment suggests that not only scale, but also the diversity of the dataset matter.\n\nFurthermore, if one evaluates only on subsets of the dataset, while the 100DOH model often does as well as the Hands23 model, the 100DOH model’s performance is substantially worse on other subsets. For instance, on VISOR, New Videos, Articulation, Hands23 is better by 2.3, 2.3, and 2.6 AP respectively. However, on COCO Hands23 is better by 17.6%. We note that this is only testing hand detection; we have often found that the other aspects of performance struggle (e.g., 100DOH is known to struggle with egocentric hand side as reported on the project website)\n\n**Q8: Will the dataset format match prior work?**\n\n**A8:** Yes, we intend to have the dataset format match prior work so that it is as easy as possible for the community to use.\n"}]

Author rebuttal:
We put all figures mentioned for each response in this PDF.

Meta-review: The reviews on this paper are mixed.  Two of the reviewers recommend acceptance, while two others recommend rejection.  One weakness, as pointed out by several reviewers, relate to the ambiguity in the definitions and formulation of the interacting and second interacting objects.  

Having carefully read the paper, reviews, and author responses, the AC recommends accepting the paper.  The scale and collection efforts of the paper are impressive and will no doubt benefit the community working on downstream tasks that will use the hand detector.
Decision: Accept (poster)