Title: Towards A Richer 2D Understanding of Hands at Scale
Venue: NeurIPS 2023
Accepted: yes
Keywords: human-object interaction; hand object detection; hand detection
Abstract: As humans, we learn a lot about how to interact with the world by observing others interacting with their hands. To help AI systems obtain a better understanding of hand interactions, we introduce a new model that produces a rich understanding of hand interaction. Our system produces a richer output than past systems at a larger scale. Our outputs include boxes and segments for hands, in-contact objects, and second objects touched by tools as well as contact and grasp type. Supporting this method are annotations of 257K images, 401K hands, 288K objects, and 19K second objects spanning four datasets. We show that our method provides rich information and performs and generalizes well.

Towards A Richer 2D Understanding of Hands at Scale

Abstract

As humans, we learn a lot about how to interact with the world by observing others interacting with their hands. To help AI systems obtain a better understanding of hand interactions, we introduce a new model that produces a rich understanding of hand interaction. Our system produces a richer output than past systems at a larger scale. Our outputs include boxes and segments for hands, in-contact objects, and second objects touched by tools as well as contact and grasp type. Supporting this method are annotations of 257K images, 401K hands, 288K objects, and 19K second objects spanning four datasets. We show that our method provides rich information and performs and generalizes well.

1 Introduction

From the moment we wake up to the moment we go to sleep, most of us actively use our hands to shape and interact with the world. As infants, our own ability to interact with the world is accelerated and guided by our observations of others using their hands [55, 37]. These interactions involve not just the objects we directly hold, but also tools that enable us to interact with second objects, such as a spatula that enables us to push an egg in a pan. As adults, as demonstrated by the endless supply of instructional videos on the Internet [39], we can parse demonstrations of activities to help guide our own efforts; we can follow hands, the objects they hold, the grasps they use, and how they use tools to interact with the world. Accordingly, there have been many efforts on 2D hand understanding, each focusing on a different aspect. These include large-scale bounding box datasets [53], segmentation datasets [65, 3, 14], and in-lab grasp and contact efforts [50, 5]. Each contributes to the literature, but obtaining a unified understanding of hands in action, from box to segment to grasp to tool use, requires stitching together many models and datasets.

We contribute a model that predicts a rich, unified 2D output of hands interacting with objects that is effective on a variety of first and third person data. Our model predicts boxes for hands and in-contact/active objects like past efforts [53, 17], as well as second objects , or objects contacted by an in-use tool, which in turn enables understanding how humans use tools to manipulate their world. Like past work [53, 44], our model predicts when hands contact objects; we extend this to a richer vocabulary that distinguishes touching, holding and using, and includes grasps. Finally, by converting boxes to segments [29], our system produces segments for hands, objects, and second objects.

These outputs are built on two key components - a model and a dataset. The model is a straightforward, easily-extensible deep network for parsing interaction from an image. We build on standard detection machinery [24] with standard losses and use a simple inference method. Powering this model is a large dataset, Hands23, consisting of 257K images, 401K hands, 288K objects, and 19K second objects. Hands23 provides unified annotations for four datasets: EPIC-KITCHENS [13] VISOR [14], the 2017 train set of COCO [35], Internet Articulation [46], as well as on our introduced dataset of interaction-rich videos, New Days. All Internet data has Creative Commons licenses and has had faces obfuscated after automatic and manual detection as well as minors removed from the dataset.

Articulation

COCO 2017

EPIC-KITCH. VISOR

Right, Object Contact,

Tool

Used

Neither:toached

ObjC,Pow-Pris

UjcPre-Pris

0b0NPFin

We analyze our approach and dataset through a series of experiments. Across datasets, our results show that hand detection models trained on our dataset have strong zero-shot performance compared against past hand detection datasets [53, 16, 1, 44], demonstrating the expansiveness of Hands23. Within our new dataset, our experiments demonstrate that our model can detect our detailed handobject state both well and better than past efforts such as [53]. Past work in this space [53] found adoption in a variety of communities interested in learning from hand-object interaction, ranging from parsing robotics demonstrations [31, 60, 68] to guiding video understanding [20, 15] and more [58, 40, 62]. To this end, we conclude by showing applications of our model to finding grasps in action as well as on external datasets.

2 Related Work

Our work aims to provide a model and dataset for obtaining rich information about hands in a wide variety of images. Our key contribution lies in enabling richer output, which includes second objects that are interacted with by tools, more fine-grained contact, grasps, and segments. As such our work spans a few directions in computer vision, machine learning, and robotics.

2D Hand Understanding. There have been extensive efforts to understand hands interacting with objects. One direction has been generating increasingly rich datasets for egocentric [3, 1, 12, 14, 65, 48, 21] and third person data [41, 44, 43, 53, 22, 42], with each dataset covering a particular use case and perspective. Our new dataset, Hands23 provides a unified dataset which builds on and annotates four data sources: EPIC-KITCHENS VISOR [14], COCO [35], Internet Articulation [46], and a new video dataset. Hands23 contributes to this literature on three fronts: it introduces a richer contact vocabulary than [43, 53, 21, 14] that includes fine-grained contact in terms of touching-vs-holding and tools, grasps building on the vocabulary of [11], second objects, and segments automatically obtained from SAM [29]; second, it spans both first and third person data; finally, to the best of our knowledge, it is the largest with 250K images and 400K hands. There are specialized datasets still richer than ours: [50] has a fine-grained grasp vocabulary and [65] has human-annotated segments; however, these datasets are more limited in scope and are complementary to our efforts.

To take advantage of this new data, we introduce a new detection-based model that can predict our new rich output state. We build on standard detection machinery [24] to accomplish our goal. With the advent of 2D understanding datasets, there has been a flurry of works proposing new mechanisms to recognize active objects [17, 61] and improve detection and segmentation performance [65]. We see this work as complimentary and likely to assist our model as well.

Applications of 2D Hand Understanding. The ability to reliably obtain basic 2D information about hands has unlocked many use cases. Naturally, some have focused on hands themselves, such as making more diverse and challenging data accessible to 3D hand reconstruction systems [7, 23, 51]. However, since hands are the key to many interactions, many other domains stand to benefit. These include ML areas like vision, where hands can guide video understanding [20, 15], as well as robotics, where hand-object detection helps parse natural human demonstrations [31, 60, 68] and video data is used for pretraining [47, 54]. Excitingly, reliable detection enables applications like post-stroke rehabilitation [58], understanding the human visual experience for neuroscience [40], and finding lost objects [62]. Our work enhances these efforts by providing richer and more accurate outputs.

Understanding Whole-Body and 3D Interactions. Our focus is specifically hands and objects in 2D, but there is a wide literature on understanding interactions with different goals. The most related threads of research are whole body interaction and 3D hand understanding. Whole-body interactions [33, 10, 26, 56, 66, 38, 27, 45, 28] are usually evaluated on different styles of data and pose orthogonal challenges. Similarly, full 3D understanding of hands in action [64, 67, 5, 52, 42, 22, 36, 32] usually provide a richer 3D output space at the cost of working in more constrained settings. Our work may help bring these 3D techniques to more unconstrained settings.

3 Dataset

We contribute a new dataset of images of hands engaged in interaction with the world. All data is Creative Commons or was user-collected [12] and has had faces blurred and minors removed for privacy. The data covers four datasets: a new video dataset that we refer to as New Days, COCO [35], EPIC-KITCHENS [13] VISOR [14], and Internet Articulation videos [46]. These disparate datasets are unified by a new set of annotations including: (1) boxes for hand s, held objects , and second objects ; (2) contact information that distinguishes touching, holding, and using; (3) grasp contact labels for a subset of the data; and (4) automatically extracted segments for boxes. We include a datasheet [18] for our proposed dataset in the supplement.

3.1 Underlying Image Data

Our underlying data is obtained from four different datasets. These datasets cover still images, third person video, and egocentric video.

New Video Dataset. We gather a fresh set of video data of people engaged in interaction with the world, following the general strategy outlined in [16, 53]. The supplemental contains the full details, but in short, we follow three steps: we generate a vast pool of potential videos via a combinatorial set of text queries; we automatically select videos as likely to contain interactions (and unlikely to contain children, lecturing, or have cartoons) using a classifier trained on video thumbnails; finally, we sample frames from the selected videos to contain a mix of up-close and far-away scenes.

Past Data. We use three public datasets: the 2017 training set of COCO [35]; EPIC-KITCHENS VISOR [14, 13], and Internet Articulation [46]. We use the full dataset from Internet Articulation and VISOR, and use COCO images containing non-crowd people with area ≥ 10 3 pixels.

Data Splits. We provide ≈ 80 / 10 / 10% train/val/test splits that split video data by channel and are backwards compatible with existing datasets. These are documented in the supplement.

Copyright & Privacy. All underlying image data used are public; when posted by users, they were posted publicly with a Creative Commons license selected. Obtaining explicit permission from each user for inclusion in the data is impractical due to the scale of the data. However, for privacy, we have: (1) blurred faces of people via automatic methods combined with multiple rounds of manual correction; (2) found and removed images containing minors; and (3) will provide a means for people to remove their data from the dataset. These procedures are detailed in the supplemental.

0.0

0.1

0.2

0.3

0.4

0.5

Normalized Hand Size

10 0

10 1

Density

Hands23

100DOH

New Videos

VISOR

COCO

Hands23

New Videos

VISOR

COCO

3.2 Annotations

We provide unified annotations for all four datasets. Annotation is needed, even for a dataset like VISOR [14] that already has annotations, as VISOR 'hands' have a varied definition that includes the forearm.

Hand and Side. All annotations are built on top of a hand. We obtain annotations for left/right hands as bounding boxes, defining hands as terminating at the wrist following [53, 43]. To avoid tiny hands in COCO [35], we only annotate non-crowd instances with at least 10 3 pixels of area.

Auxiliary Labels. For each identified hand, we obtain annotations for a rich set of labels, including boxes for contacted objects and secondary objects.

Contact. We label contact following the contact vocabulary of [43]: { no , object , self , other-person }.

Fine-Grained Contact: Holding-vs-Touching and Tools and Containers. Past datasets [53, 43, 14] have not distinguished touching and holding objects. For more nuance, we obtain annotations for { use tool } and the Cartesian product of { touch , hold } and { tool , container , neither }. We define tools as objects that are used like a hand to physically interact (e.g., spoons, tennis rackets) and containers as objects that can hold something (e.g., bins, trays, bottles). We distinguish use from hold by requiring physical contact with a second object, and distinguish hold from touch by whether the hand is expected to drop the object. We provide these annotations only to provide potentially useful information if a downstream task aligns with these definitions. Tasks that do not distinguish tools can, for instance, marginalize over the probabilities to obtain touch/hold/use.

In Contact Object, Secondary Objects. For hands in contact with objects, we also obtain annotations of the box of the contacted object following [53]. For self and other-person contact, this box is the full person. Furthermore, for in-contact objects that are tools, we obtain annotations of the box of the second object (e.g., lettuce if a hand is holding a knife that is cutting lettuce).

Hand Grasp. For a 65K subset of hands in contact, we obtain seven grasp types. The first distinction annotated is Prehensile grasps (holding) and Non-Prehensile grasps (non-holding). Prehensile grasps are then divided into five categories from the Cutkosky [11] taxonomy, producing the Cartesian product of { Power, Precision } and { Prismatic, Circular } as well Lateral grasps. Non-Prehensile grasps are also divided into Finger Only and Palm grasps.

Segmentation. We automatically generate segments for the hands, first objects, and second objects by converting boxes to segments. We use the Segment Anything [29] Model (SAM) and prompt it with a box. For hands, we accept the SAM output unmodified, while for first objects and second object, we take the output with the hand mask subtracted.

3.3 Dataset Analysis

We next analyze the data. Examples of the proposed data appear throughout the paper, but we report some summary statistics, as well as a comparison with other datasets.

Data Analysis. We analyze the subsets of the data that have been annotated in Table 1 and compare with the most comparable dataset, 100DOH [53]. An extended table with more datasets appears in

the Supplement. Our new video dataset New Days has similar statistics to 100DOH and the full Hands23 is 2 . 5 × the size of 100DOH.

Due to photographic bias [57], each subset of Hands23 contributes a unique challenge: VISOR shows many tools in use and thus second objects, and COCO has many images with many people but few second objects (since still images rarely capture actions-in-progress). We spatially analyze the hands in Figure 2. Hands23 shows good coverage across sizes and locations. On the other hand, COCO and VISOR have a strong preferential size due to photographic biases (unlike internet datasets like 100DOH and New Days). VISOR similarly shows a strong left/right bias for locations of left/right hands, and COCO has few hands in the top of the image. Together, however, the datasets provide good coverage of a wide variety of scenarios.

Cross-Dataset Performance. To analyze Hands23's difficulty, we do cross-dataset analysis for hand detection. We train a ResNet-101 [25] Faster-RCNN [49] to follow [53]. We report zero-shot generalization performance compared to within-dataset train/test performance in Table 2. Training on Hands23 is often close to within-dataset performance. On the other hand, training on other datasets and testing on Hands23 produces poor performance. 100DOH [53] is the best performing dataset and produces a 7% AP drop (in addition to having a less rich set of labels).

Demographics and Realism. Hands23 has the demographics of its underlying data and is thus not representative of the world's people. We did an initial check for the orders of magnitude differences seen in other models [6]. We compared results across binarized perceived gender presentation and Fitzpatrick scale (1-3 vs 4-6) following [6]; we note the same caveats about binarization.

We obtained 100 samples of each category and compared hand/object false positive/negative rates using the model described in subsequent sections. VISOR was both substantially easier for detection and heavily skewed light-skinned in our analysis. Controlling for all such difficulty factors relating to demographics (e.g., video styles) is challenging, but to control for this known factor, we excluded VISOR. A full evaluation is in the supplement, but most error rates were close and within 1 standard error of the mean: e.g., hand FPRs across skin tone were 3 . 2 ± 1 . 8 (darker) vs 3 . 0 ± 2 . 1 (lighter) and across perceived gender were 4 . 3 ± 2 . 1 (female) vs 2 . 2 ± 1 . 6 (male). We did observe a slight increase in object FPR across perceived gender ( 24 . 5 ± 4 . 9 female vs 16 . 9 ± 4 . 3 male) and skin tone ( 16 . 0 ± 3 . 8 darker vs 12 . 1 ± 4 . 0 lighter), although neither was significant according to Fisher's Exact Test [59]. We nonetheless would recommend that users be cautious and not deploy the system in high stakes settings.

Regardless of demographics, the content of the data is not necessarily representative of everyday life. For instance, many people who are eating are doing performative or competitive eating. Thus, while the data depicts actual interactions of people with objects (e.g., how one picks up an egg), we urge caution in learning priors (e.g., will someone eat 50 hard boiled eggs).

4 Method

For simplicity, we build our approach on top of standard RCNN object detection machinery [49, 19]. Our goal is to provide a simple and easily extended model for the community to build upon. In addition to predicting the object class, bounding box regression, and segment from a ROI pooling layer, we also predict a set of auxiliary labels. Our experiments are done with a standard MaskRCNN [19] model, but our approach can be generalized to other frameworks like DETR [8]. Conceptually, the

Interaction Per ROI Pair

Output

Input

Detection Per ROI

Side

Grasp

Detection

Contact

Touch

Side:

{left, right}

Contact:

{no, object, self, other-person}

Touch:

{use_tool} + {touch, hold} × {tool, container, neither}

Grasp, Prehensile:

{power, precision} × {prismatic, circular} + lateral

Non-Prehensile:

{finger only, palm}

underlying detection architecture needs to provide a feature f i for the i th hypothesized box. Given this feature, we predict the labels annotated in our dataset as follows (illustrated in Figure 3):

Standard Detection. The model predicts the standard targets, consisting of the object class and bounding box regression parameters. In our setting, in addition to the implicit background class, there are three foreground classes: hand (regardless of side), object , and second object .

Segmentation. We predict a segment using a PointRend [30] head that is set up identically to [30].

Auxiliary Hand Information: Hand Side, Contact, Fine-Grained Contact, Grasp. The model predicts auxiliary hand task as independent multiclass outputs using independent two-layer MLPs.

Hand/Object, Object/Second-Object Interaction. We train the network to classify whether pairs of objects are interacting. This is distinct from past work on hand-object association [53, 19] that regresses an offset vector for the hand that is compared with detected object centers.

Given features f i and f j for boxes i and j , the network classifies the interaction between boxes i and j . The network inputs the concatenation f i , f j , and a relational feature ϕ ij and as output, predicts a multiclass output of no contact, object contact, self contact, other-person, or object-second-object. Hand/Object pairs are only trained on the first four classes; Object/Second-Object pairs are only trained on the first and last category; impossible combinations such as Hand/Hand are not trained. Because the interaction MLP is evaluated on the many pairs of boxes, it is a small, two layer network.

The relation feature ϕ ij aims to help the network quickly reject unlikely interactions and includes: the vector v between the box centers and its normalized version v / || v || 2 2 . To help recognize boxes that overlap, ϕ i,j contains the minimum and maximum values of the signed distance function for box i , evaluated at box j 's corners, and scaled proportional to box i 's diagonal: these are negative if box j 's corners are inside box i and represent the distance to box i measured in hand-units.

Inference. For inference, we apply a simple greedy method. We start with all hand, object, and second object boxes with detection scores ≥ t H , ≥ t O and ≥ t S respectively. Then, for every hand, we find the object with the highest predicted interaction score; if the hand-object interaction score is ≥ t A , or the tool-interaction score is ≥ t I then we accept the interaction. We associate second objects to objects using an identical algorithm and threshold. Objects without a hand, and second objects without an object are removed. We set t H = 0 . 7 , T O = 0 . 5 , T S = 0 . 3 , T A = 0 . 1 and T I = 0 . 7 via grid search on the validation set for thresholds that achieve best evaluation results. The performance is relatively insensitive to the hand threshold, but low thresholds for hands led to lead to poor performance on interaction prediction.

Training and Loss Balancing. We train using a standard Mask-RCNN recipe. All of our additional losses are standard multinomial classification problems, and so we use a standard cross-entropy loss. We sum all the losses together and weight all but one equally: we scale the association loss by 0 . 1 , which we found to stabilize training. For the backbone, we use a standard FPN [34] ResNet101 [25] backbone. Full details are in the supplement.

5 Experiments

We evaluate hand, object, and second-object detection. We also evaluate our auxiliary heads, including hand side, contact, and grasp. We demonstrate results on our new Hands23 dataset, and

Ours

Ours

Ego4D

OlCPoe-Pri

100DOH

Ours

≤ 2

≤ 5

≤ 5

more

more

more

h-hNN

Neither:held

Neither:held

Neither:heldl:held

Neither:touched

ObjC,Pow

SelfC,Pow-Pris

SelfC,Pre-Pris

ObObjC,Pow-l

NoC

ObjC,Pow-Pris

ObjC,Pow-Pris

OCNNIin

SelfC,Pow-PObjC.Pow-Cire

Nei

ObjC,Pow-Pris

NoC

R

NoC

NoC

R

NoC

NObjCPre-Pris

ION

RON

RONN

0

R.PAN

R

RON

L

RON

IAN

[-P

R-

show applications of our method on entirely unseen data. In running these experiments, we aim to validate the usefulness of our hand-object detector to the broader community.

5.1 Experimental Setup

We evaluate primarily on the new Hands23, using splits defined in §3. We now define the metrics used for our output, covering box detection, segmentation, hand state and interaction classification.

Detection. Following past work [44, 43, 53], we use Average Precision with an IOU threshold of 0 . 5 . This lower threshold is used because the precise hand/forearm boundary is often unclear in images. We report the COCO AP (averaged over IOU thresholds) in the supplement.

Segmentation. We evaluate as in instance segmentation; for consistency, we use a threshold of 0 . 5 . Our ground-truth comes by prompting SAM [29] with our ground-truth boxes. We evaluated the SAM masks compared to crowd annotations. We obtained annotations for segments of 1000 hands and 1000 in-contact objects and evaluated on segments that workers came to consensus on. Median IoUs were high (hands: 0 . 84 , objects: 0 . 89 ) and most objects had IoU ≥ 0 . 75 (hands: 72% , objects: 77% ). IoUs are close to those for COCO reported in [4]. We hypothesize that the slight drop in precision for hands is due to differences in where hands end.

Hand State. Past work [53] evaluated hand state questions (e.g., side) by calculating AP, but treating hands as correct only if both the box and state are correct. This metric mixes detection and state and is hard to interpret. Since hand detection performs well ( ≈ 85% mAP), we instead evaluate accuracy on the true positives, or the chance that correctly detected hands have the correct hand state.

Hand Association. Following the approach for hand state, we evaluate hand detection at a fixed operating point. For each correctly detected hand, we analyze whether there is a correctly associated object box. We define true positives as hands for which there there is an object box and the predicted object box has IOU ≥ 0 . 1 ; we set a low IoU to account for ambiguities in the object's extent and to avoid also implicitly evaluating object detection. We define false positives as false detections, and false negatives as hands with ground-truth object boxes and no predicted boxes.

5.2 Results

We next analyze the performance of the system. We evaluate multiple outputs on a variety of datasets.

Performance on Hands23. We show qualitative results in Figure 4, including random results and quantitative performance in Table 3. More analysis appears in the supplement. Our method produces strong performance on this data: hands are detected accurately, and objects and second objects show strong performance. Performance does drop when hands and objects are small, and in cluttered images. Common grasp confusions include precision-vs-power for prismatic grasps, and precision prismatic and non-prehensile fingers. Contact is recognized well. For fine-grained contact, the most common confusions are about the object type or use vs hold: tool held is most confused with tool use and neither held , for instance. Interaction association performs well but struggles if there are heavy occlusions or shadows. Mask predictions do not perform well when it is a large scene object (e.g., the sea the boat is surfing on or the snow the ski pole is sticking into) and the edges are not sharp on thin or crowded objects. Common failure modes are visualized in the supplement.

Interaction Prediction Performance. One key methodological difference between the proposed approach and past work [53] is the mechanism for predicting interaction: [53] produces a single vector per hand that points to the center of the object box the hand is interacting with. This works well on images with relatively few hands, but does not work well on many images. We compare the approaches quantitatively in Table 4 and qualitatively in Figure 5: our approach has a slight degradation when there are many ( ≥ 6 ) hands while the performance of [53] plummets.

Blurring. We now analyze the impact of facial blurring on hand recognition by analyzing the performance of systems trained and tested with either blurred or unblurred faces. We aim to address the two questions also asked by [63]: does face blurring lead to missed detections, and does a model trained on blurred faces work when tested with unblurred faces, as will be common in practice?

We report the four options in Table 5 for detection, segmentation, and one representative hand state question, hand side. Full state is in the supplement. Working entirely with blurred data ( ✓ / ✓ ) causes a slight dip compared to working entirely with unblurred data ( ✗ / ✗ ), but performance is close. Training on blurred data and testing on unblurred data ( ✓ / ✗ ) is virtually identical - a tiny ( 0 . 1 ) drop in AP. The model has few false positives on faces, typically when they are large and in the upper half of the image. This may impact deployment and could be mitigated by either pre-blurring all input to the model or adding additional hard negatives of faces.

Generalization to Ego4D. Having trained an effective model, we now use it on the Ego4D [21] dataset. We show zero-shot generalization results in Figure 4. Quantitatively, our model yields a 86.9 Box AP / 60.0 Segmentation AP on the on the validation set.

We were unable to annotate Ego4D due to strict restrictions on data redistribution, but can use self-training to integrate Ego4D and our data. For each hand box, we add the output generated by our model plus SAM masks and retrain. We show qualitative results in the supplement, but quantitatively fine-tuning for 110K iterations raises Ego4D performance to 90.9 Box AP / 66.3 Segmentation AP. Continuing to train on Hands23 does not increase for the same number of iterations performance.

Pre-Pris

ObjC,Pre-Pris

Cont:touched

WObjCNP-Fin

HPSPEC1

Pre-Pris 0.98

X36d

20

OLED - 4K -

Pre-Pris 1.00

CPU15-103

4CORES.S

TOP5W-370

Pre-Pris 0.99

SSD 256 G8

RAM 8GB

ONB0MG

Applications: Finding Configurations of Objects and Grasps. The ability to recognize grasps, hands, objects, and second objects enables us to find particular types of interactions in our data. These may help robotics research find, for instance, demonstrations of bi-manual manipulations in video data or examples of people using a precision prismatic grasp. We show some examples in Figure 6 for a particular bimanual manipulation involving a tool as well as a grasp.

6 Discussion, Impacts, and Limitations

We have introduced a model that produces a rich output of hands engaged with interaction, as well as a dataset for supporting this model. We hope that our model will enable autonomous systems to better understand interaction, thereby enabling applications ranging from parsing robotics demonstrations [68] to helping stroke rehabilitation [58]. Nonetheless, surveillance systems may benefit from the increased ability to recognize hands and held object. We expect that the domain shift [57] between our data and many surveillance use-cases [9] will make use less likely, but note that there are other sorts of surveillance (e.g., via interactive home systems) that may stand to benefit more directly.

Our model is imperfect. While detection often works well, small hands and hands undergoing occlusion are often missing, and occlusions and shadows interfere with interaction association. Our grasp categorization adds richness to our understanding of the diversity of hands grasps, but may fail for grasps that fall between our categories. Additionally, predictions on masks are not good when facing large scene objects and it is hard to get accurate sharp boundaries for very thin objects.

Our data is reflective of its sources, which may not be representative of the data expected in many situations. Beyond the usual vision domain shifts (the real world does not look like the Internet, for instance), this non-representation can be in terms of demographics. For example, since we removed minors, our model may perform poorly on children. Likewise, while we did not find orders of magnitude difference, it is likely that there are some performance differences across demographics; we thus would extensive testing before use, and discourage use in high-stakes settings. Finally, while facial blurs had limited impact on performance, they did have impact, such as on-face predictions of hands that downstream systems may need to account for.

Acknowledgments. This material is based upon work supported by the National Science Foundation under Grant No. 2006619. We thank the University of Michigan DCO for their tireless continued support.

References

The vision for intelligent vehicles and applications (VIV A) challenge, laboratory for intelligent and safe automobiles, UCSD. http://cvrr.ucsd.edu/vivachallenge/.

Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprint arXiv:1906.00910 , 2019.

Sven Bambach, Stefan Lee, David J Crandall, and Chen Yu. Lending a hand: Detecting hands and recognizing activities in complex egocentric interactions. In Proceedings of the IEEE international conference on computer vision , pages 1949-1957, 2015.

Rodrigo Benenson, Stefan Popov, and Vittorio Ferrari. Large-scale interactive object segmentation with human annotators. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11700-11709, 2019.

Samarth Brahmbhatt, Chengcheng Tang, Christopher D. Twigg, Charles C. Kemp, and James Hays. ContactPose: A dataset of grasps with object contact and hand pose. In The European Conference on Computer Vision (ECCV) , August 2020.

Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency , pages 77-91. PMLR, 2018.

Zhe Cao, Ilija Radosavovic, Angjoo Kanazawa, and Jitendra Malik. Reconstructing hand-object interactions in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1241712426, 2021.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16 , pages 213-229. Springer, 2020.

Tatjana Chavdarova, Pierre Baqué, Stéphane Bouquet, Andrii Maksai, Cijo Jose, Timur Bagautdinov, Louis Lettry, Pascal Fua, Luc Van Gool, and François Fleuret. Wildtrack: A multi-camera hd dataset for dense unscripted pedestrian detection. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5030-5039, 2018.

Yixin Chen, Sai Kumar Dwivedi, Michael J Black, and Dimitrios Tzionas. Detecting human-object contact in images. arXiv preprint arXiv:2303.03373 , 2023.

Mark R Cutkosky et al. On grasp choice, grasp models, and the design of hands for manufacturing tasks. IEEE Transactions on robotics and automation , 5(3):269-279, 1989.

Dima Damen, Hazel Doughty, Giovanni Maria Farinella, , Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision. CoRR , abs/2006.13256, 2020.

Dima Damen, Hazel Doughty, Giovanni Maria Farinella, , Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100. International Journal of Computer Vision (IJCV) , 130:33-55, 2022.

Ahmad Darkhalil, Dandan Shan, Bin Zhu, Jian Ma, Amlan Kar, Richard Higgins, Sanja Fidler, David Fouhey, and Dima Damen. Epic-kitchens visor benchmark: Video segmentations and object relations. In Proceedings of the Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks , 2022.

Victor Escorcia, Ricardo Guerrero, Xiatian Zhu, and Brais Martinez. Sos! self-supervised learning over sets of handled objects in egocentric action recognition. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XIII , pages 604-620. Springer, 2022.

David F. Fouhey, Weicheng Kuo, Alexei A. Efros, and Jitendra Malik. From lifestyle vlogs to everyday interactions. In CVPR , 2018.

Qichen Fu, Xingyu Liu, and Kris Kitani. Sequential voting with relational box fields for active object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2374-2383, June 2022.

Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. Datasheets for datasets. Communications of the ACM , 64(12):86-92, 2021.

Georgia Gkioxari, Ross Girshick, Piotr Dollar, and Kaiming He. Detecting and recognizing human-object interactions. In CVPR , 2018.

Mohit Goyal, Sahil Modi, Rishabh Goyal, and Saurabh Gupta. Human hands as probes for interactive object understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3293-3303, 2022.

Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic,

Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do, Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Christian Fuegen, Abrham Gebreselasie, Cristina Gonzalez, James Hillis, Xuhua Huang, Yifei Huang, Wenqi Jia, Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi, Yunyi Zhu, Pablo Arbelaez, David Crandall, Dima Damen, Giovanni Maria Farinella, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. Ego4D: Around the World in 3,000 Hours of Egocentric Video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2022.

Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: A method for 3d annotation of hand and object poses. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3196-3206, 2020.

Yana Hasson, Gül Varol, Cordelia Schmid, and Ivan Laptev. Towards unconstrained joint hand-object reconstruction from rgb videos. In 2021 International Conference on 3D Vision (3DV) , pages 659-668. IEEE, 2021.

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision , pages 2961-2969, 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Chun-Hao P Huang, Hongwei Yi, Markus Höschle, Matvey Safroshkin, Tsvetelina Alexiadis, Senya Polikovsky, Daniel Scharstein, and Michael J Black. Capturing and inferring dense full-body human-scene contact. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13274-13285, 2022.

Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Transactions on Pattern Analysis and Machine Intelligence , 36(7):1325-1339, jul 2014.

Wei Jiang, Kwang Moo Yi, Golnoosh Samei, Oncel Tuzel, and Anurag Ranjan. Neuman: Neural human radiance field from a single video. In Proceedings of the European conference on computer vision (ECCV) , 2022.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643 , 2023.

Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick. Pointrend: Image segmentation as rendering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9799-9808, 2020.

Sateesh Kumar, Jonathan Zamora, Nicklas Hansen, Rishabh Jangir, and Xiaolong Wang. Graph inverse reinforcement learning from diverse videos. In Conference on Robot Learning , pages 55-66. PMLR, 2023.

Taein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Marc Pollefeys. H2o: Two hands manipulating objects for first person interaction recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10138-10148, 2021.

Yong-Lu Li, Hongwei Fan, Zuoyu Qiu, Yiming Dou, Liang Xu, Hao-Shu Fang, Peiyang Guo, Haisheng Su, Dongliang Wang, Wei Wu, et al. Discovering a variety of objects in spatio-temporal human-object interactions. arXiv preprint arXiv:2211.07501 , 2022.

Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2117-2125, 2017.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision , pages 740-755. Springer, 2014.

Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. Hoi4d: A 4d egocentric dataset for category-level human-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2101321022, 2022.

Jeffrey J Lockman and Björn A Kahrs. New insights into the development of human tool use. Current directions in psychological science , 26(4):330-334, 2017.

Zhengyi Luo, Ryo Hachiuma, Ye Yuan, and Kris Kitani. Dynamics-regulated kinematic policy for egocentric pose estimation. In Advances in Neural Information Processing Systems , 2021.

Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips. In ICCV , 2019.

Joao Mineiro and Gavin Buckingham. O hand, where art thou? mapping hand location across the visual field during common activities. Experimental Brain Research , pages 1-13, 2023.

A. Mittal, A. Zisserman, and P. H. S. Torr. Hand detection using multiple proposals. In BMVC , 2011.

Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. Interhand2. 6m: A dataset and baseline for 3d interacting hand pose estimation from a single rgb image. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16 , pages 548-564. Springer, 2020.

Supreeth Narasimhaswamy, Trung Nguyen, and Minh Hoai. Detecting hands and recognizing physical contact in the wild. 2020.

Supreeth Narasimhaswamy, Zhengwei Wei, Yang Wang, Justin Zhang, and Minh Hoai. Contextual attention for hand detection in the wild. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9567-9576, 2019.

Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In CVPR , 2021.

Shengyi Qian, Linyi Jin, Chris Rockwell, Siyi Chen, and David F. Fouhey. Understanding 3d object articulation in internet videos. In CVPR , 2022.

Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. In Conference on Robot Learning , pages 416-426. PMLR, 2023.

Francesco Ragusa, Antonino Furnari, Salvatore Livatino, and Giovanni Maria Farinella. The meccano dataset: Understanding human-object interactions from egocentric videos in an industrial-like domain. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages 1569-1578, January 2021.

Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS , 2015.

Grégory Rogez, James S Supancic, and Deva Ramanan. Understanding everyday hands in action from rgb-d images. In Proceedings of the IEEE international conference on computer vision , pages 3889-3897, 2015.

Yu Rong, Takaaki Shiratori, and Hanbyul Joo. Frankmocap: Fast monocular 3d hand and body motion capture by regression and integration. arXiv preprint arXiv:2008.08324 , 2020.

Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao. Assembly101: A large-scale multi-view video dataset for understanding procedural activities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2109621106, 2022.

Dandan Shan, Jiaqi Geng, Michelle Shu, and David F Fouhey. Understanding human hands in contact at internet scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9869-9878, 2020.

Kenneth Shaw, Shikhar Bahl, and Deepak Pathak. Videodex: Learning dexterity from internet videos. In Conference on Robot Learning , pages 654-665. PMLR, 2023.

Linda B Smith, Swapnaa Jayaraman, Elizabeth Clerkin, and Chen Yu. The developing infant creates a curriculum for statistical learning. Trends in cognitive sciences , 22(4):325-336, 2018.

Omid Taheri, Nima Ghorbani, Michael J Black, and Dimitrios Tzionas. Grab: A dataset of whole-body human grasping of objects. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16 , pages 581-600. Springer, 2020.

Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR 2011 , pages 1521-1528. IEEE, 2011.

Meng-Fen Tsai, Rosalie H Wang, and Jo´ se Zariffa. Recognizing hand use and hand role at home after stroke from egocentric video. arXiv preprint arXiv:2207.08920 , 2022.

Graham JG Upton. Fisher's exact test. Journal of the Royal Statistical Society: Series A (Statistics in Society) , 155(3):395-402, 1992.

Chen Wang, Linxi Fan, Jiankai Sun, Ruohan Zhang, Li Fei-Fei, Danfei Xu, Yuke Zhu, and Anima Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play. arXiv preprint arXiv:2302.12422 , 2023.

Takuma Yagi, Md Tasnimul Hasan, and Yoichi Sato. Hand-object contact prediction via motion-based pseudo-labeling and guided progressive label correction. arXiv preprint arXiv:2110.10174 , 2021.

Takuma Yagi, Takumi Nishiyasu, Kunimasa Kawasaki, Moe Matsuki, and Yoichi Sato. Go-finder: a registration-free wearable system for assisting users in finding lost objects via hand-held object discovery. In 26th International Conference on Intelligent User Interfaces , pages 139-149, 2021.

Kaiyu Yang, Jacqueline Yau, Li Fei-Fei, Jia Deng, and Olga Russakovsky. A study of face obfuscation in ImageNet. In International Conference on Machine Learning (ICML) , 2022.

Lixin Yang, Kailin Li, Xinyu Zhan, Fei Wu, Anran Xu, Liu Liu, and Cewu Lu. Oakink: A largescale knowledge repository for understanding hand-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20953-20962, 2022.

Lingzhi Zhang, Shenghao Zhou, Simon Stent, and Jianbo Shi. Fine-grained egocentric hand-object segmentation: Dataset, model, and applications. In European Conference on Computer Vision , pages 127-145. Springer, 2022.

Yang Zheng, Yanchao Yang, Kaichun Mo, Jiaman Li, Tao Yu, Yebin Liu, Karen Liu, and Leonidas J Guibas. Gimo: Gaze-informed human motion prediction in context. arXiv preprint arXiv:2204.09443 , 2022.

Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. Freihand: A dataset for markerless capture of hand pose and shape from single rgb images. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 813-822, 2019.

Kateryna Zorina, David Kovar, Florent Lamiraux, Nicolas Mansard, Justin Carpentier, Josef Sivic, and Vladimir Petrik. Multi-contact task and motion planning guided by video demonstration. In ICRA 2023-International Conference on Robotics and Automation , 2023.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper builds a large-scale hand-object interaction dataset named Hands23 that contains 257K images, 401K hands, 288K objects, and 19K second 33 objects and combines different image sources (Visor, Epic-kitchen, COCO, Internet articulation and novel videos). Compared to the previous hand-object interaction datasets, it includes fine-grained contacts, detailed hand-grasp labels, second objects, and object segmentation. The resulting dataset covers a large amount of hand-object interactions in both ego and exo views. Based on the dataset, they show that a baseline model (Mask-RCNN) can perform well on Hands23 and also generalizes better to other datasets (than previous work). Thus, Hands23 provides a useful resource for the community to study hand-object interactions with applications from computer vision to robotics.\n\nSTRENGTHS:\n1) The proposed dataset is large and covers a wide distribution of hand-object interaction scales in both first and third-person views. \n\n2) Data were richly annotated: hand bounding boxes and segmentation (standard), contact (follows [43], in-contact object (following [53], secondary object (novel), hand grasps (following Cutkosky taxonomy). Thus, the authors innovatively combined to provide a more comprehensive annotation than previous work at scale. Thus, the proposed dataset contains more fine-grained annotations, e.g., second object, detailed contacts and hand-grasp labels.\n\n3). Ethics, privacy and demographics were particularly well addressed. Faces were obfuscated for privacy and Demographics and Realism of the data was analyzed. \n\n4). The proposed baseline model can predict all the rich interaction information and shows good generalization ability to Ego4D dataset.\n\n5) This is a comprehensive paper with excellent Supp. Materials.\n\nWEAKNESSES:\n1) One of the main advantages of the proposed dataset is the “second object” defined as an object in contact via a handheld tool. However, the motivation for introducing the second object and the corresponding possible applications are not as well discussed. \n\n2) This paper shows the zero-shot generalization to Ego4D dataset. However, Ego4D only contains the first-person view, which is less challenging due to rel. fixed hand scale and few (hand) occlusions. Therefore, it would be good if the authors could also show the zero-shot generalization to third-person data.\n\n3) While I do think the paper makes significant advances (thus my score), it is not higher, as one would hope for even richer annotations. \n\n4) One more minor comment, the authors state that “We only annotate non-crowd instances with at least 10^3 pixels of area.” While that makes sense it would also be good to annotate crowded areas (just like it is done for pose estimation data in COCO).\n\nRegarding relations to prior work I have two minor suggestions:\n- This paper gives fine-grained contacts annotation ({touch, hold} and {tool, container, neither}). However, some previous works in human object interaction (e.g., Discovering Human Interactions with Large-Vocabulary Objects via Query and Multi-Scale Detection) provide even more detailed object and contact classes (e.g., unlocking door). Therefore, the advantage of the current annotation type could be further discussed.\n- Line 29: “we extend this to a richer vocabulary that distinguishes touching, holding and using, and includes grasps” Here, GRAB [56] should also be mentioned as it does something similar. Perhaps one should also state that EPIC kitchen has a much broader vocabulary?\n\nQUESTIONS:\nOverall the paper is clear,  but I do have a few suggestions/questions: \n-\tObject labels in the figures are not too self-evident (e.g., ‘ObjC, NP-Fin’ in Figure 1, etc.)\n-\tThe caption for Table 2 does not mention the metric (the text does, but would be good to add to caption) \n-\tIn the caption of Table 2 you state: “no other dataset produces good results on Hands23.” Yet, 100DOH gives a reasonable 78.5. Likely the only difference is, thus, scale? What if you train on a fraction of Hands23 that has as many images as 100DOH? \n- Last but not least, I would find it really useful to get an assessment of the accuracy of the human labelers.\n\nOverall, this is a strong paper, and indeed I was wondering why they did not include Ego4D in the dataset. This comment clarified it: “We were unable to annotate Ego4D due to strict restrictions on data redistribution, but can use self-training to integrate Ego4D and our data” That’s great, but could already be mentioned earlier and I think Ego4D and a few other datasets (e.g. Grab, Discovering Human Interactions with Large-Vocabulary Objects via Query and Multi-Scale Detection) should be in Table 1. \n\nIn the Supp Mat it is stated that the data format for the release is not clear yet. I think it would be ideal to follow, e.g., COCO or Epic kitchen for consistency in the literature.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper introduces a hand object interaction-related dataset, Hands23, that providing rich labels for hand images (segemantion and boundingbox of left hand, right hand, object and second object, types of contact, grasp and touch). It labels the existing three datasets (COCO, EPIC-KITCHENS VISOR, and Internet Articulation), as well as self-collected videos. Based on the new dataset with rich labels, the paper uses RCNN to train a multi-task model, which performs and generalizes well.

STRENGTHS:
- [Size and rich labels] The presented dataset is much larger than existing datasets and provide segemantion and boundingbox labels of hands and objects.

- [Experiments] The model trained on Hands23 for hand or object detection shows good generalization performance. It outperforms the models trainend on other datasets.

- [Annotation] The annotation procedure is elaborated in great detail in the supplementary.

WEAKNESSES:
- [Multi-task] The paper highlights the importance of rich labels for hands and introduces a multi-task framework (Fig.3). It would be interesting to provide more insights on the interaction of different sub-tasks. For example, do the auxiliary labels affect hand segmentation or bounding boxes? what is the benefits of richer labels?

QUESTIONS:
- [Action] As Hands23 consists of a large number of videos, what do you think about action labels?

- [SAM] The segmentation labels are generated with the help of Segment Anything (Line.142). I am wondering if the paper could provide some observations or insights when working with Segment Anything or even other large models to generate labels.

LIMITATIONS:
Yeah. The authors have addressed the limitations in the conclusion section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a method and dataset for hand-object understanding. Specifically, the dataset provides bounding boxes and segmentation annotations for (1) hands, (2) objects that are in contact with hands, and (3) second objects touched by tools. The annotation also includes contact and grasp type for second objects touched by tools. Overall, there are annotations for ~260K images spanning four different datasets. The segmentations masks are automatically extracted using the off-the-shelf method, Segment Anything.\n\nSTRENGTHS:\n- The proposed dataset has images from both first-person and third-person views. This can help develop hand interaction detection methods that generalize well on different domains. \n- The dataset has images from other popular benchmarks like EPIC-KITCHENS and COCO. This could be beneficial to the community to develop and train unified methods for human understanding.\n- The authors demonstrate the benefits of data through cross-dataset experiments. \n- The proposed dataset annotates richer contact vocabulary, such as distinguishing between touching vs holding.\n\nWEAKNESSES:\nThe proposed architecture does not convince me fully. There are a few major concerns. \n\n- Currently, the method first detects three 'object' classes: hand, object, and second object. These classes are detected independently. However, there is a clear correlation between these three classes. For instance, the object's location in contact with the hand depends on the location of the hand. Also, the location of the second object depends on the location of both the hand and the first object. However, in the current method, this is not reflected; the three classes are detected independently.\n- In L217-L218, every hand is matched to an object with the highest interaction score. This excludes the possibility that a hand can contact more than one object. For example, a hand can hold a knife and the chopping board at the same time. The current model design does not reflect this.\n- When detecting object and second-object interaction, does the model also consider hand-object interaction? For example, consider object A (first object) and object B (second object). When inferring the interaction score between the two objects, will the model consider if object A was in contact with the hand? If object A was not in contact with the hand, then object B cannot be the second interaction object. Is there any such explicit modeling?\n\nQUESTIONS:\nPlease see the weakness section.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a framework that outputs rich interaction information for hands that are in an interactive state on a daily basis, and a large dataset that supports the model. The framework is based on the standard RCNN object detection mechanism and has a simple structure that can be easily extended to meet subsequent information additions for interactive hand understanding.\n\nSTRENGTHS:\n- A large dataset of human hand interactions is proposed, which provides rich and high-quality data annotation that facilitates the understanding and prediction of human hand behavior.\n- A framework for manual interaction understanding, with a simple structure that is easy to extend and conducive to directly supplement more interactive information.\n- Additional categories of how human hands interact with objects are classified and understood.\n\nWEAKNESSES:\n- The understanding of human-object interaction is still in the form of classification mapping, which is a large and tedious classification and lacks simplicity.\n- The framework is based on the object detection approach, which may be less applicable for large scenes or scenes that are messy.\n- It could be ragarded as an incremental supplement for the work of Shan et al, not innovative enough.（ Dandan Shan, Jiaqi Geng, Michelle Shu, and David F Fouhey. Understanding human hands in contact at internet scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 482 pages 9869–9878, 2020.)\n\nQUESTIONS:\nThe narrative of the article is clear and there is little doubt.\n\nLIMITATIONS:\nA large number of 3D hand-object interaction state estimation methods already exist in the community, which provide a more fine-grained analysis of contact. The prominent problem with current 2D-level interactions exists in the robustness to small and multiple targets, but these are subsumed into the Limitation of this work.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}