SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Data Pre-Processing. We processed recorded camera data to obtain time-synchronized color and depth images discussed in Table 2 over all views, ensuring that dropped frames are accounted for if any. We backprojected the depth pixels into 3D using the depth image intensity values. We use the transformations between the depth cameras to the reference camera 0 to fuse backprojected pixels into a single point cloud per frame number oriented in camera 0. We used depth-to-color transforms
(f)
Figure 4: Dataset Analysis: (a) histograms of object properties, (b) t-SNE for object models, (c) Overall (dark) and per-pair (light) mean trajectory plots with start points marked using spheres, (d) t-SNE of trajectories for HOH compared to Carfi et al. [13], (e) histogram of grasp types per Cini et al. [18],(f) histogram of grasp ratings, and (g) log histogram of handedness counts at transfer.
to color-map the fused point clouds. We ran OpenPose [12] and SAM over all Kinect color images to acquire upper body skeletons and complete image segmentation masks.
GT Annotation and Verification. As shown in Figure 3, we manually annotated three frames—G, T, and R—marking key events. G contains first giver contact marking the grab portion of reach and grab, T contains simultaneous giver and receiver grasp marking the middle region of object transfer, and R contains last receiver contact on the object marking the end of handover. We marked SAM masks for three entities—giver hand, object, and receiver hand—in G and T for giver, G, T, and R for object, and T and R for receiver. Masks were marked when present and multiple masks per entity are fused into a single mask. We verified and corrected all SAM masks. We checked all OpenPose skeletons and found that only  $4.6\%$  had either missing or inaccurate joints. We store OpenPose confidence values. We labeled handedness and grasp type using the 28-class grasp taxonomy of Cini et al. [18].
Post-Processing. We used Track Anything [53] to track annotator-segmented SAM masks from frames G to R. We backprojected tracked masks via the camera calibration to generate giver, object, and receiver point clouds. We denoised full, giver, object, and receiver point clouds to remove outliers, and use median filtering based on the top  $k$  nearest neighbors per point to smooth the colorization, with  $k$  being 10. We save both pre- and post-cleaned point clouds.
6DOF Object Pose. We provide GT object 6DOF pose by aligning the object 3D model to all frames from G to R for all handover interactions. First, we conduct a frame-to-frame ICP [7] alignment of the object point cloud to estimate inter-frame cloud-to-cloud transformations of the object point cloud for all frames, linking all frames from G to R. Next, we conduct an automated alignment of the 3D model to the object point cloud in the G frame. We sample a diverse range of orientations to ensure exhaustive  $SO(3)$  coverage. We transform each 3D model using each orientation and fine-tune the transformed model's alignment to the object point cloud using ICP. We choose the 3D model alignment with the smallest ICP distance. We use the inter-frame transformations to align the 3D model to all frames from G to R. Example frames from the alignment are shown in Figure 3.
Objects. Figure 4 provides a summary of multi-modal analysis of our dataset. Object histograms in Figure 4(a) show that while our objects are skewed toward lower bounding box volume (BBV), mesh volume (MV), and mass (medians of  $9.73 \times 10^{6} \mathrm{~mm}^{3}$ ,  $2.54 \times 10^{6} \mathrm{~mm}^{3}$ , and  $167 \mathrm{~g}$ , and skewness of 2.83, 2.77, and 3.47 respectively), the bounding box diagonal is normally distributed with mean of  $233.58 \mathrm{~mm} \pm 88.50 \mathrm{~mm}$ , and 17 objects are over  $0.5 \mathrm{~kg}$ . The bounding box occupancy (MV-to-BBV
ratio) is slightly skewed toward lower values (median 0.37, skewness 0.27) indicating that HOH has high count of non-full objects, e.g., the spray bottle in Figure 2. The histogram of convexity, computed as MV-to-convex-hull ratio [2], shows though our dataset tends toward convex objects (median 0.67, skewness -0.51), 42 objects have a convexity  $< 0.5$ , i.e., embody concavities. The spread of coefficients t-SNE plot of object geometry, in Figure 4(b), computed using 8,000 3D point samples per 3D object model surface, confirms the object diversity in our dataset.
Trajectories. Figure 4(c) shows overall (dark) and per-pair (light) mean trajectories for giver hand, object, and receiver hand. Means are obtained by resampling trajectories to have 100 samples, and averaging within left-seated and right-seated givers. Trajectory means show transport and object transfer phases for giver hand, inclination of object toward receiver indicative of giver intention to participate in collaborative handover, giver retraction after transfer, and looped arc for receiver toward end of handover. Left- and right-seated givers mirror each other. Spread of per-pair means demonstrates trajectory diversity, echoing prior findings of individual dependence in handover trajectory [6]. HOH trajectory diversity is confirmed by the t-SNE plot in Figure 4(d) where the coefficients show a higher spread than the trajectories of Carfi et al. [13]. To ensure comparability across HOH and Carfi trajectories, we align the trajectories within each entity and dataset using Procrustes prior to t-SNE computation, since participants may be spatially displaced in the Carfi dataset. We find trajectory lengths follow a normal distribution with giver, object, and receiver means of  $3.26\mathrm{s}\pm 0.97\mathrm{s}$ ,  $3.60\mathrm{s}\pm 0.94\mathrm{s}$ , and  $3.49\mathrm{s}\pm 0.95\mathrm{s}$ . Object trajectories are longer since the receiver may react slightly later than the giver, while the giver may complete their motion before the receiver's end of handover.
Grasp. The distribution of grasp types in Figure 4(e) organized by the taxonomy in Cini et al. [18] shows a majority of power grasps especially near C1, or larger diameter grasps apt for HOH object sizes. High precision grasp counts are found for the giver, e.g., C6/C8 thumb-finger and C14 tripod grasp, common for rod-like or handled-equipped items. For the receiver, we see high counts for C6, F18 extension-type grasp on thin flat items, and F29 or stick grasp likely during receipt of objects with protrusion affordances. The supplementary details taxonomy nomenclature. The comfort rating histogram in Figure 4(f) indicates skew toward higher comfort levels (giver comfort skewness of -1.23 with  $16.21\%$  over 4, receiver comfort skewness of -1.21 with  $15.7\%$  over 4). We observe a high overall giver-receiver Pearson correlation of 0.38. As shown by the log histogram for handedness in Figure 4(g), for unimanual grasps the giver and receiver use the same hand in  $76.8\%$  of the interactions, and opposite hands in  $20.4\%$ . 76 interactions have bimanual grasp.
4 Experimental Results Showing Use Case
The primary use of our dataset is to drive AI research in HRI. Unfortunately, no off-the-shelf algorithms exist to directly evaluate multi-person data such as HOH for robotic understanding of handover parameters. The only approach for learning-driven grasp lacks public code and data [54] and requires access to articulated hand pose, making it unusable with HOH. Full-fledged HOH-data-driven robotic control is outside the scope of this paper (and in fact consists of components spanning a wide range of future work). In this section, we create and evaluate deep neural networks to show the benefit of HOH data for four tasks of relevance to the robotic manipulation pipeline: (1) Use object point cloud to predict human giver grasp point cloud or o2gg, (2) Use object point cloud to predict object orientation at transfer point or o2or, (3) Use object and giver point cloud to predict human receiver grasp point cloud or g2rg, and (4) use human giver hand motion to predict receiver motion trajectory or g2rt. o2gg and g2rg enable robotic givers to bias grasp near preferred human giver grasp, and robot receivers to bias grasp away from / close to preferred human giver / receiver grasp. o2or enables robot manipulators to present objects at transfer in human-preferred poses. g2rt enables robot receiver motion planning in response to human giver motion.
Implementation. We adapt PoinTr for o2gg and g2rg and Informer for g2rt. We use PointNet [47] with 2 dense layers to generate quaternion orientation for o2or. We obtain input point clouds for o2gg and o2or from Gpre, and for g2rg from Tpre, where Gpre and Tpre are a few frames prior to G and T, containing object only and object+giver hand respectively. Outputs for o2gg, g2rg, and o2or consist of giver hand point cloud at G, receiver hand point cloud at T, and rotation from 6DOF pose at T represented as a quaternion. We train two networks each for o2gg, o2or, and g2rg, one that uses complete input point clouds enabling handover parameter analysis with  $360^{\circ}$  access to geometry through, e.g., multi-view fusion, and one that uses partial input point clouds, emulating
Table 3: Metrics for experimental results.
Complete	Partial
Task	Metric	GT	Best to Object	GT	Best to Object
o2gg	CD	0.483±0.187	0.135±0.078	0.135±0.052	0.277±0.188
o2or	MEAE	0.851±0.253	0.365±0.153	0.843±0.227	0.411±0.136
g2rg	CD	0.539±0.648	0.147±0.182	0.206±0.121	0.288±0.275
g2rt	MAE	0.160±0.070	0.107±0.037	N/A	N/A
Task	Metric	Complete	Partial
o2gg	%OLGO	17.39%±10.37%	58.12%±19.46%
g2rg	%OLRG	1.44%±4.73%	1.13%±3.87%
Figure 5: Visualization of outputs from o2gg, g2rg, and o2or. Insets for o2or show input point cloud. Camera angle shown for partial clouds.
single-viewpoint RGB-D sensors. We generate partial data by rendering each scene from 6 randomly generated viewpoints per scene. We pre-register all GT input and output point clouds for the complete networks to the Gpre frame, in order to assess pre-movement parameter estimation. Partial point clouds are left in the rendered viewpoint and object orientation at the appropriate handover timepoint to emulate real time behavior. Since the proposed g2rt focuses on point trajectories rather than objects, we use the trajectory centroids from Section 3.3 to train a single version of g2rt. We manually pre-transform GT to set the origin at the table center,  $xz$ -plane as ground, and right-seated givers aligned with left-seated.
Training. We randomly divide the 8 objects in each form/function bin into sets A, B, and C with 3, 3, and 2 objects per bin or 51, 51, and 34 total objects to be used in train only, test only, and train+test respectively. The training set uses random  $75\%$  of data from sets A and C. The test set uses the remaining  $25\%$  of data from set C, and  $100\%$  from set B. For g2rt, given the potential for time-varying person movement to be a behavior signature, we keep train and test participant pairs mutually exclusive. We use data from 11 pairs (22 with RR) in the training set, and the remaining 9 (18 with RR) in the test set. Hyperparameter choices and computing details are included in the supplementary.
Evaluation Metrics. We provide GT evaluation metrics, particularly Chamfer Distance (CD) for o2gg and g2rg, mean Euler angle error (MEAE) for o2or, and mean absolute error (MAE) for g2rt. As plausible parameters may not correspond to GT, we report best GT metrics to all test object instances (accounting for symmetry), to increase the chance of finding similar parameters. We report percentage overlap of giver hand with object (%OLGO) for o2gg and receiver hand with giver hand (%OLRG) for g2rg to gauge affordances for robot givers and receivers for safe handover.
Results. Table 3 shows the results of the evaluation metrics. Figures 5 show qualitative results of predictions for o2gg, g2rg, o2or, and g2rt, the latter two contextualized within the handover providing the source input. Additional results are provided in the supplementary. When predictions diverge from GT, they correspond to plausible outputs, e.g., though the actual giver grasp for the grill brush at the bottom left of Figure 5 for o2gg is on the handle, the predicted grasp is on the brush. Variability receiver hand orientation is observed for g2rg. For o2or, we notice that the alignment of the predicted object when deviating substantially from GT, corresponds to a plausible extension direction during transfer. Outputs of g2rt show that, even without object structure, simple trajectory prediction enables the receiver to meet the giver trajectory near the shared space. From Table 3, we
Figure 6: Visualization of outputs from g2rt for various pairs and objects.
notice that the mean best-to-object metrics show a drop compared to the GT, indicating that a more likely candidate for each parameter is found in the dataset, and demonstrating the dataset's diversity in representation of handover. %OLGO in Table 3, indicates affordance availability, especially using complete data. %OLRG demonstrates that giver-receiver overlap is negligible, showing ability of g2rg to predict away from giver hand and enabling its use for robot receiver grasp biasing.
Future Work. The experimental results presented in Section 4 provide a starting point for algorithm development in HRI using HOH. The diverse, multimodal, and richly annotated data in HOH greatly opens the scope for large-scale AI algorithms in HRI. Algorithms can be developed to leverage comfort ratings to rank grasp, pose, and path prediction outputs, conduct realtime object-dependent robotic giver motion prediction and proactive generation of robotic receipt motion by giver motion forecasting, and enhance the robotics pipeline with object segmentation and tracking algorithms in the presence of multi-person occlusions due to handover. Transfer position estimation can be conducted by using upper body point clouds and/or skeletons to establish spatial relationships. The analysis in Section 3.3 provides an overall view of HOH. A large scope exists for future studies involving detailed analyses of alignment between giver and receiver comfort, trajectory velocity and timing, and coordination, with respect to grasp type, object categories, and participant pairs.
Limitations. HOH lacks grip force due to the use of instrumentation-free setup. Currently, grip force is collected using heavily-designed batons [40, 16, 19, 30] that constrain grasp type, are difficult to control for weight, and lack geometric diversity. The scope remains to instrument everyday objects with reliable minimally-invasive grip force units. Our participants occupy a narrow range over age and ability. We recognize the challenges in recruiting participants such as children, older adults, and individuals with different abilities, health and social concerns toward mutual interactions with unfamiliar partners, and potential cognitive barriers toward providing informed consent. Future collections can benefit from best practices on population-specific recruitment [41, 11, 3]. Though HOH has 6DOF object pose, it currently lacks GT hand pose annotation. Full manual annotation of hand pose in markerless data in the presence of occlusions and motion is a daunting task. Future work includes adapting HOnnotate [25] to operate with multi-person hand interactions.
Societal Impacts. HOH provides the societal benefit of informing social robots on how to perform safe handover, important toward the development and enhancement of trust and collaborative goal accomplishment [24, 45]. Social robots aware of object-dependent handover improve fluidity of post-handover operations involving multi-object use such as assembly and activities of daily living. Improved trustworthy social robots, engaging in safe handover of objects have the potential to fill current shortages of in-home aides for older adults [1] and individuals with disabilities, expected to be of special concern in the light of a declining caregiving workforce [43] and increased demand [10]. Joint attention on shared objects, a component of handover, has been shown to encourage social closeness [52]. Long-term impact of human-robot bonding on older adult isolation or childhood development remains understudied. Social robots developed without coupling short-range focused data such as ours with studies on longer-range attachment behavior are likely to introduce negative societal consequences. If a robot on which an individual has developed a dependence malfunctions or has to be discontinued, it may introduce anxiety and regression of collaborative goals within the individual, akin to the loss of a loved one. We share full details of our setup, capture, computing, and code in supplementary to promote continued wider-scale data collections for societally-aware HRI.
This work was funded by National Science Foundation grant IIS-2023998. We thank Mingjun Li and Nikolas Lamb for insightful discussions on algorithm development and sensor calibration. We also thank Priyo Ranjan Kundu Prosun, Thomas Dubay, Ben Molloy, Irfan Yaqoob, Numan Zafar, Jyothinadh Minnekanti, Sichao Li, Houchao Gan, Xinchao Song, Kun Han, Alaina Tulskie, Holly Rossmann, Rosalina Delwiche, Cameron Hood, Odin Kohler, Alexander Cohen, Christian Soucy, and Gianna Voce for assistance with data collection and data annotation.