HOI4ABOT: Human-Object Interaction Anticipation for Human Intention Reading Collaborative roBOTs
Authors: Esteve Valls Mascaro
Daniel Sliwowski
Dongheui Lee
Venue: CoRL 2023 Poster (2023)
Keywords: Human-Object Interaction
Human-Robot Collaboration
Human Intention

Abstract: Robots are becoming increasingly integrated into our lives, assisting us in various tasks. To ensure effective collaboration between humans and robots, it is essential that they understand our intentions and anticipate our actions. In this paper, we propose a Human-Object Interaction (HOI) anticipation framework for collaborative robots. We propose an efficient and robust transformer-based model to detect and anticipate HOIs from videos.  This enhanced anticipation empowers robots to proactively assist humans, resulting in more efficient and intuitive collaborations. Our model outperforms state-of-the-art results in HOI detection and anticipation in VidHOI dataset with an increase of 1.76% and 1.04% in mAP respectively while being 15.4 times faster. We showcase the effectiveness of our approach through experimental results in a real robot, demonstrating that the robot's ability to anticipate HOIs is key for better Human-Robot Interaction.

HO14ABOT: Human-Object Interaction Anticipation for Human Intention Reading Collaborative roBOTs
Esteve Valls Mascaro', Daniel Sliwowski', Dongheui Leel.2
Technische Universität Wien (TU Wien), Autonomous Systems Lab Institute of Robotics and Mechatronics (DLR), German Aerospace Center
festeve. valls mascaro, daniel. sliwowski, dongheui. .eee @ttwietwwnnn.. ac. at evm? Himm.github.o/HOI4AABB page
Next chair
Next to laptop
Next cup
Holding Cup
Detection
Anticipation
Robot starts grabbing bottle
Robot starts pouring
T-1
Figure I: Overview our HO14ABOT framework. robot leverages RGB data to detect and anticipate the human-object interactions in its surroundings and assist the human in timely manner. The robot anticipates the human intention of holding the cup. so it prepares itself for pouring by grabbing the bottle. The robot reacts the human holding the cup by pouring water.
Abstract: Robots are becoming increasingly integrated into our lives, assisting us in various tasks. To ensure effective collaboration between humans and robots, it is essential that they understand intentions and anticipate actions. In this paper, propose Human- Interaction (HOI) anticipation framework for collaborative robots. We propose an efficient and robust transformer-based model to detect and anticipate HOIs from videos. This enhanced anticipation empowers robots to proactively assist humans, resulting in more efficient and intuitive col- laborations. Our model outperforms state-of-th oftte-of-theaare art results in HOI detection and anticipation VidHOI dataset with an increase of 1.76% and 1.04% mAP re- spectively while being 15.4 times faster. We showcase the effectivenees of our ap- proach through experimental results in real robot, demonstrating that the robot's ability to anticipate HOIs is key for better Human-Robot Interaction.
Keywords: Hamman-Object Interaction, Collaborative Robots, Human Intention
Introduction
In recent years, the field of robotics has witnessed significant interest human-robot interaction (HRI), with focus enhancing the ability of robots to assist humans in various tasks [1, 2, 3, 4]. To facilitate effective human-robot collaboration (HRC), it is crucial for the robot to possess an
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

understanding of both the surrounding environment and the individuals within it, including their intentions. For example, consider the scenario visualized in Fig. I where a robot assists a person in the kitchen. By recognizing the person's intention to prepare a drink and understanding their actions such as reaching for the cup, the robot can proactively provide the necessary support in a timely manner, such as picking up bottle and pouring water. Therefore, by recognizing and anticipating human-object interactions (HOIs), the robot gets solid understanding of the person's 's intention and better caters to their needs [1].
While HOI is a long-standing challenge in the computer vision community, most approaches only consider the detection of these interactions from single frames [3, 6, 7, 8, 9, 10]. However, to minimize the latency when person is assisted by a robot, the detection is not enough, but the anticipation needed [11, 12, 13]. Therefore, consider the task of HOI detection and anticipa- tion, and we propose to leverage temporal cues from videos to better understand human intention. HOI recognition in videos has been explored recently [14, 15, 16, 17]. In this paper, propose a real-time deep learning architecture that combines pre-trained models with spatio-temporal con- sistency to successfully detect and anticipate HOIs. Our model outperforms the state-of-the-an in VidHOI dataset [14] in terms accuracy and speed. Moreover, we ensemble our framework with behavior trees [18] to adapt real-time the robot actions for better interaction with the human. We implement our framework in real robot and demonstrate the effectiveness of our approach in the pouring task, showcasing the robot's ability anticipate HOIs and proactively assist the human while reducing latency in the execution.
The contributions of our paper are summarized next:
A real-time transformer-bee based model for HOI detection and anticipation.
A novel patch merging strategy align image features to pre-extracted bounding boxes.
To the best of our knowledge, we are the first to assess HOI anticipation in real robot experiment for collaborative task.
2 Related Works
2.1 Human Intention in Robotics
Recognizing and predicting human intention crucial to ensure seamless human-robot collabora- tion (HRC) [12, 13, 19, 20J. [12] observed significant differences in the robot's contribution and commitment in an experiment of a human carrying car parts shared workspace with an antic- ipatory robot to assemble them. Recent works in computer vision have highlighted the potential of harnessing human intention better anticipate future human actions [21, 22, 23]. In particu- lar, [23] leverages the detection of human-object interactions (HOIs) within a scene to understand this high-level intention of the individuals. Despite the benefits of using HOIs, their application in robotics from vision data has not been extensively explored [1]. [4] proposes a conditional random field (CRF) to assess the feasibility a robot executing a given task based on the anticipated human actions. The CRF predicts the next human actions by considering object affordances and positions in the future. However, [4] is not scalable to tasks the CRF relies hand-crafted fea- tures. Instead, train our model in the largest HOI video dataset available to learn robust features that enhance the robot's ability to anticipate human intention. Recently, [24] proposed spatial- attention network to extract scene graphs from images in an industrial scenario. However, [24] neglects the time dependency the task and does not anticipate the human intention to enhance HRC. [25, 26, 27] also adopted scene graphs but focused on task planning.
2.2 HOI Detection and Anticipation
HOI focuses on localizing the humans and objects in scene and classifying their interactions using (human, interaction, object) triplet (e.g. (personl, hold, cup)). HOI task has recently gained attention in the computer vision community due to its promising applications in downstream tasks,

---

such as scene understanding [28] or action recognition [29]. The primary focus is the detection of HOI from images |5, 6, 7, 8, 9, 10]. Some [7, 8, 9] adopt a one-stage approach, directly operating on the images to predict the HOI triplet. However, these methods require higher training resources and do not benefit from pre-trained object detections. On the contrary, [5, 6, 10] employ a two- stage method to first locate the objects and humans in the image using pre-trained models and then classify each interaction using multi-stream classifiers. In particular, [10] uses VIT transformer [30] to extract the patched features and proposes Masking with Overlapped Area (MOA) extract features per object or human through a self-attention layer. Our work shows that weighting the patched features is sufficient to outperform MOA while not requiring any additional parameters.
While processing individual frames may be adequate for HOI detection, we argue that HOI antici- pation benefits from leveraging the temporal aspects inherent in these interactions. Several studies in HOI detection address this temporal dimension by focusing videos [14, 15, 16, 17]. [16] fuses patched features at multiple levels to generate instance representations utilizing a deformable tokenizer. [14] employs a two-stage model that uses 3D convolutions to merge features across the temporal dimension. [15] also adopts a two-stage approach but relies on a spatio-temporal trans- former [31] to detect the interactions in videos. Finally, [17] extends the architecture from [15] by concatenating the human and object temporal features and fusing them with the human gaze information using cross-attention. [17] is the first work to propose both HOI detection and antici- pation in videos. Similarly to [10], [17] also adopts focal loss [32] to tackle the HOI imbalance in training. We adopt the findings from [17] but observe their model to not be feasible to work in real-time. Moreover, [17] trains unique model for each anticipation horizon in the future. Instead, we propose a novel real-time multi-head model that can detect and anticipate HOIs in a single step.
2.3 Task and Motion Planning
For robot to effectively assist and collaborate with a human in a particular task, it needs to under- stand the structure and order of actions involved, enabling the robot achieve desired goals [33]. Finite State Machines (FSM) have been the standard choice for representing the task structure for long time [34, 35]. However, scaling FSM poses challenge due to their lack of modularity and flex- ibility [18]. Recently, Behavior Trees (BT) [18] have gained popularity as they can facilitate task planning in HRC tasks [36, 37], where the environment is dynamic. Our work adopts BT and defines its behavior based on the anticipated human intention and its uncertainty. Once a suitable chain of actions has been found by the task planner, motion planning responsible for determining the low- level movements of the robot. Motion planning is a core problem in robotics [38, 39, 40, 41, 42]. [38, 39] proposed to randomly sample points in the state space towards the goal. However, they consider humans as obstacles or constraints, not collaborators. Some approaches [40, 41] formulate motion planning mining the cost function related to humans is not trivial. Alternatively, motion generators can be learned from human demonstrations to obtain more natural movement [42, 43]. Dynamic Move- ment Primitives (DMPs) [42] have been successfully employed HRC, by dynamically adapting their parameters [44, 45, 46).
optimization problem, but their applications in HRC are limited as deter-
3 Methodology
In this section, present our Human-Object Interaction Anticipation for CollAborative roBOTs (HOI4ABOT) framework. First, we formulate the HOI detection and anticipation task. Then, we describe the integration of the deep learning architecture into the robot framework.
3.1 Human-Object Interaction
Let V .f_..... fo] be a frame sequence of duration T 1. The goal is to predict the interaction class i in the subsequent time between any human HH, and object Om pair P. {H,, 0m) observed during the video V, where S N,O I < M, 0 S ks < K M + N A visual illustration of our HOI4ABOT architecture is depicted in Fig. 2.
3

---

T+1
9
DC
"cup"
cls
...
Box Embedder
6
b.
B
cis
fm
E
K.
Object
c
Q
6
B
Pogl2D
Human Blander
e
E
K.
Backbone
b27b22
FC cv
W
FC,..
Weighled Sum
AvgPool
FC,..
cit
Figure 2: HO14ABOT architecture overview. We consider a video of + frames with the pre- extracted object and human bounding boxes B Our module initially extracts relevant features per frame (left) to later on detect and anticipate HOIs (right) later. First, a VIT backbone [47] extracts patch-based local E and global cls, features per each frame I. Then, obtain features per human e,, and object e/n by aligning E' to their bounding boxes, as shown in light blue. We also project each B' to B using a box embedder [48], and the object category Sm using CLIP [49]. Our Dual Transformer, shown purple, leverages the human and object-constructed windows (sequences in red and blue respectively) through two cross-aties atos-aattntiinn transformers, where Key, Query, and Value are used in the attention mechanism. q is learnable parameter to learn the evolution of the location in time. Finally, we project the enhanced last feature from the Human Blender to detect and antici- pate HOIs several time horizons i the future through our Hydra head (shown in light green).
Detection and tracking. HOI4ABOT is two-stage method. First, we leverage off-the-sher shelf state- of af-the-art art object detection and tracking methods to identify the bounding boxes B,n R(T+1)x4 label Cm. and track identifier id,, for any object On {idm Cm Bm in the video V. Bm represents list of XY pixel coordinates of the top-left corner and right-bottom
[b=" .... bbd] corner of the bounding box that locates a given object Om at each frame f, of V. We obtain the same information for each human H., In the second stage, we exploit each individual pair Px = (TH,,, 0m) ) to predict its interaction class o in a given time horizon using various data modalities. This requires understanding the visual features of the pair, how their spatial relationship evolves through time B₃ [B,,, B..] and also the intrinsic semantics of the object Cm.-
Visual features. We use Dinov2 [47] as a pre-trained Visual Transformer (ViT) [30] backbone to divide each frame f, into
patches and project each patch pI to a visual token oj that encodes the image information of that patch In total, the image encoder obtains E' I xd that captures the local visual features, plus the global context vector cls, IR of frame fz.
We develop a simple but efficient technique, called Patch Merger, to extract individual features per human and object from frame through single step. Let Of be an object m with its box by at frame f.. First, we create binary mask for f,, where 1 denotes pixel laying within by,, We convert the binary mask in sequence of patches following [30]. Then, obtain a weighting vector w!, by computing the percentage that b%,, overlaps each patch using 2D Average Pooling and normal- ization. Finally, we compute the weighted sum of local visual features ct,, = Σ w E', obtaining the individual representation of Of m Compared to [10], which normalizes along the patch dimen- sion and uses a quantized sequence as the attention mask for self-attention layer, our algorithm is parameter-free, more efficient, and shows better performance in our experiments.
We propose capture the context within frame using cls, E Rd, contrary to the spatial transformer proposed in [17]. We claim that this context (e.g. a kitchen, an office) should be invariant in short time periods and be the dominant component among all cls, tokens. Consequently, we use Average Pooling reduce the N cls; features single representation cls ArgPPoo([cls__.. clso))), which is the context of the scene.

---

Spatial features. For each bounding box b we extract the XY normalized pixel coordinates for the top-left corner and right-bottom corner. Then, we adopt a positional encoding using random spa- tial frequencies [48] to embed the location of each point and merge these two corner representations into one box representation by., E Idd using a fully connected layer. This process also applied to humans, thus obtaining b E I to encode each human H position in the scene.
Object semantics. Leveraging the object semantics is essential to understanding the possible inter- actions in given pair. While 'holding cup' or "holding bottle' are both feasible, 'holding a car' becomes more unrealistic. Thus, we extract object semantic information Sy E E per object On to facilitate the model predicts the intention class i.. For that, we use the CLIP text encoder [49].
Pair Interaction. We construct a temporal architecture that leverages the evolution of the in- teractions between human HI, and an object On in time. We process each pair independently, and therefore we focus on single pair in the formulation. We stack both the visual tokens E, le,, , .... eel and the spatial features B, [b,, ..... 6 in time and construct a human temporal window WH.,
[B,, En]. Similarly, we also construct an object's temporal window Wom = [B... Em]. We add a sinusoidal positional encoding to WH, and Wom: Later, we prepend the global visual feature and a learnable spatial parameter qq. cls] to WH q learns the evolution of the location of the human in time through the attention mechanism. We also extend Wom by prepending the semantic token Sm that encodes the object type. Therefore, we obtain a temporal feature WH..
H E (T+))xx and Wom E (TT+)xx per pair.
To extract the HOI relationships between H., and Om. train dual transformer with cross- attention layers. First, an Object Blender transformer enhances the object window Wo.. based on the human knowledge W WHn- Then, the blended object features Wom are used extend the human representation W WH, in the Human Blender transformer to WHn Finally, we extract the last token from WH.: which encodes the most current status of the scene, and classify the interaction pair 2 using a fully connected layer. As a given human-object pair can have multiple interactions simultaneously, we use sigmoid funetion and define a threshold classify the current interactions.
Multi-head classification for multiple future horizons. The goal is to predict the interaction class i% in the subsequent time between any human Hr, and object O... pair IP = (H,, 0m We considered the problem of HOI detection (T = 0) and also the anticipation in multiple future horizons (T > 0). Contrary to [17] that proposes one trained model for each T. we developed a single model that can predict multiple time horizon interactions. For that, we froze the HO14ABOT trained in the detection task, and train an additional linear layer that projects the last token from WH,, to the interaction for the particular r. We call this shared backbone the Hydra variant, which allows us to sinultaneously predict interactions across multiple r, making our model faster and more efficient. We consider our Hydra variant with A number of heads.
3.2 Motion generation and task planning
Motion Generation. The proposed framework segments the complex movements into simpler movement primitives, which are learned with DMPs. To collect demonstrations of each move- ment primitive, we employ kinesthetic teaching, where an operator guides the robot's end effector by physically manipulating it [50]. Generating the motion requires estimating the goal position, which we obtain through the use of calibrated vision system that relies on pre-trained object detector (i.e. YOLOv8 (51]) and depth camera. The position of the goals with respect to the robot base is computed using the intrinsic and extrinsic camera matrices.
Task planning. Properly scheduling the acquired movement primitives is crucial to reach a desired goal. We implement Behavior Trees (BT) [18] as a ROS node that subscribes to the predicted HOIs and their confidence. The reactiveness of BT's allows adapting the robot's behavior by considering the anticipated human intention and changing to the appropriate sub-tree if needed. This is motivated by how humans interact with each other. For example, if bartender observes client approaching the bar, they can prepare for the interaction by grabbing glass, thus reducing the serving time.
5

---

Table Detection mAP.
Table 2: Anticipation mAP in Oracle mode.
Method
Full
Non-Rare
Rare
Preson-wiss wise top-5
Method
mAP
Oracle Mode
Rec
Prec
Acc
F1
ST-HOI
17.6
27.2
17.3
29.09
74.76
41.36
36,61
50.48
QPIC [54]
21.4
32.9
20.56
STTran [15]
3
27.59
74.79
40.86
36,42
50.16
TUTOR [16]
26.92
37.12
23.49
5
27.32
75.65
41.18
36.92
50.66
STTran [15]
28.32
42.08
17.74
ST-Gaze (17)
38.61
52.44
27.99
37.59
72.17
59.98
51.65
62.78
ST-Gaze [17]
33.14
71.88
60.44
52.08
62.87
Ours (Dual)
40.37
54.52
29.5
32.75
71.25
59.09
51.14
61.92
Ours (Stacked)
40.55
53.94
30.26
38.46
73.32
63.78
55.37
63.59
Detection Mode
Ours
(Dual, Seratch)
34.58
73.61
61.7
64.48
STTrum [15]
7.61
13.18
3.33
33.79
72.33
63.96
55.28
65.21
ST-Gaze [17]
10.4
16.83
5.46
37.77
74.07
64.9
56.38
66.53
Ours
Ours (Dual)
11.12
18.48
(Dual, Hydra)
34.75
74.37
64.52
66.4
Ours (Stacked)
10.79
17.79
34.07
73.67
65.1
66.4
Robot control. The generated poses from the motion generator are passed to the controller. In our system, we employ a Cartesian impedance controller [52, 53] to achieve the compliant behavior of the manipulator. This controller enhances the safety human-robot collaboration by allowing the robot to respond compliant manner external forces and disturbances.
4 Experiments
4.1 Dataset and Metrics
We train and evaluate our model on the VidHOI dataset [14], the largest dataset available for human- object interactions videos. This dataset encompasses 7.3 million frames with 755,000 annotated interactions of one frame per second. To assess the performance our approach, we adopted the same evaluation metrics those presented in [17]. We computed the mean average precision (mAP) using the method presented in [54]. The mAP@ 50 incorporates the precision-recall curves for all interaction classes. To determine correct HOI triplet, three conditions need to be met: (i) the detected bounding boxes for human and object must overlap with their corresponding ground truths with an Intersection over Union (IoU) of 50 %, (ii) the predicted object category is correct, (iii) the predicted interaction correct. Following standard evaluation VidHOI, we report mAP across three different HOI sets: (i) Full: all interaction categories. (ii) Non-Rare: frequent interactions in the validation set (more than 25 appearances), (iii) Rare: non-frequent interactions (less than evaluated
25). Additionally,
the human and object detections from ground truth, and in Detection mode, where those are predicted using YOLOv5 [55] as in [17]. Finally, we computed the Person-wise top-k metrics [17] where the anticipation was considered correct if one the top-k predicted interactions matched the ground truth.
approach in Oracle mode, where
4.2 Quantitative evaluation
HOI4ABOT outperforms state-of -hr models [14, 54, 16, 15, 17] in terms of accuracy and speed across all different tasks and scenarios, as shown in Table I and Table 2. Moreover, Table 2 shows how our Hydra variant outperforms all models the anticipation task, even training from seratch a separate model each anticipation horizon. We consider that the detections provide a great deal of information regarding what human doing now, and what they might be interested in doing next. By using the Hydra variant ground the anticipation what is happening at the present time.
4.3 Ablation study
This section analyses our proposed approaches and their impact on the performance of the HOI task. All results depicted in Table For simplification, we only consider the HOI detection task.

---

Firstly we explore different variations in the extrac- tion and arrangement of features to compose the hu- man and object windows. We compare our Patch Merger strategy to the MOA strategy from [10]. Us- ing MOA requires an additional self-attention block, which increases the model's 's parameters while under- performing, Moreover, we explore different feature aggregation strategies to classify an interaction. In- stead of using the last observed token in WH for classification, we prepend an additional learnable to- ken to WH. which aggregates the interaction relationships, inspired by the VIT class token [30]. However, Table 3 shows that classifying from the last observed features is better while not requiring additional parameters. Last, we consider varying the order of the cross-attention branches, first the Human Blender and second the Object Blender. We claim that the decrease performance due to the different behavior between humans and objects: objects are static and therefore less informative than humans, which are dynamic and lead the interaction.
Table 3: Albation study in HOI detection.
Variant
mAP
Feature blender MOA Interaction token = Learnable
40
40.29
Main branch Object
39.85
Transformer type Single
40.26
Transformer type Stacked Dual
40.55
40.37
Secondly, we assess our dual transformer by comparing it with other variants. We consider the Single variant when only using the Human Blender transformer, which is not able to effectively capture the HOls. We also consider stacking both WH. E R((+2)xd and Wom E R( to a single feature window pair, WPk E (T-2)xxx We observe slight improvements this variant in terms of mAP when detecting in the Oracle mode, but it underperforms the Detection mode and for the anticipation tasks, as shown in Appendix E.
Finally, we compare the inference time of our model to [17] to assess the efficiency in real-world applications in robots. Our Dual variant is 15.4 times faster than [17] for the detection task. [17] requires extracting gaze maps, which drastically slows down the inference speed of their model. When using our Hydra model, we obtain interactions for the time horizons 0, 1, 3, and using one forward pass, with nearly the same inference speed and parameters using head. More information can be found in Appendix D.
4.4 Real World Experiments
HOI detection and anticipation essential for robots to comprehend the surrounding humans and better predict their needs, so the robot can assist in timely manner. We conduct real experiments with a Franka Emika Panda robot to showcase the benefit of our approach in collaborative robots beyond the offline VidHOI dataset. The VidHOI dataset contains user-collected videos of humans, mostly performing outdoor activities that can not be easily related to robotic collaboration tasks. We consider the 'pouring task' in a kitchen scenario where the robot assumes the role of bartender with the goal of pouring beverage for the human. The scenario shown in Fig. 1. To assess the performance of model in unseen scenarios, we collected 20 videos of 5 people in our kitchen lab. The human is instructed to grab the cup and informed that the robot will assist them in the task. We manually annotate the time the person grabs the cup to use as ground truth. Our Hydra variant detects and anticipates the HOI between person and cup in real-time. When the robot anticipates that the human will be near the cup, it proceeds to grab the bottle. However, the human moves away the robot releases the bottle and returns to the initial pose. The robot proceeds to pour the liquid into the cup after detecting that the human is holding it.
We assess our real-world experiments by considering well-established metrics in HRC [13]. [13] proposes to evaluate human-robot fluency in the joint task by considering four objective metrics. Human Idle Time (H-IDLE) and Robot Idle Time (R-IDLE) are proposed to evaluate the percentage of the total task time that the respective agent is not active, which reflects the team coordination and the inefficiency of the agent in the task. Concurrent Activity (C-ACT) measures the percentage of total task time in which both agents are active concurrently (the action overlap between different members). A higher C-ACT indicates better secter-synchrroiized team. Functional Delay (F-DEL) mea- sures the delay experienced by the agents immediately after completing activity: the percentage
7

---

Human Idle Time
Robot Idle Time
Concurent Activity
Functional Delay
80
Percentage [96]
30
25
70
25
10
20
60
20
8
15
50
-20
15
40
0
3
0
1
3
1
3
5
0
3
5
Ta
Ta
Ta
Ta
Figure 3: Mean objective fluency metrics for pouring experiments for different confidence thresholds
{0.3, 0.5, 0.7} in the HOIs prediction.
of total task time between the completion of one agent's action and the beginning of the other agent's action. A negative F-DEL indicates that actions are overlapping and implies an efficient use of team members' time. Figure 3 summarizes the average objective fluency metrics across pouring periments. The results indicate that HOI anticipation allows for better human-robot coordination and efficiency of each other's 's time, thus making the task more fluent. We observe substantial im- provement in Figure 3 when using anticipation (Ta > 0) compared to detection (T, = 0). Additional quantitative and qualitative results are provided in Appendix B.
Limitations
Despite outperforming state-of-the-art models HOI from videos, we observe from qualitative experiments the challenge of the implementation the real world. First, there is a domain gap between the VidHOI dataset, mainly representing humans daily scenes and our robotic scenario. For instance, anticipating that 'a human is holding cup' is challenging, despite being correctly detected. We explore the VidHOI dataset and observe that most people already appear with the cup in their hand. To overcome this issue, we sample more frequently clips where the interaction changes in the anticipation horizon. Still, this is insufficient to ensure correct anticipation 'holding a cup' with higher confidence. Other datasets are not better suited for our problem as they mainly are image-based [5, 56] or do not track the humans and objects in videos [57]. Future research directions consider training with a dataset more coupled our robotics scenario to improve the model predictions. This would allow extend our experiments more complex daily scenarios. Second, in our real experiments, visible so that object detection can recognize them. Finally, the employed DMPs could be expanded or replaced by visual servoing to consider goal-following behaviors.
assume that the objects present in the scene are sufficiently
6 Conclusions
In this paper, we proposed a Human-Object Interaction Anticipation for CollAborative roBOTs framework (HOI4ABOT). We consider the task of detecting and anticipating numan-object inter- actions (HOI) in videos through a transformer architecture. We train and evaluate HOI4ABOT in the VidHOI dataset and outperform current state-of-the-an across all tasks and metrics while being 15.4x faster. Moreover, our model runs in real-time thanks our efficient design. Additionally, we extend our HO14ABOT model with multi-head architecture, which can detect and anticipate HOIs across different future horizons in a single step. We demonstrate the effectiveness of our approach by implementing our model in a Franka Emika Panda robot. We show that anticipating HOIs in real- time is essential for a robot to better assist a human a timely manner and we support our findings with real experiments. In conclusion, our approach demonstrates its effectiveness and defines a new road to explore, where intention reading plays a crucial role for robots in collaboration scenarios.

---

Acknowledgments
This work is funded by Marie Sklodowska-Curie Action Horizon 2020 (Grant agreement No. 955778) for the project 'Personalized Robotics as Service Oriented Applications' (PERSEO).
References
[1] N. Robinson, B. Tidd, Camphell, D. Kulić, and P. Corke. Robotic vision for human-robot interaction and collaboration: survey and systematic review. Hum.-Robot Interact., 12(1),
feb 2023, doi: 145/3377/3377 URL httph:tt///// org/ 10 1145/3570731.
[2] B. Abbasi, N. Monaikul, Z. Rysbek, B. D. Eugenio, and M. Zefran. A multimodal human- robot interaction manager for assistive robots. 2019 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6766- 2019, doi: 1091100119/1100008897. 2019.8968505.
[3] V. Ortenzi, A. Cosgun, T. Pardi, W. P. Chan, E. Croft, and D. Kulić. Object handovers: A review for robotics. IEEE Transactions on Robotics, 37(6):1855-1873, 2021. doi:10.1109/
TRO.2021.305336.
[4] H. S. Koppula and A. Saxena. Anticipating human activities using object affordances for reactive robotic response. IEEE Transactions Pattern Analysis and Machine Intelligence, 38(1):14-29, 2016. doi: 10001000//AAM..015555522000
[5] B. Xu, Y. Wong, J. Li, Q. Zhao, and M. S. Kankanhalli. Learning to detect human-object interactions with knowledge. In 2019 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2019-2028, 2019, 10::1000109//C/PP22000.9
[6] O. Ulutan, A. S. M. Iftekhar, and B. Manjunath. Vsgnet: Spatial attention network for detecting human object interactions using graph convolutions. pages 13614-13623, 06 2020. doi: 10. LU90/CVPR42600.2200011..
[7] J. Lim, V. M. Baskaran, J. M.-Y. Lim, K. Wong, J. See, and M. Tistarelli. Erntt: An effi- cient and reliable human-object interaction detection network. IEEE Transactions Image Processing, 32:964 979, 2023. doi:10.1 00.1199/11122022333312222
[8] X. Wu, Y.-.. Li, X. Liu, Zhang, Y. Wu, and C. Lu. Mining cross cross-person cues for body-part interactiveness learning in hoi detection. In Avidan, G. Brostow, M. Cissé, G. M. Farinella, and T. Hassner, editors, Computer Vision ECCV 2022, pages 121-136, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031-19772-7
[9] Y. Liao, A. Zhang, M. Lu, Y. Wang, X. Li, and S. Liu. Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20091-20100, Los Alamitos, CA, USA, jun 2022. IEEE Computer Society. 109 100.1000////P2288....0200 https://doi. httt::://oad.eeeccppputteeeeeee www.eccomppeessociitty.y...ee http://////...10////////66665// 011.:010000001222288222200...191 01942
URL
[10] J. Park, J.-W. Park, and J.-S. Lee. Viplo: Vision transformer based pose-conditioned self-loop graph for human-object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17152-17162. 2023.
[11] I.. Psarakis, D. Nathanael, and N. Marmaras. Fostering short-term human anticipatory behavior in human-robot collaboration. International Journal of Industrial Ergonomics, 87:103241, 2022. ISSN 0169-8141. dai.tt/.//bbbooggg//./.oiiieee.p www. www. www.eiiieee.rret...mm http://ccttcctcoooocc///////rrte http:///....eooeotooo///////////
URL https: //
[12] G. Hoffman and C. Breazeal. Cost-based anticipatory action selection for human- robot flu- ency. Robotics, IEEE Transactions 23:952 961, 2007.

---

[13] C. A. Garcia, W. Montalvo-Lopez, and M. V. Garcia. Human-robot collaboration based on cyher-physical production system and mqtt. Procedia Manufacturing. 42:315-321, 2020. ISSN 2351-9789. doithttss//do..or//0...0000 http://do.....oooooooooooooooaaa sciencedirect http://wwwc.ccc......aaaaaaaaaaa
URL https:/ //www.
International Con-
ference on Industry 4.0 and Smart Manufacturing (ISM 2019).
[14] M.-J. Chiou, C.-Y. Liao, L.-W. Wang, R. Zimmermann, and J. Feng. St-hoi: A spatial- temporal baseline for human-object interaction detection in videos. In Proceedings of the 2021 Workshop on Intelligent Cross-Data Analysis and Retrieval, ICDAR '21, page 9-17, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450385299. 101:10011445/34439444443336
URL wtw.hhtp////ete org/10 10. 1145//46394444.... 34690977.
[15] Y. Cong, W. Liao, H. Ackermann, B. Rosenhahn, and M. Yang. Spatial-temporal trans- former for dynamic scene graph generation. In 2021 IEEE/CVF International Confer- ence on Computer Vision (ICCV), pages 16352-16362, Los Alamitos, CA, USA, oct 2021. IEEE Computer Society. 10i:10. 109//CC44992222221.0000 URL https://doi. iesecomputersocieevy wwwp://ciiiittto.ooooooooooooo 110./ 11199ICCC449922
[16] D. Tu, W. Sun, xiongkuo min, G. Zhai, and W. Shen. Video-based human-object interaction detection from tubelet tokens. In A. H. Oh, A. Agarwal. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openrevier. net/forum? wid.orunmmiiddddi...... LsENM.
[17] Z. Ni, E. Valls Mascaró, H. Ahn, and D. Lee. Human-object interaction prediction in videos through gaze following. Computer Vision and Image Understanding, page 103741, 2023. ISSN 1077-3142. //ww.acieeccdirect. cot http:///ccarrre//rree/a////////0
http:p/s//oo..rr///000.o..///... 103744. URL https:
[18] M. Colledanchise and Ogren. Behavior Trees in Robotics and Al: An Introduction. 07 2018. ISBN 9781 17781138597322 10:110000200119984444000090 1201/97804294881055
[19] Z. Li, Y. Mu, Z. Sun, S. Song, J. Su, and J. Zhang. Intention understanding in human-robot interaction based on visual-nlp semantics. Frontiers in Neurorobotics, 14:610139, 2021.
[20] Semeraro, A. Griffiths, and A. Cangelosi. Human-robot collaboration and machine learning:
A systematic review of recent research. Robotics and Computeer-nneerrateed Manufacturing, 79: 79:102432, 2023, ISSN 0736-5845. oot.tt/s///daoo///////o/o/////i0 URL attps://wwes http:///wwww.scincciierreee wwweenceiieec..c........... httpapaii....occccaacc...o.o///a
[21] E. Mascaro, H. Ahn, and D. Lee. Antentioncconditin long-term human egocentric action anticipation. In Proceedings of the IEEE/CV Winter Conference on Applications of Computer Vision (WACV), pages 6048-6057, January 2023.
[22] D. Roy and B. Fernando. Action anticipation using latent goal learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2745-2753, January 2022.
[23] Wei, Y. Liu, T. Shu, N. Zheng, and S.-C. Zhu. Where and why are they looking? jointly inferring human attention and intentions complex tasks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6801-6809, 2018. 109110.109/CVPP.2011 00711.
[24] S. Li, P. Zheng, Z. Wang, J. Fan, and L. Wang. Dynamic scene graph for mutual-cognition generation in proactive numant collaboration. Procedia CIRP, 107:943- 948, 2022. ISSN 2212-8271. boichtps://doioorg/1100 http://0..001...p.rrrr.200000000 sciencedirect.. wwteedirrttcccccccceieeeeoo http://ceeeeaartea////////////// turing systems transformation Proceedings of the 55th CIRP Conference Manufacturing
URL https://www.
Leading manufac-
Systems 2022.
10

---

[25] S. Amiri, K. Chandan, and S. Zhang. Reasoning with scene graphs for robot planning under partial observability. IEEE Robotics and Automation Letters, 7(2):5560-5567, 2022.
[26] C. Agia, K. Jatavallabhula, M. Khodeir, O. Miksik, V. Vineet, M. Mukadam, L. Paull, and
F. Shkurti. Taskography: Evaluating robot task planning over large 3d scene graphs. In Con- ference on Robot Learning, pages 46-58. PMLR, 2022.
[27] M. Patel and S. Chernova. Proactive robot assistance via spatio-temporal object modeling. In 6th Annual Conference on Robot Learning, 2022.
[28] Y. Xu, J. Zhang, Q. Zhang, and D. Tao. ViTPose: Simple vision transformer baselines for human pose estimation. In Advances in Neural Information Processing Systems, 2022.
[29] Q. You, H. Jin, Z. Wang, C. Fang, and J. Luo. Image captioning with semantic attention. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4651- 4659, 2016.
[30] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x 16 words: Transformers image recognition at scale. International Conference on Learn- ing Representations (ICLR), 2021.
[31] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems (NeurIPS), 30, 2017.
[32] Y. Cui, M. Jia, T.-Y. Lin, Y. Song, and S. Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9268-99699 0008-997,,200 2019.
[33] H. Guo, F. Wu, Y. Qin, Li, K. Li, and K. Li. Recent trends in task and motion planning for robotics: A survey. ACM Comput. Surv., feb 2023. ISSN 0360-0300. doi: 10. (0.1144/3388333 URL https:////doo org/10 1145/3583136. Just Accepted.
[34] M. Awais and D). Henrich. Online intention learning for human-robot interaction by scene observation. In 2012 IEEE Workshop on Advanced Robotics and its Social Impacts (ARSO), pages 13-18, 2012. doi: 20110///RSO2.222.221323..
[35] I. El Makrini, M. Omidi, F. Fusaro, E. Lamon, A. Ajoudani, and B. Vandcrborght. A hi- erarchical finite-state machine-based task allocation framework for human-robot collaborative assembly tasks. In 2022 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1
2022. doi: 10. 10011000/544771222222999999
[36] C. Paxton,, A. Hundt, F. Jonathan, K. Guerin, and G. D. Hager. Costar: Instructing collaborative robots with behavior trees and vision. 20/7 IEEE international conference on robotics and automation (ICRA), pages 564-571. IEEE, 2017.
[37] M. Carvalho, J. Avelino, A. Bernardino, R. Ventura, and P. Moreno. Human-robot greeting: tracking human greeting mental states and acting accordingly. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1935-1941, 2021, doi:10.1 109/ IROS51168 202
[38] L. Kavraki, P. Svestka, J.-C. Latombe, and M. Overmars. Probabilistic roadmaps for path planning in high-dimensional configuration spaces. IEEE Transactions on Robotics and Au- tomation, 1264):566-580, 1996. doi:10.1 100:10.1119/700 109770004340
[39] Kuffner and Valaee Rrt-connect: An efficient approach to single-query path planning. In Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH33065),, volume 2, pages 995-1000
vol.2, 2000. 0001100..00///B00.00000.44..

---

[40] M. Kalakrishnan, S. Chitta, E. Theodorou, P. Pastor, and S. Schaal. Stomp: Stochastic trajec- tory optimization for motion planning. In 201/ IEEE International Conference on Robotics and Automation, pages
2011. doi:10.1 109/ICRA.200 201.599020., 1.5980280.
[41] C. Park, J. Pan, and D. Manocha. Itomp: Incremental trajectory optimization for real-time replanning in dynamic environments. 06 2012.
[42] P. Pastor, M. Kalakrishnan, F. Meier, F. Stulp. J. Buchli, E. Theodorou, and S. Schaal. From dy- namic movement primitives to associative skill memories. Robotics and Autonomous Systems, 61:351-361, 04 2013. doi:10..016f.rrbbo2222..299
[43] S. Calinon and D. Lee. Learning Control, pages 1261-1312. 01 2019. ISBN 978-94-007- 6045-5, 100100077--94-007770
[44] M. Wu, B. Taetz, Y. He, G. Bleser, and S. Liu. An adaptive learning and control framework based on dynamic movement primitives with application to human-robot handovers. Robotics and Autonomous Systems, 148:10 2022. ISSN 0921-8890. doi.htpss//6oi.or/10..00 j.robo.200111033933. URL http::::www..ccceeeeeeeeeeeeeeee htt.:/acccccccccoo//eeeeeeera//a $092188902100218)
[45] M. Prada, A. Remazeilles, A. Koene, and S. Endo. Dynamic movement primitives for human- robot interaction: Comparison with human behavioral observation. In 2013 IEEE/RSJ In- ternational Conference on Intelligent Robots and Systems, pages 1168-1175, 2013. doi: 10.
[46] M. Prada, A. Remazeilles, A. Koene, and S. Endo. Implementation and experimental validation of dynamic movement primitives for object handover. In 2014 IEEE/RSJ In- ternational Conference on Intelligent Robots and Systems, pages 2146-2153, 2014. doi:
10. 10.1109/ROOS 2014.69228....
[47] M. Oquab, T. Darcet, Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, Fernandez, D. Haz- iza, F. Massa, A. El-Nouby, al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07/999 2023.
[48] M. Tancik, P. Srinivasan, B. Mildenhall, S. Fridovich-Keil Keil, N. Raghavan, U. Singhal, R. Ra- mamoorthi, J. Barron, and R. Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537-
7547, 2020.
[49] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 9748-8763. PMLR, 2021.
[50] M. Ito, K. Noda, Y. Hoshino, and J. Tani. Dynamic and interactive generation of ob- ject handling behaviors by small humanoid robot using dynamic neural network model. Neural Networks, 19(3):323-333 2006. ISSN 0893-6080. doithttps://doi.orr//11. 1016//. neumet.2006.02.007. URL https:/ //::://wwww htwp//wwww...o////c http://wwiwnceeteee..o sciencedirect................... wetpcc/creec.coo//ccieeeeeeeeeee com///cieenc//artiiccle////
$0893608006000311. The Brain Mechanisms of Imitation Learning.
[51] Jocher, A. Chaurasia, and Qiu. YOLO by Ultralytics, Jan. 2023. https:t::://it
com/ulltralytics///ee comm//uutrrlllyyieeslllaaaaaaaat
[52] N. Hogan. Impedance control: An approach to manipulation. In 1984 American Control Conference, pages 304-313, 1984. doi: doi:100.991/ACCC.1994. 1984444899999444883333.
[53] N. Hogan. Impedance Control: An Approach Manipulation: Part II II-Implementation. Jour-
nal of Dynamic Systems, Measurement, and Control, 107(1):8-16, 03 1985. ISSN 0022-0434. URL https: / //toi//iii//// wtt..111111...111.1..
12

---

[54] M. Tamura, H. Ohashi, and T. Yoshinaga. QPIC: Query-based pairwise human-objec interac- tion detection with image-wide contextual information. In CVPR, 2021.
[55] G. Jocher, A. Chaurasia, A. Stoken, J. Borovec, NanoCode012, Y. Kwon, K. Michael, TaoXie, J. Fang, imyhxy, Lorna, Z. Yifu, C. Wong, A. V, D. Montes, Z. Wang, C. Fati, J. Nadar, Laughing, UnglvKitDe, V. Sonck, tkianai, yxNONG, P. Skalski, A. Hogan, D. Nair, M. Strobel, and M. Jain. ultraltiics/yolov:: v7.0 - YOLOv5 SOTA Realtime Instance Segmentation, Nov. 2022. URL http:p::/doo.o.r///////// 10. 5281/22eodd 7347926.
[56] S. Gupta and J. Malik. Visual semantic role labeling. arXiv preprint
2015.
[57] Ji, R. Krishna, L. Fei-Fei, and C. Niebles. Action genome: Actions as compositions of spatio-temporal scene graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10236-10247, 2020.
[58] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. International Conference on Learning Representations (ICLR), (6-9):2019, 2019.
13

---

Appendix
A Implementation Details
In this section, we offer a comprehensive summary of the implementation details to aid in the repro- duction of the experiments and the replication of the results. All experiments were conducted using single NVIDIA RTX A4000 graphics card with 16GB of memory and an Intel 17-12000K CPU.
Hyperparameters. All trained models are conducted using the same strategy [17]. We use the official code from http:://gghuuu..oomiiiiifi/ http:/ii///ooiidiiiiiiioooooo
and implement our HOI4ABOT model into their framework. All training settings are summarized in Table 4. We adopt Cross Binary Focal Loss [32] with 7 = 0.5 and 3 = 0.9999, which improves training extremely imbalanced datasets, such as VidHOI [14]. We train our models using the Adam W optimizer [58]. We define scheduler for the learning rate, with an initial value of 10 10-8 that increases to a peak value of x 10 in 3 warm-up epochs. The learning rate then decreases with an exponential decay with factor 0.1. We run the training for 40 epochs.
Model configuration. All trained models use a similar configuration, but some variants such as Stacked or Single are adapted to ensure having similar number of trainable parameters in the architecture (57.04M). All models reported in our paper use the DINOv2 [47] as the image fea- ture extractor, using the smallest variant available ViT-B/14 that only contains 22.06M parameters; and CLIP [49] for the semantic extractor, with the largest available variant ViT-L/14 that contains 85.05M parameters. However, due to the fact that the number of objects in the dataset is limited, we pre-extracted the features for all possible objects. For our baseline HOI4ABOT model, we consider two transformer models with cross attention layers, each of them with depth 4 and MLP expansions of ratio 4.0. Each transformer uses the multi-head attention variant with 8 heads to better extract relationships within sequence of features. Moreover, we consider sinusoidal positional embedding facilitate learning the temporal information of sequence. Finally, we consider the embedding size of each extracted feature, bounding box, or image feature, as 384. The embedding size for the prepended class token is also 384, this is the embedding dimensions of the features extracted using DINOv2. For the semantics, CLIP obtains a feature of dimensionality 764.
Table Training settings.
Table 5: Model settings.
Optimizer Weight Decay
AdamW
Transformer Depth
4 8
1.0c-2
Number of Hcads
Scheduler
ExponentialDecay
Feature Extractor
DINOv2: VIT-B/14 [47]
Warmup Epochs
3
Semantic Extractor
CLIP: ViT-L/14 [49]
Initial LR
1e-8
Embedding Dimension Positional Embedding
384
Peak LR
le-4
Sinusoidal
Exponential Decay
0.1
Exponential Decay
0.1
Epochs
40
Mainbranch
humans 4.0
Random Seed
1551
MLP ratio
Augmentation
Horizontal Flip
Flip Ratio
0.5
Batch Size
16
Dropout
0.1
B Experimental Scenario
Task description. Our HOI4ABOT framework enhances human intention reading through HOI anticipation. We conduct real-world experiment with a Franka Emika Panda robot to support our proposed approach. Fig. provides a step-by-step overview of the considered bartender scenario. First, the robot detects human in the scene and anticipates the human intention approach kitchen island. When the robot anticipates with confidence that the human will be close to the cup,
14

---

The rabot antitiaaes the human the buttle presaration.
Ante poting Himrotion
The human changes their intention
Makopstig op elerection
goes
bpille:
bottle
The assitance needed langer sotot moves back to
robot will be cisce the cup soon, and preparac
Inticesting eieraction. Greating botte...
walts for the human grao the cep.
Gradling bottle..
is holding the cup,
Gretbing belle... moddin delected point
proceecs pour
Spttler
bptter
poling
Gretbing oup iteraction Preceed pount
Gnabbing bottle. Proceed to pout
ngitie:
Figure 4: Real-world experiments scenario.
executes a movement to grab the bottle, thus preparing for pouring. the intention of the human changes, the robot adapts behavior and moves back the initial position after placing the bottle on the table. on the other hand, if the human proceeds grab the cup, the robot pours the drink and goes back to its initial position. This preparatory behavior reduces the serving time while improving the overall experience for the human.
Additional Qualitative Results On the project website (https: //eps/////mmmm/o/ htt///emm..tt.bbbbb 1o/HOI4ABOT.
page/) we present additional qualitative results that showcase the ability of our model to operate more ambiguous scenarios (with multiple objects and people instances, and cluttered scenes) and execute different motions depending on the predicted interactions. Our model predicts the interac- tion associated with specific human and object instances, which associated with the identifiers obtained from the tracker. Therefore, we are able to execute different movements depending on the
15

---

Bottle
Successful
Reachable zone
Cup
Non-Reachable Zone
Confidence
0.3
Serving time [s]
0.7
4
2
0
3
Figure 5: Human waiting time to be served the drink for different confidence thresholds ((0.3, 0.5, 0.7) and anticipation heads Ta 3,
Figure 6: Quantitative evaluation of the pouring task. We overlay on the image of the workspace of the robot the position of the bottle and the cup. Green signifies successful task execution and red failed cases.
interaction (like pouring, pushing, turning off the lights) and the object category (grabbing from the side in case of cup bottle grabbing from the top in case of bowl) instance (cup-1 and cup-2). For instance, we consider the case of multiple cups in the scene, where the robot conditions its pouring behavior based on the cup the human holds. Additionally, we also show the ability to operate in an ambiguous situation with multiple objects and people instances.
Evaluation of the use case scenario of HO14ABOT. To validate our hypothesis, extend our evaluation of the framework in real-world experiments with additional quantitative metrics. First, we assess the human waiting time until the robot proceeds to serve. Fig. 5 shows the quantitative benefit of our approach by considering the absolute time human waits to be served (serving is considered until the robot starts pouring). The results indicate that our robot behaves proactively when anticipating HOIs and therefore reduces the time to wait until drink is poured, compared to the reactive behavior observed if the robot only detecting HOIs. Fig. shows slight reduction in the waiting time when reducing the confidence threshold in the prediction: be more confident in the human intention the robot waits more. In addition, waiting time for different anticipation horizons (Ta 3, 5}). This subtle variation might be caused because of the dataset limitation pointed out in the main manuscript. Secondly, we measure the effectiveness of our robot pouring a drink in our real-world trials by considering the success rate of the pouring task in 20 new real-world experiments. Four lab members were instructed to approach the robot and grab the cup. Each person did 5 repetitions. Our framework correctly executes the pouring task 17 of 20 executions, resulting success rate of 85%.
observe only slight decrease in the
C Motion Generation and Task Planning
Motion Generation. Our framework decomposes the complex movements into simpler movement primitives, which are learned with DMPs. For instance, the pouring task consists of muitiple steps, like grabbing the bottle, moving to the cup, tilting the bottle, and placing the bottle back. Learning the entire movement as a single primitive is possible, but this might oversimplify the motion, partic- ularly for sharp movements, compromising accuracy. In our experiments, each motion segment was learned from a single demonstration. We verified the success rate the pouring scenario with 60 different arrangements of the objects 'Bottle' and 'Cup'. Figure in the attached document shows that robot successfully pours 53 out of 60 executions. resulting in success rate of 88.33%. We can observe that failure cases occur mainly when the objects are arranged close to the non-reachable
16

---

Repest
Root
Cathlags
Pound Check
Main Selector
Pour branch
Approach branch
Move branch
lfler
Oipnn
open
botte
Pourre
Grabbed
check
botte
Grabbed
Execute
check
bottl
bodBu
Giobbed
Figure 7: Schematic of the Behaviour Tree for our HO14ABOT framework.
areas for the robot. Working close the non-reachable zone is more problematic as the robot is operating near its kinematic constraints. However, this issue can be solved by rearranging the robot base position adequately to the user's need.
Task Planning: Behavior Tree. In this section, describe the structure of the Behavior Tree [18] used in real-world experiments, which is shown in Fig. The primary focus of this work is to enhance human-robot collaboration through human intention reading using HOI anticipation. We conduct simple real-world experiment with a Franka Emika Panda robot to showcase the benefits of our approach. This paper does not intend to provide general development of BT for HOI tasks. However, the same methodology employed can be extended more complex scenarios thanks to modularity BT.
The entire tree is built from three sub-trees: the Pour branch, the Approach branch, and the Move Away branch. First, the Pour branch is responsible for pouring the liquid into the cup. It is executed once the bottle is grabbed, and the 'hold' interaction between the human and the cup is detected. To achieve this conditional execution we add the Execute check behavior at the beginning of the branch. Then, we reset the Grabbed flag and set the Poured flag to prevent any potential duplication of pouring into the cup. Secondly, the goal of the Approach branch is to grab the bottle. This sub-tree is executed when the bottle is not currently grabbed and the robot anticipates the 'next to' interaction with confidence greater than pre-defined threshold. Once the bottle is grabbed, the Grabbed flag is set. Thirdly, the Move Away branch is responsible for releasing the bottle and moving it back to its initial position. This branch is executed when the bottle is grasped by the robot and the robot anticipates the interaction 'next to' with a confidence lower than a predefined threshold. After executing the movements the Grabbed flag is reset.
The appropriate sub-branch is selected by using the Main Selector composite node. This node attempts to execute each sub-tree starting from left to right. The selector node executes the next branch in the sequence when the check in the preceding branch is not satisfied. Finally, the last behavior in the sequence is an Idle behavior where the robot waits for short period of time.
The root of the tree is sequential node, which first collects all messages from the appropriate ROS topics, next checks if the beverage has been already poured, and finally executes the Main Selector. To achieve continuous operation, the Root node is decorated by Repeat modifier, which executes the root node indefinitely.
17

---

103
Inference time [ms]
ST-GAZZ [17]
Ours (Dual)
Ours (Stacked)
10a
20
60
80
100
120
140
160
Number of interactions
Figure 8: Model performance depends on the number of interactions for different architectures. Our variants ('Dual' and 'Stacked') have similar inference times (curves overlap) while outperforming by large margins the ST-GAZE model [17]
150
Dual Detection + Anticipation
Inference time [ms]
Dual Hydra
Dual Detection
100
50
0
20
40
60
80
100
120
140
160
Number of interactions
Figure 9: Model performance depends the number of interactions for different model variants. The proposed multi-head approach allows us to detect and anticipate HOIs at multiple time horizons while maintaining similar inference speed as the 'Dual' version (purple and dark orange curves overlap). We observe the benefit of the Hydra compared to rumning specific 'Dual' transformer per detection and per anticipation.
D Inference time
Our model is able in real-time thanks to the efficient design and reduced dimensionality.
Inference time versus the number of human-object pairs. Due to the nature of HOIs, each interaction needs to be computed for each human-object pair existing in the scene at a given time step. Therefore, to speed up the results and parallelize the forward pass for a given video, we stack all found human-object pairs the batch dimension. Still, we consider it necessary to observe how different models' inference speed affected by the number of pairs in a given video. Therefore, we 1000 executions of model processing a given video with interactions. We implement all models reported in Fig. and 9 in the same batch strategy and observe similar tendency in the increase of the inference time for higher number of interactions.
Efficiency comparison with current state-of-the-art [17]. Both HO14ABOT and [17] adopt a transformer-based architecture to comprehend the temporal relationships between the humans and objects in the scene. However, our model is designed to be efficient and to run in real-time despite having large number of interactions, contrary [17]. The comparison of the efficiency of both models is depicted in Fig. 8, which shows that our HOI4ABOT outperforms [17] by large margins in
18

---

Table Anticipation mAP in Oracle mode.
Preson-wise top-S
Method
mAP
Rec
Prec
Ace
F1
29.09
74.76
41.36
36.61
50.48
STTran [15]
3
27.59
74.79
40.86
36.42
50.16
5
27.32
75.65
41.18
36.92
50.66
1
37.59
72.17
59.98
51.65
62.78
ST-Gaze [17]
33.14
71.88
60.44
52.08
62.87
32.75
71.25
59.09
51.14
61.92
1
38,46
73.32
63.78
55.37
65.59
Ours (Dual, scratch)
3
34.58
73.61
61.7
54
64.48
5
33.79
72.33
63.96
55,28
65.21
37.77
74.07
64.9
56.38
66.53
Ours (Dual, Hydra)
3
34.75
74.37
64.52
56,22
66.4
5
34.07
73.67
65.1
56.31
66.4
36.14
70.03
64.61
53.99
64.34
Ours (Stacked, Scratch)
3
34.65
73.85
62.13
54.15
64.77
34.27
72.29
61.81
53.65
64.03
37.8
72.05
65.58
56.23
66.09
Ours (Stacked, Hydra)
3
34,9
72.96
65.05
56.3
66.2
5
35
72.86
65.18
56.36
66.2
terms of speed. Next, we list the major differences in the model design that cause our improvement. First, we do not use any additional modality to predict HOIs, compared to [17] that leverages pre- extracted gaze features to capture the human's attention. Predicting these gaze features costly as it requires detecting and tracking each human's 's head in the scene, predicting the corresponding gaze per human, and matching it the corresponding body. Thus the speed decreases considerably depending the number of humans in the scene. Moreover, [17] also considers an initial spatial transformer that leverages all humans and objects per frame, thus [17] speed is more affected by the number of frames considered.
Efficiency comparison of the Hydra HOI4ABOT, Human intention reading requires understanding both current and future HOIs. Therefore, develop multi-head HO14ABOT, called Hydra, that allows to predict HOIs different time horizons in the future through single forward step. While Table 6 shows the benefit of our //ydra variant compared training from scratch, in this subsection we focus on the benefit of efficiency. Fig. shows the inference time in milliseconds depending on the number of human-object pairs across different variants. We consider the Dual Detection as the baseline of our HO14ABOT model when only predicting the HOI in the present. Dual Detection Anticipation is an optimized model that uses two dual transformer blocks that benefit from same image backbone, one for HOI detection and the other for HOI anticipation in single future Dual Hydra performs HOI detection and anticipation for [0, 1, 3.5) 5] in single step by using our multi-head strategy. We observe the benefit of our Hydru variant compared to the model ensemble, as it has a comparable speed to the single head while anticipating HOIs in three additional future horizons.
3. Finally,
E Extensive comparison with variants
Our HO14ABOT model outperforms the current state-of-the-art across all tasks and metrics in the VidHOI dataset, as shown Tabel 6. In this section, we extend the comparison from the manuscript for the HOI anticipation for our Dual and Stacked variants, both when being trained by scratch or through the multi-heat Hydra mode. Our results show that the Stacked variant obtains slightly better performance in the mAP for longer futures. We consider this marginal improvement to be motivated because of the width difference in the transformer blocks, well the bigger representation space from which project when classifying the HOIs. The Stacked variant is based on single self-
19

---

attention block that operates on the human windows and object windows stacked in time. Therefore, the Stacked transformer has double the width compared to the Dual variant. Given that the output of a transformer model has the same shape as its input, the obtained tokens are also wider in the Stacked variant. Having a bigger embedding dimension the projected token allows the encoding of more information, which could result in better performance. However, Table shows that the Stacked variant has lower recall and therefore lower F1-Score. These findings might indicate that the Stacked variant struggles when anticipating HOIs in the videos where the interaction changes in the anticipation horizon, being more conservative in its predictions. Therefore, we consider the Dual variant to be optimal as it balances both precision and recall metrics across all tasks, as shown by outperforming all other models in the Fl-score for the Hydra version.
20


Reviews:
[{"review_id": "sBzV7DGDVpF", "reviewer": "Reviewer_pnXM", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a method for predicting human activity to allow a robot to anticipate and execute appropriate collaborative actions.  The system uses a vision-based machine learning model to predict the action, behavior trees to choose the robot action, and kinesthetic teaching combined with a motion planner to move the robot.  The proposed method achieves a significant speedup over prior approaches with a minor improvement in accuracy.", "summary_of_recommendation": "Overall, this is an interesting human activity recognition system but the robotics experiments seem incomplete.  Further validation to demonstrate that the proposed method works across tasks and users would make this a strong submission to a future venue.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "The authors may wish to highlight any evidence that supports the idea that this method would improve human-robot interaction in practice.  Please also provide more detail on the experimental design.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- This work tackles a difficult and important problem in robotics (anticipating human behavior)\n- The work includes a fully implemented system on a robot\n- The proposed methods for human activity recognition is likely to be of interest to roboticists\n\nWeaknesses:\n- The validation with the robot (\"real world experiments\") is not well-designed and is lacking detail (was the human in the study a member of the research team? was the study approved by a university ethics board?).  For a paper where the primary robotics innovation is incorporating the method in a system, it is important to have a strong validation to demonstrate that the method is effective and robust.\n- Overall, the robotics side of the work is somewhat weak; although the authors argue the method will improve human-robot interaction, the paper does not provide sufficient evidence to support that claim, or provide experimental evidence that the method is effective across different types of interaction.\n\n\nMinor points:\n- the authors may want to refer to their system as a \"collaborative\" rather than assistive robot -- typically \"assistive robotics\" are robots intended to be used by disabled people.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "V9CjhAUNAHo", "reviewer": "Reviewer_AxBS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The key idea of the paper is to model human-object interaction using temporal cues from video data using transformers. A major motivation for the development of the model is real-time inference of human-object interaction detection and prediction for robotic assistance.\n\nA major contribution of the paper is the development of a human-object interaction model that is able both detect and anticipate human object interaction better than existing models. The model is efficient and is able to provide real-time inference. Other contributions include testing on a real robot in a assistive setting by planning robot actions based on the predicted object interaction.", "summary_of_recommendation": "My recommendation is based on the following\n- While the modeling approach uses transformers similar to some of the existing methods, the way temporal data is abstracted and how the transformer model is designed is distinct enough to improve HOI prediction.\n- The model uses efficient techniques for real-time inference using temporal data suitable for real-world robotics applications.\n- The demonstration of the model and corresponding robot planning on a robot assistance task motivates the necessity for HOI prediction.\n- While the paper provides a lot of details on the model approach and the evaluation experiments, some important details are missing which are required to even understand the validity of some of the assumptions and for potential replication.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Responses to the following comments could help improve the paper.\n\n- The intuition behind the model development could be explained in more detail. Why did the authors think this method/model would work?  \n\n- Similarly, the authors could also provide some intuition on why their models worked better than existing SoA methods, particularly with that of [1], which had used additional gaze data but still did not perform as well as the methods discussed in the paper. Did they use unique features that are more informative? Did their techniques better process/use the video data?\n\n- It is difficult to understand Figure 3. What are the values, 0.3, 0.5, and 0.7 representing? The authors could also report prediction results for HOIs for this real-world experiment to understand how the model developed on offline data translates to real-world data.\n\n- It is unclear if the context variable, $cls$ is also obtained from ViT architecture or if there was additional processing done to get $cls$. It is hard to understand the validity of the assumption in Line 145 without knowing how $cls$ was calculated.\n\n- What do the authors consider as context? Is it the environment the interaction is happening in? Is it arrangement of objects? It would be useful to understand what is defined as context here, to understand whether assuming context to be homogeneous within a video is valid. For example, in the scenario demonstrated by the authors, a user might want to have a drink and come closer to the cup but instead change their mind and start working on their laptop. This could be considered as the context changed from \"wanting a drink\" to \"having to work\".\n\n- The Limitations section primarily focusses on the dataset used for model development and its lack of robotics-related scenarios. While this is indeed a limitation, the authors could also expand the limitations section to discuss potential limitations of their models and methodology.\n\n\n- (Line 44) The authors have stated that, “The proposed HOI4ABOT outperforms the state-of-the-art in VidHOI across all HOI tasks.” in their contributions. While this is an interesting result, I would not call this as a contribution but rather as a result/performance of the model they developed, which is the contribution.\n\n- Figure 2 is a very detailed diagram and the authors have provided a lot of information to aid the readers understand the architecture. However, some information is unclear/could be included.\n    - The arrows could be made more visible. It is hard to track the direction of data flow.\n    - The text uses variables $E^t$, $B^t$ and $\\hat{B}^t$ but the diagram uses only $e^t$, $b^t$ and $\\hat{b}^t$. The authors can clarify what these variables are.\n    - Some variables are not explicitly explained, especially on the right diagram (q, $s_m$, $S_m$, $\\hat{WH}$, K, V, Q, etc.)\n\n\n- Following clarifications would be useful\n\n    - In Line 141, the authors claim their patch merger algorithm works better than [2]. Are these results included somewhere in the paper to support this claim?\n\n    - In Line 190, the authors can provide details on the motion primitive identification. For example, how many trajectories were collected, how many motion primitives were identified in total, etc.\n\n    - What is the train-test-val split for the dataset used in the experiments?\n\n    - (Line 260) Is the time horizon in seconds?\n\n    - (Line 270) What are the exact annotations in the dataset that seem appropriate for this assistive task?\n\n    - (Line 272) Are all 20 videos of the same human?\n\n\nReferences\n1. Z. Ni, E. Valls Mascar ́ o, H. Ahn, and D. Lee. Human-object interaction prediction in 364 videos through gaze following. Computer Vision and Image Understanding, page 103741, 2023. ISSN 1077-3142. doi:https://doi.org/10.1016/j.cviu.2023.103741. URL https:  //www.sciencedirect.com/science/article/pii/S1077314223001212.\n2.  J. Park, J.-W. Park, and J.-S. Lee. Viplo: Vision transformer based pose-conditioned self-loop graph for human-object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17152–17162, 2023.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is well-written and easy to follow. The work discusses an important problem of human-object interaction (HOI) that would be useful for robot assistance to humans and thereby improve human-robot interaction. \n\nStrengths\n- The model uses video data which is fairly ubiquitous but abstracts and combines different information that potentially improves HOI identification.\n- The model uses efficient techniques to improve inference times, particularly for varying prediction horizons.\n- The experiments conducted in a real-world setting with actual robots demonstrates the importance of predicting HOIs.\n\nWeaknesses\n- Definitions should be more clearly explained and more explanations could be provided to support the validity of the assumptions\n- Some details on the modeling approach are unclear or missing\n- Limitations of the modeling approach not discussed in detail\n\n\n\n\n\n\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "O79fUQnNuS", "reviewer": "Reviewer_fhCH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a transformer-based model to detect HOIs from videos. The paper contributes a Patch Merger technique for combining the information from a Visual Transformer patch encoding with an object detection bounding box to generate individual representations of all objects. Visual tokens and spatial features are stacked to form human and object temporal windows. Two cross attention transformers combine the human window information with the object window, in order to predict the human-object interaction class. To predict HOIs at multiple time horizons, the authors adopt a multihead variant. \n\nThe model was trained and evaluated on the VidHOI dataset, and performance was compared against five baselines. The results demonstrate that the approach outperforms these baselines on the detection of HOIs task. The multihead pretraining strategy outperforms training the baselines from scratch for different anticipation horizons and facilitates earlier prediction of the human intention.", "summary_of_recommendation": "This is a well-written paper which contributes a Human-Object Interaction detection and anticipation framework. The approach is novel and uses a transformer-based model to detect HOIs from videos, while leverages pretrained models to enhance the inputs to the transformer-based model. I would recommend acceptance with minor edits.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "\n- In Section 3.2, what complex movements required decomposition into simpler movement primitives? What was the performance and robustness of the kinesthetic teaching?\n- In Table 1, in Oracle mode, the Ours (Stacked) method outperforms Our (Dual). However, in Detection mode, Ours (Dual) outperforms Ours (Stacked). In Table 3, the results show Dual (Scratch) does not perform as well as Dual (Hydra). Why this is the case? Is training a Dual model specifically on a particular prediction horizon overfitting to the data?\n- In Section 4.4, are there other quantitative or qualitative measures to demonstrate how do early HOI predictions enable robots to be more effective at this task? Do the human user perceptions of this assistive robot support the need for early HOI predictions?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n\n-  This is a well-written paper which clearly contributes a Human-Object Interaction detection and anticipation framework. \n- The novelty in the approach lies in how the approach leverages pretrained models and aligns these model outputs in a patch-merging strategy, as well as the dual transformer used for the Object Blender and Human Blender transformers.\n-  Being able to anticipate HOIs is an important problem in developing assistive robots which understand human needs. \n- The authors compare the method to a good number of baselines and demonstrate that their methods outperform baselines on the evaluation tasks. \n\n\nWeaknesses/Areas of Improvement:\n\n- In Section 3.2, Motion generation and task planning, details on what complex movements required decomposition into simpler movement primitives, are needed to better understand this was for the robot pouring task. The human operator guides the robot in the beverage pouring task. What was the performance and robustness of the kinesthetic teaching?\n- In Table 1, in Oracle mode, the Ours (Stacked) method outperforms Our (Dual). However, in Detection mode, Ours (Dual) outperforms Ours (Stacked). Why might this be the case? Section 4.2 could provide more discussion on these differences. \n- In Table 3, the results show Dual (Scratch) does not perform as well as Dual (Hydra). Section 4.2 could be expanded to elaborate more on why this is the case. Is training a Dual model specifically on a particular prediction horizon overfitting to the data?\n- Figure 3 should be explained in more detail, as the finding that HOI anticipation facilitates earlier prediction of the human intention, is not immediately clear. The group within the legend should be more clearly explained. \n- In Section 4.4: Real World experiments, since this method is developed towards designing assistive robots, this section would benefit from more evaluation measures of the system. How do early HOI predictions enable robots to be more effective at this task? Do the human user perceptions of this assistive robot support the need for early HOI predictions?\n- The discussion section focuses on interesting limitations within the human data. It would be great for the authors to discuss limitations within the methodology, assumptions required for the nature of the videos, and the generalizability of this framework on other datasets. \n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "MfxAX2GrOb", "reviewer": "Reviewer_5Maz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a Human-Object Interaction (HOI) anticipation framework, HOI4ABOT, to detect and anticipate HOI from videos with less latency. HOI4ABOT, as a two-stage model, first extracts and obtains features for objects and humans per frame with Patch Merger method, and then detect and anticipate HOI using a dual transformer module with cross-attention layers. The paper also proposes a Hydra variant to predict multiple time horizon interactions. The authors conduct experiments with VidHOI dataset and show that HOI4ABOT outperforms state-of-the-art in both mAP and speed, demonstrating that HOI4ABOT is capable of accurately and efficiently detect and anticipate HOI, thus enabling robot to better assist human. The authors also conduct real-word experiments with convincing real-time performance. ", "summary_of_recommendation": "The paper proposes to accelerate the HOI detection and anticipation from videos. The paper is well-written and overall technical sound. The literature review is thorough and experiment results are extensive and convincing. The only doubt is its performance in scenarios with more ambiguity and complexity.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1.\tThe Figure2 is a little bit unclear. Some labels/arrows are missing/incorrect. For example, $E^T$ is missing, there are no arrows from ViT to $E^T$ and $cls_t$, the input to box embedder seems should be $b_m^t$. And for the right part it could be better if the flow is from top to bottom instead of bottom to top.\n2.\tThe question raised in weakness. \n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n\n•\tThe paper is well motivated in such that it proposes improving the efficiency of HOI detection and anticipation from videos for assistive robot.\n\n•\tThe real-world experiment is real-time and well designed with both ‘near the cup’ and ‘away from cup’ cases considered, demonstrating the accuracy, efficiency, and flexibility of proposed method.\n\n•\tThe video clearly demonstrates the real-world experiment.\n\nWeaknesses:\n\n•\tThe task has objects that are very distinct. For example, in the kitchen task, objects like laptop, cup, kitchen are different thus less ambiguity appeared. However, if some similar objects like cup and bowl(human approaches kitchen but plans to grab bowl instead of cup, and corresponding robot actions are different), kitchen and table(they are in similar direction), or even two different cups are present (and the human moves to cup1, and then moves away from cup1 to grab cup2 near the laptop), will the proposed method still be able to quickly distinguish in real time and outperform other methods (as gaze information seems to be important in these cases)? \n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers commend the paper for being well-written and organized. The proposed method to detect human-object interaction using transformers with videos was generally well received by reviewers. Reviewer pnXM brings up valid points which could benefit being addressed in the final paper, including the validation study only being conducted with a small number of lab members. I encourage the authors to further address any remaining points raised in the reviews for the final version of the paper.
Decision: Accept (Poster)