Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper presents a transformer-based model to detect HOIs from videos. The paper contributes a Patch Merger technique for combining the information from a Visual Transformer patch encoding with an object detection bounding box to generate individual representations of all objects. Visual tokens and spatial features are stacked to form human and object temporal windows. Two cross attention transformers combine the human window information with the object window, in order to predict the human-object interaction class. To predict HOIs at multiple time horizons, the authors adopt a multihead variant. 

The model was trained and evaluated on the VidHOI dataset, and performance was compared against five baselines. The results demonstrate that the approach outperforms these baselines on the detection of HOIs task. The multihead pretraining strategy outperforms training the baselines from scratch for different anticipation horizons and facilitates earlier prediction of the human intention.
review: Strengths:

-  This is a well-written paper which clearly contributes a Human-Object Interaction detection and anticipation framework. 
- The novelty in the approach lies in how the approach leverages pretrained models and aligns these model outputs in a patch-merging strategy, as well as the dual transformer used for the Object Blender and Human Blender transformers.
-  Being able to anticipate HOIs is an important problem in developing assistive robots which understand human needs. 
- The authors compare the method to a good number of baselines and demonstrate that their methods outperform baselines on the evaluation tasks. 


Weaknesses/Areas of Improvement:

- In Section 3.2, Motion generation and task planning, details on what complex movements required decomposition into simpler movement primitives, are needed to better understand this was for the robot pouring task. The human operator guides the robot in the beverage pouring task. What was the performance and robustness of the kinesthetic teaching?
- In Table 1, in Oracle mode, the Ours (Stacked) method outperforms Our (Dual). However, in Detection mode, Ours (Dual) outperforms Ours (Stacked). Why might this be the case? Section 4.2 could provide more discussion on these differences. 
- In Table 3, the results show Dual (Scratch) does not perform as well as Dual (Hydra). Section 4.2 could be expanded to elaborate more on why this is the case. Is training a Dual model specifically on a particular prediction horizon overfitting to the data?
- Figure 3 should be explained in more detail, as the finding that HOI anticipation facilitates earlier prediction of the human intention, is not immediately clear. The group within the legend should be more clearly explained. 
- In Section 4.4: Real World experiments, since this method is developed towards designing assistive robots, this section would benefit from more evaluation measures of the system. How do early HOI predictions enable robots to be more effective at this task? Do the human user perceptions of this assistive robot support the need for early HOI predictions?
- The discussion section focuses on interesting limitations within the human data. It would be great for the authors to discuss limitations within the methodology, assumptions required for the nature of the videos, and the generalizability of this framework on other datasets. 

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This is a well-written paper which contributes a Human-Object Interaction detection and anticipation framework. The approach is novel and uses a transformer-based model to detect HOIs from videos, while leverages pretrained models to enhance the inputs to the transformer-based model. I would recommend acceptance with minor edits.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The key idea of the paper is to model human-object interaction using temporal cues from video data using transformers. A major motivation for the development of the model is real-time inference of human-object interaction detection and prediction for robotic assistance.

A major contribution of the paper is the development of a human-object interaction model that is able both detect and anticipate human object interaction better than existing models. The model is efficient and is able to provide real-time inference. Other contributions include testing on a real robot in a assistive setting by planning robot actions based on the predicted object interaction.
review: The paper is well-written and easy to follow. The work discusses an important problem of human-object interaction (HOI) that would be useful for robot assistance to humans and thereby improve human-robot interaction. 

Strengths
- The model uses video data which is fairly ubiquitous but abstracts and combines different information that potentially improves HOI identification.
- The model uses efficient techniques to improve inference times, particularly for varying prediction horizons.
- The experiments conducted in a real-world setting with actual robots demonstrates the importance of predicting HOIs.

Weaknesses
- Definitions should be more clearly explained and more explanations could be provided to support the validity of the assumptions
- Some details on the modeling approach are unclear or missing
- Limitations of the modeling approach not discussed in detail







robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: My recommendation is based on the following
- While the modeling approach uses transformers similar to some of the existing methods, the way temporal data is abstracted and how the transformer model is designed is distinct enough to improve HOI prediction.
- The model uses efficient techniques for real-time inference using temporal data suitable for real-world robotics applications.
- The demonstration of the model and corresponding robot planning on a robot assistance task motivates the necessity for HOI prediction.
- While the paper provides a lot of details on the model approach and the evaluation experiments, some important details are missing which are required to even understand the validity of some of the assumptions and for potential replication.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper presents a method for predicting human activity to allow a robot to anticipate and execute appropriate collaborative actions.  The system uses a vision-based machine learning model to predict the action, behavior trees to choose the robot action, and kinesthetic teaching combined with a motion planner to move the robot.  The proposed method achieves a significant speedup over prior approaches with a minor improvement in accuracy.
review: Strengths:
- This work tackles a difficult and important problem in robotics (anticipating human behavior)
- The work includes a fully implemented system on a robot
- The proposed methods for human activity recognition is likely to be of interest to roboticists

Weaknesses:
- The validation with the robot ("real world experiments") is not well-designed and is lacking detail (was the human in the study a member of the research team? was the study approved by a university ethics board?).  For a paper where the primary robotics innovation is incorporating the method in a system, it is important to have a strong validation to demonstrate that the method is effective and robust.
- Overall, the robotics side of the work is somewhat weak; although the authors argue the method will improve human-robot interaction, the paper does not provide sufficient evidence to support that claim, or provide experimental evidence that the method is effective across different types of interaction.


Minor points:
- the authors may want to refer to their system as a "collaborative" rather than assistive robot -- typically "assistive robotics" are robots intended to be used by disabled people.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, this is an interesting human activity recognition system but the robotics experiments seem incomplete.  Further validation to demonstrate that the proposed method works across tasks and users would make this a strong submission to a future venue.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes a Human-Object Interaction (HOI) anticipation framework, HOI4ABOT, to detect and anticipate HOI from videos with less latency. HOI4ABOT, as a two-stage model, first extracts and obtains features for objects and humans per frame with Patch Merger method, and then detect and anticipate HOI using a dual transformer module with cross-attention layers. The paper also proposes a Hydra variant to predict multiple time horizon interactions. The authors conduct experiments with VidHOI dataset and show that HOI4ABOT outperforms state-of-the-art in both mAP and speed, demonstrating that HOI4ABOT is capable of accurately and efficiently detect and anticipate HOI, thus enabling robot to better assist human. The authors also conduct real-word experiments with convincing real-time performance. 
review: Strengths:

•	The paper is well motivated in such that it proposes improving the efficiency of HOI detection and anticipation from videos for assistive robot.

•	The real-world experiment is real-time and well designed with both ‘near the cup’ and ‘away from cup’ cases considered, demonstrating the accuracy, efficiency, and flexibility of proposed method.

•	The video clearly demonstrates the real-world experiment.

Weaknesses:

•	The task has objects that are very distinct. For example, in the kitchen task, objects like laptop, cup, kitchen are different thus less ambiguity appeared. However, if some similar objects like cup and bowl(human approaches kitchen but plans to grab bowl instead of cup, and corresponding robot actions are different), kitchen and table(they are in similar direction), or even two different cups are present (and the human moves to cup1, and then moves away from cup1 to grab cup2 near the laptop), will the proposed method still be able to quickly distinguish in real time and outperform other methods (as gaze information seems to be important in these cases)? 

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper proposes to accelerate the HOI detection and anticipation from videos. The paper is well-written and overall technical sound. The literature review is thorough and experiment results are extensive and convincing. The only doubt is its performance in scenarios with more ambiguity and complexity.
review_title: null

Meta-review: The reviewers commend the paper for being well-written and organized. The proposed method to detect human-object interaction using transformers with videos was generally well received by reviewers. Reviewer pnXM brings up valid points which could benefit being addressed in the final paper, including the validation study only being conducted with a small number of lab members. I encourage the authors to further address any remaining points raised in the reviews for the final version of the paper.
Decision: Accept (Poster)