Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes GIRAF, an LLM-based framework to include gesture information in instruction following. GIRAF first uses several descriptors to describe scenes (objects and their 3D positions) as well as human speech and gestures. It then uses an LLM-based planner to generate the task plan by prompting with the instruction and the gesture description. To execute the plan, the referred object is identified by intersecting the pointing ray with the candidate objects. The experiment on a real robot demonstrates that GIRAF provides a more effective communication interface and is preferred by humans. To evaluate if the LLMs can understand diverse gestures described in different fidelity, this paper additionally proposes a GestureInstruct dataset for evaluation.
review: Strengths 
- Incorporating gestures into instruction following tasks is very useful for designing effective human-robot interaction.
- The robot experiment conducted in this paper demonstrates that users consider using gestures as an easy and preferred way to communicate.

Weaknesses 
- It is unclear what the role of LLMs is from the description of the paper. It seems to me that the LLMs can interpret the same pointing gesture differently based on the language instruction and the history. However, there is no experiment to evaluate this context-dependent reasoning which left a question that if we need an LLM here or if it is just to help to parse the instructions.
- The perception API identifies the object/location the user is pointing to. Then a simple approach can be just extracting the action (i.e. verb) referred to in the instruction and then completing the action with the object intersecting with the pointing ray. How does the proposed LLM-based approach compare to a baseline like this or other prior works that also consider pointing gestures?
- The evaluation using GestureInstruct only shows that LLMs can recognize gesture descriptions. However, it doesn’t mean that the classified gestures can be integrated into task planning as the pointing gesture. It will be helpful if the paper can demonstrate how an LLM can take diverse gestures into consideration for task plans.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: It is important to include gestures in the communication modality as it has been demonstrated as effective by user studies. However, the evaluation in its current form cannot justify the benefit of using LLMs in gesture reasoning.

-----
Post-rebuttal update:
I would like to thank the authors for updating the paper to address my concerns. I agree with the authors that the LLM provides context for disambiguating the gesture but I didn't see an experiment that tests how well the LLM helps in disambiguation, for example, just extracting the verb from language and combining it with an object pointed by the gesture can be a good baseline for comparison. In general, I think this is an initial study and a good resource for the community, but I would encourage the authors to include a discussion about the provided information from language (as in the table presented in the rebuttal) in the paper and evaluation on how well LLMs help disambiguate. I'm updating my score accordingly.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper presents a system investigating hand gestures for enhanced human-robot interaction by incorporating off-the-shelf components from prior work. The core contribution appears to be demonstrating the value of gesture information for HRI through the proposed system, rather than introducing novel technical methods.
review: **Strengths**

- The paper proposes a "relatively" complete system for identifying gestures and human language, and feeding this information into the LLM planner and some robot motion execution primitives.
- A user study is included in the paper, making the evaluation more objective.
- The whole system is pretty straightforward.

**Weaknesses**

- Although the paper reports quantitative data, there is not enough clear context provided for each number. For instance, what does the success rate represent? How many total tests were conducted? This lack of clarity applies to many parts of the paper, such as Figure 2, 3, or the success rate on the GestureInstruct dataset.
- From a high-level perspective, humans typically use gestures when they are at a distance from objects. However, in the examples shown in the paper (drawer opening, tool fetching), the human operator is very close to the objects, to the point that there isn't really a need for robots to do the task. Humans can do them themselves. Therefore, these tasks appear somewhat contrived. To make the approach more compelling, it would be beneficial to show examples where human subjects stand far away from the object, and the robot can still identify the human gesture and where the fingers are pointing at, and complete the task. It might be the case that when humans are far away from the object, it would become more challenging to correctly identify the gesture and where the human is pointing at, and hence, the proposed pipeline might not work well.
- The system has many components, and it is likely that each component can have its own failure modes. For instance, when does the heuristic-based referent searching fail? Unfortunately, there is limited discussion on these failure modes at present.
- It seems that there are no physical experiments for gestures other than "pointing"?
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: The paper investigates an interesting use of gestures for human-robot interaction. However, the experiments conducted so far have been shown in contrived setups. It remains unclear how well the system will function when human subjects are far away from the object, where gestures are more important in practice. Additionally, the paper only has limited real-world experiments demonstrating how the system works with other gestures, and the limitations are not thoroughly discussed.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper presents an approach to prompt LLMs for robot planning conditioned on language instructions and gestures. It first uses off-the-shelf perception models to parse the scene and recognize human gestures, and then prompts LLMs with both the scene and gesture descriptors (as code/templates) as well as the language instructions to generate task plans. It then calls low-level controllers to execute the plans. Evaluation on a real robot shows promising results. 
review: Strengths:
1. The idea is technically solid and straightforward.

2. The writing is clear and the videos are helpful.

3. Analyses and discussions are quite informative.

Weaknesses:
1. The method involves combining multiple existing models. It would be good to estimate errors from each model and how errors can be accumulated in difficult cases. E.g., what will happen if perception models fail?

2. It would be good to discuss what type of tasks could be solved by this system with or without gestures. I imagine that some tasks can be directly solved by instruction following (language only) without gestures, while others would have to rely on accurate gesture recognition. And there could be tasks that simple gestures are not sufficient. 

3. LLMs-based planning does not inherently have a safety guarantee. How would you address failures and potentially unsafe plans generated by LLMs?
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This paper presents an interesting idea to use LLM for gesture & language-guided robot planning. There are concerns about safety and robustness to errors made by individual modules. My final rating will depend on how authors address these concerns.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: This paper’s goal is to enable robots to recognize gestures and contextualize them using real time language instructions from humans. The challenge here is that the intent behind a gesture strongly depends on the context (pointing at a cup can mean pour into the cup or pick up the cup, but you don’t know which one the human wants without more information than just the gesture). Similarly, language on its own can be ambiguous without gestures ("open this drawer" is very unclear). They propose a framework that leverages LLMs to contextualize the meaning of human gestures and infer human intent from them. The idea is to have a model for identifying the gesture, then prompt an LLM for reasoning about the gesture-language pair and generate a robot policy from that. The paper demonstrates that the proposed method, GIRAF, is preferred by users in a user study when compared to a language-only baseline, and also curates and publishes a dataset of gestures that can be useful for future research.
review: Overall, I thought this paper was very well-written and enjoyable to read! The framework is sound, novel and clever; the experiments are thorough and compelling.

I have 2 comments that I’d like to see addressed.

First, a strong motivation for the paper is that gestures can be ambiguous (e.g. pointing at a cup could mean “pick up the cup” or “pour in this cup”), yet looking at Fig. 3 every gesture has only one corresponding language utterance. I would’ve liked to see language resolving gesture intent ambiguity, yet as far as I can tell that isn’t explored. Essentially, the user study shows that gestures are helpful for disambiguating language, but the intro also motivates the other direction: how language can be useful to disambiguate gesture intent. Have you tried an experiment like this, where you measure success rate for disambiguating between multiple meanings of the same gesture?

My second comment is that I’d like to see more discussion on the types of situations where the human knows how to use an object yet can’t name it. In the “open this drawer” it is clear why using gestures is important: the human clearly could say “open the 3rd drawer on the 2nd row”, but that is cumbersome and gesturing helps with that. But in a situation like “give me this tool”, when would a human not know how to name the tool yet know how to use it? What prevents the human from saying “give me the screwdriver”? I think it’s really important for the applicability of the method to have some discussion on when these kinds of situations emerge, otherwise the reader might be left feeling like this problem is a little bit manufactured.

Small comments:
- Abstract typo: “deictib” should be “deictic”
- Intro typo: “a pointing” should just be “pointing” 
- Fig. 3 caption “gestuers” → gestures
- Discussion: “langauge” → language

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I really like this paper and I think it would be a wonderful contribution to CoRL! I think the ideas presented are novel, the multimodal framework could potentially be very useful in the embodied robotics community, and the gestures data set would be helpful for future researchers. The paper is somewhere between a weak and a strong accept for me, mostly because of the 2 issues above I would like to see addressed.
review_title: null

Meta-review: Summary of the paper:

The paper presents GIRAF, which enhances human-robot interaction by integrating gestures with language instructions. GIRAF leverages Large Language Models (LLMs) with off-the-shelf perception models to interpret scene descriptors, recognize human gestures, and generate task plans accordingly. Evaluation on a real robot reveals the potential benefits of this method. Additionally, the authors introduces the GestureInstruct dataset, designed to examine the ability of LLMs to understand and categorize diverse gesture descriptions.

Strengths:
- Novel Integration: The paper's main strength is combining gestures with language instructions to design effective human-robot interaction.
- Practical Implications: The robot experiment underlines that gestures are an intuitive and preferred method of communication for users, potentially reshaping future HRI designs.
- Comprehensive System Design: The manuscript presents a holistic system, encompassing gesture identification, language processing, LLM-based planning, and execution of robotic tasks. A user study adds weight to its practical evaluation.
- Dataset Contribution: The GestureInstruct dataset is a promising tool for further research in gesture-based robotic instructions.
- Clarity & Resources: Reviewers appreciated the clear writing, informative discussions, and the helpful videos accompanying the paper.

Weaknesses:
- Role & Necessity of LLMs: Multiple reviewers expressed concerns regarding the clear definition and the actual need for LLMs in the context of gesture reasoning. Questions were raised about the LLM's contribution beyond gesture classification and task planning.
- Comparative Analysis: There's a perceived gap in comparing the proposed LLM-based method with other baselines, especially those which already account for pointing gestures. This makes it challenging to pinpoint the actual advantages of GIRAF.
- Evaluation Ambiguity: The quantitative results lack clear context regarding the metrics' meanings and the total tests performed, making it hard to interpret the evaluation's significance.
- Experimentation Limitations: Some experiments, like those involving only "pointing" gestures or where the human is very close to the object, seem contrived and don't necessarily reflect real-world scenarios.
- System Robustness: The approach's reliance on several existing models introduces potential error propagation. There are concerns about the robustness and safety of plans generated by LLMs, especially when individual modules fail.
- Contextual Nuances: The paper suggests gestures can help clarify language instructions, but there is a limited exploration on how language can clarify ambiguous gestures. Also, some presented situations, like "give me this tool," might seem artificially constructed without sufficient real-world applicability discussions.

Rebuttal and Discussion:

The authors responded to the questions from the reviewers thoughtfully and addressed several of the concerns raised. Furthermore, they submitted a revised manuscript reflecting the results of the discussion, and the quality of the paper improved with detailed data added to the appendix. However, not all concerns have been completely resolved, and the revised manuscript exceeds the page limit. Therefore, further revisions are required.

Recommendation:

Given the novel integration of gestures and language for HRI, GIRAF's proposition is promising. However, for a clearer evaluation of its significance and a broader acceptance in the community, it is imperative to address the stated concerns, particularly around the precise role of LLMs, comparative analyses with baselines, and the system's robustness in diverse scenarios.
Decision: Accept (Poster)