Gesture-Informed Robot Assistance via Foundation Models
Authors: Li-Heng Lin
Yuchen Cui
Yilun Hao
Fei Xia
Dorsa Sadigh
Venue: CoRL 2023 Poster (2023)
Keywords: Planning with Gestures
Human-Robot Interaction
LLM Reasoning

Abstract: Gestures serve as a fundamental and significant mode of non-verbal communication among humans. Deictic gestures (such as pointing towards an object), in particular, offer valuable means of efficiently expressing intent in situations where language is inaccessible, restricted, or highly specialized. As a result, it is essential for robots to comprehend gestures in order to infer human intentions and establish more effective coordination with them. Prior work often rely on a rigid hand-coded library of gestures along with their meanings. However, interpretation of gestures is often context-dependent, requiring more flexibility and common-sense reasoning. In this work, we propose a framework, GIRAF, for more flexibly interpreting gesture and language instructions by leveraging the power of large language models. Our framework is able to accurately infer human intent and contextualize the meaning of their gestures for more effective human-robot collaboration. We instantiate the framework for three table-top manipulation tasks and demonstrate that it is both effective and preferred by users. We further demonstrate GIRAF’s ability on reasoning about diverse types of gestures by curating a GestureInstruct dataset consisting of 36 different task scenarios. GIRAF achieved 81% success rate on finding the correct plan for tasks in GestureInstruct.
Videos and datasets can be found on our project website: https://tinyurl.com/giraf23

Gesture-Informed Robot Assistance via Foundation Models
Li-Heng Lin', Yuchen Cuilr, Yilun Hao', Fei Xia", Dorsa Sadigh
Gestures are fundamental and significant mode of non-verbal com- munication among humans. Deictic gestures (e.g. pointing), in particular, offer valuable means of efficiently expressing intent in situations where language is inaccessible, restricted, or highly specialized. As result, it is essential for robots to comprehend gestures in order to infer human intentions and establish more ef- fective coordination with them. Prior work often rely on a rigid hand-coded library gestures along with their meanings. However, interpretation of gestures often context-dependent, requiring more flexibility and common-sente reasoning. In this work, propose framework, GIRAF, for more flexibly interpreting gesture and language instructions by leveraging the power of large language models. Our framework is able to accurately infer human intent and contextualize the meaning of their gestures for more effective human-robot collaboration. We instantiate the framework for interpreting deictic gestures in table-top manipulation tasks and demonstrate that is both effective and preferred by users, achieving 70% higher success rates than the baseline. We further demonstrate GIRAF's ability on reason- ing about diverse types of gestures by curating Gesturelnstruct dataset consisting of 36 different task scenarios. GIRAF achieved 81% % success rate finding the correct plan for tasks in Gesturelnstruct. Project website: sinyurl.com giraf23
Abstract:
Keywords: Planning with Gestures,
LLM Reasoning
1 Introduction
Gestures important form of communication that frequently use in everyday activities such at traffic intersections, restaurants, and department stores. They used to disambiguate and communicate our intent especially when language is not available or limited (e.g.. waving at a car to pass through), or when language is too specialized (e.g. pointing at hex screwdriver vs cross slor screwdriver instead of referring to them by their name as shown in Fig. 1). The ability to understand human gestures is thus critical for autonomous robots to predict human intent, and be able to respond effectively. While prior work has studied how gestures can be beneficial in human-robot interaction, these approaches often rigid, requiring extensive engineering effort for predefining library of fixed gestures accompanied with their corresponding meaning [1. 2, 3, 4]. This not only is expensive, but also assumes fixed mapping between gestures and their meaning. However, interpretation of gestures can be highly dependent on the context and may require situated reasoning leading to different human intents. For example, pointing at cup calls for potentially different actions, such as picking up the cup or pouring into the cup given different contexts and history.
Recent works have demonstrated that foundation models such as large language models (LLMs) trained on internet data have enough context for commonsense reasoning [5, 6, 7, 8]. making moral judgements [9, 10, 11], or reward design [12, 13]. Similarly, we expect LLMSs to be able to reason about and interpret gestures under the assumption that they are prompted with a textual description of the gesture and the context. Addressing this assumption is often referred to as the grounding problem. However, grounding gestures not as simple as recognizing the gesture class (e.g.. thumbs-up,
Computer Science Department, Stanford University, Stanford, CA, USA. Google Deepmind. Corresponding author. Email: yuchenc@stanforr edu
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Task Specification
Scene Descriptor Paese
LLM Task Planner
Robot Execution
Oiject Locatization
toasons about speech and gesture Instructions detect, referred, "tool",
Motien Primittres
Object Segmurmation
pictup, pitse__11_f.p
Cbject Clewiikation
Perception API
Gesture Referent Dosotinn
Oulput - Hfas pt)..... Pxo))
Semanti Elaer
The bay"s a) (herect"
Fand Lenoliitt wtt.a/nc.ooooaaaaaa. rntert. popoders)
(haaner", pil
hasd_vvrr
"Give me that tool"
Descriptor Drunan
Speech
Recogeilion
thench 5o) merl(s
LLM gonerates sequence of sobot actions (polley at
Clasefication
piikkp, 281 att ht_pee/tooi__pps haed_over()
Instruction:
Give
tool
Centure:
[painting]
Figure I: GIRAF System Diagram. GIRAF consists of 1) grounding modules including a scene descriptor Dseene. human descriptor Dituman and set of perception API: and an LLM task planner that is prompted with few-shot demonstrations reason ahout the high-level multiomodal task specification and generate code calling functions from other modules. Grounding modules are colored brown, and reasoning modules are colored yellow. As an example, human user provides language instruction "give me that tool" along with pointing gesture to specify the particular tool they want. GIRAF process information about the objects in the scene with Dscese and grounds the human input with Dummmn The LLM task planner is prompted interpret the multimodal instruction and produce policy code that commands the robot to complete the human's request.
waving, or pointing), as it also requires detecting the context the gesture is presented in such as the referent of deictic gestures (i.e. what the person pointing at).
While recent work shows that vision-language models (VL.Ms) can provide some amount of grounding by describing the scene, these models are limited in two key ways: First, many of them focus on describing the objects but not the state of the human, and thus their context is limited to the scene but not the history of human actions or intents. Second, even if one could extend VLMs to go beyond the scene and describe humans, existing VLMs lack enough geometric reasoning capabilities to accurately interpret gestures. The absence of geometric reasoning in pre-trained VLMs is not surprising since it falls outside the scope of their original objectives. Furthermore, fine-tuning VLMs to capture geometric reasoning about gestures requires considerable amounts of 3D data [14].
Instead of only relying on VLMs fully reason about gestures, insight is to leverage combina- tion of existing pre-trained vision models along with other contextual information such as language instruction to ground LLMs to reason about human gestures and accurately infer their intent. We pro- pose a system, GIRAF cresture-Informedd Robot Assistance via Foundation-Modell)) that leverages expert models for identifying gestures, and prompts an LLM for reasoning about languuee instructions and generating robot policies. Specifically the L.MM will directly generate robot code, for both perceiving and acting, given the language and gesture descriptions. An overview of the system is shown in Fig. 1. In this example, the user wants tool that they do not know how to describe in language and use gesture to specify what tool the robot should grasp. The user provides language and gesture specification, in this case "Give me that tool" along with pointing gestures towards the tool to disambiguate the language instruction. Our framework consisting of a Human Descriptor and a Scene Descriptor detect the pointing gesture and objects in the scene. Then our LLM Task Planner reasons about the task and uses a Perception API to detect the object being referred by the gesture. The LLM then directly generates motion primitives such as picking up and handing over the item.
To evaluate the effectiveness of our system, we instantiate GIRAF for instruction following in table- top manipulation settings. We first conduct a user study focusing on interpreting deictic pointing gestures, showcasing how language alone can be inefficient or ineffective for specifying a task. Our user study shows that users significantly prefer GIRAF over a language-only baseline and are also better at achieving task goals when using GIRAF with more than 70% higher success rates. We then construct a Gesturelnstruct dataset demonstrate the capability of LLMs reason about gestures in various contexts in zero- or few-shot manner. GIRAF achieved 81% success rate finding the correct plan for tasks in GestureInstruct.

---

2 Related Work
Gestures for Human-Robot Interaction, As an important mode of non-verbal communication, gestures have been studied both within and outside the context of human-robot interaction (HRI). The full literature is out of the scope of this paper and we refer the readers the survey work of Vuletic et al. [15] gestures in HRI. In this section we focus the most relevant literature of interpreting rercch-related gestures or co-verbal gestures [15]. Prior work on incorporating gestures for HRI often relies on defining a fixed set of gesture vocabulary and hand-coding their mappings to robot actions [1, 2, 3, 4]. One major limitation these rule-based methods is that the human user needs to learn which gestures to use for eliciting which robot response within predefined context. Prior work has also explicitly focused on understanding deictic gestures such as pointing, Matuszek al. [16] and Chen et al. [17] take data-driven approach for interpreting natural deictic gestures. Whitney et al. [18] propose a multimodal Bayes filter for interpreting referential expressions. These methods are specialized referent detectors that require either large amounts of in-domain training data or extensive engineering effort for each novel task domain. In contrast, GIRAF offloads fusion of multimodal instructions to LLMs and implements a heuristic-based referent detection module leveraging VLMs and simple geometry so that can perform referent detection in a zero-shot manner.
LLM-based Planning for Roboties. Large language models have been adapted the user interface for various roboties applications. LLMs have shown promising performance in a variety of complex reasoning tasks [5, 6, 10, 12]. Multiple efforts have demonstrated planning capabilities of LLMs in robotics, including zero-shot generation of high-level task plans [19]. reasoning about the state of the task from various sources of feedback [20], planning with contextually appropriate and feasible actions [21, 22], and directly producing robot code [23, 24]. These works ground task scenes using VLMs but do not explicitly model the presence of the human beyond providing language instructions and feedback. However, L.MSs and VLMs therefore these systems fail in tasks where there are multiple semantically similar objects or the user just does not have the right language to describe an item. GIRAF bypasses complex geometric reasoming from language alone by incorporating a human descriptor that grounds human gestures for providing richer feedback for LLM reasoning.
pretrained for complex geometrie reasoning and
Gesture-Informed Robot Assistance via Foundation Models
Problem Statement. We define the problem of embodied multimodal instruction following as: given task specification in the form of synchronized speech S and visual gestures V, robot needs to output control policy that satisfies this task specification. We assume access to set of parameterized robot action primitives A = (01)......... a" (0.)). The robot policy needs to return a sequence of action primitives with parameters specifying object locations.
The key insight of our system is that we can leverage LLMs pretrained on large-scale human-generatedd text corpus for reasoning about such multimodal instructions, and effectively turning the instruction following problem into two sub-problems: 1) the grounding problem of translating the raw human input S, V and scene observations O into form that LLMs can consume, and 2) the prompting problem of enabling L.MSs to reason about the grounded context for producing a robot policy π.
We propose an -Lsed framework for multimodal instruction following, named robot assistance via foundation models (GIRAF). The system diagram of GIRAF is shown in Fig. 1. In the rest of this section, discuss how approach each of the sub problems in multimodal instruction following, and provide implementation details for GIRAF.
informed
3.1 Grounding Speech, Gestures, and Scene
For the grounding problem, we introduce different descriptors to separately ground the scene and the human in the scene. The scene descriptor Dseene provides information about objects while the human descriptor D Dumman textualizes the speech instruction and the gesture provided. At the same time, instead of passively detecting all possible objects and gesture-related information and feeding them
3

---

to the LLM planner, we prompt the LLM to first reason about the speech and gesture instructions, and then decide which perception tools to use for detecting necessary information. The perception tools include expert gestures models that extract gesture information for different gesture classes.
Scene Descriptor. Dseene is used describe object categories and locations in the scene. Given an image of the scene, Dsccus performs Object Segmentation using Segment Anything[25], and then compute object centers (Object Localization) by de-projecting 2D pixels to 3D positions using depth information. We use OpenCLIP [26] to find object labels by comparing the similarity between the embedding of the segmented object image with a list of all possible object labels. This list of all possible labels can be provided by object detectors or the human user. In our experiments, we use ground truth object lists to minimize error introduced by object detectors. (Objecr Classification). The scene deseriptor outputs object information set O ((1.:P))...... (990) (90)) .POO)) where ca
and p3 the label and 3D position of object i respectively.
Human Descriptor. The goal of DHuman is to textualize human speech and gesture. For speech recognition. GIRAF processes audio input and turns speech into text with each word marked with its timing. The timing information is used for determining which visual frames to process for aligning with gesture information. We employ Microsoft Azure [27] for this purpose. Gestures can be described with different levels of details or fidelity. For example, a "pointing" gesture can also be described "index finger extends out and others curl inward", or even numeric values of the relative position of each finger tips with respect to the wrist. In addition, gestures can be static (e.g. thumbs-up, pointing at an object) or dynamic (e.g. waving, drawing circle over group of objects). It is non-trivial to design a universal gesture detector based on hand information only due to the context-dependant nature of gestures. We find LLMs are capable of reasoning about gesture representations with different level of fidelity but perform best with when they available and can be reliably detected (more details in Section 4).
gesture labels
In our instantiation of GIRAF, we employ off-the-shelf MediaPipe [28] hand detector to extract hand features and train deep neural networks to predict set of selected gesture classes. We train supervised classification models (one for static gestures, another for dyanmic ones) to classify gestures using data from EgoGesture [29]. Specifically, we extract the 1) keypoints of detected hands in image coordinates. 2) keypoints of detected hands in world coordinates, and 3) confidence of detected hands from the output of MediaPipe hand detector and concatenate them the input feature to gesture classification models. We train separate models with different architectures for classifying static and dynamic gestures. For the static gesture model, we use a three-layer multilayer perceptron (MLP) with ReLu as the activation function. For the dynamic gesture model, we input the preprocessed landmark features into a recurrent neural network (RNN) with an LSTM unit. We then feed the output of the RNN to a linear layer. We find stat-of-the-art VLMs can also provide promising gesture classification results on static gestures but often fail to detect the correct referent (see Appendix F for more details) and therefore we design specialized referent detection module.
The output of Dituman are two lines of code in comment (as shown in the bottom left box of Fig. 1) one for language instruction, the other for which we then feed to the LLM task planner.
Perception API. In this work, we mainly focus on deictic gestures, where in addition to detecting the gesture, our system needs detect the referent the gesture is referring to. The task planner will decide whether the deictic gesture is referring an object, a location, referred object, the referent detection function takes an object name Ourget and a timing &ggesture for gesture frame as inputs. We first query semantic filter with Otarget to get a set of candidate object classes C. We implement semantic filter by prompting the LLM to return a list of candidate object names given a label to harness LLM's capability of reasoning about object semantics. For example, when the user says "Give me that tool", C will only contain labels that can be classified as "tool" (subject the LLM). We then filter a list of candidate object instances O using C:
direction. To find the
O (ck,)P) pk)) E andde; e, E C)
Next, we index the visual frames using tgesture, leverage MediaPipe [28] to detect hand keypoints, and de-project 2D keypoints to 3D keypoints. We then find the pointing ray that starts at the detected

---

index finger tip Pip and goes in the direction of the vector from index finger pip joint Ppip to the tip (denoted as 7 in Fig. 1). Let 7 finger denote the unit vector from Ppip to Prip- The pointing ray. is 7 (Pri, iimger) To identify the referred object, we use a heuristic that returns the object in O that is closest to the pointing ray:
(Cragget: Ptarget) =
arg min [D(pi, 7)]
(cupi)E
To find a referred position, we follow a similar procedure as above, but instead of considering the positions corresponding to objects, we find the point in the entire point cloud that closest to 7 If the user pointing for referring to direction, we return V finger as the referred direction.
3.2 Prompting LLMs for Task Planning
To adapt LLMs for planning, we take the approach from the work of Liang et al. [23] and directly prompt the LLM to generate language model programs (LMPs) in Python by providing perception APIs, motion primitive APIs, and demonstrating how to use each of these functions. The LLM we use in our experiments is GPT-3.5 (text-davinci-003) with 0 temperature. An example plan generated by LLM for the instruction "give that tool" with a pointing gesture is shown in Fig. 1. The LLM planner first reasons about the instructions and realizes this is deictic gesture, therefore leverages perception APIs to locate human hand and the referred object. The LLM then sequences the motion primitives to pick up the detected referent and hand it over to the human. We show below another example plan generated by the LLM planner for a similar task. Here the human breaks up the instruction into two steps and request the robot to pick up the object first and then hand it over. To show that the LLM planner can also reason with gestures of different fidelity level, we use detailed description of the gesture shape instead of high-level gesture label in this example:
# Instruction 0: pick the vater jug
Geature: index ringer extenda out while othera curl inxard seeeett_reeerre_co__.pp.. (to.ddlobpppppeetteeeeeeee ("vaer jug')
vater_.jug.pos uur.
Il Instruction 1: hand it to me
Gesture: open palm upward
target_pos = datect_ detact__and__cmmettt conder_eetrrpp nove.gripper. to. (target..o));; gron.grippert
Note that we removed code for confirmation and error detection in these examples to highlight LLM's reasoning capability for interpreting gestures. In our implementation of GIRAF, we leverage Microsoft Azure [27] text-to-speech functionality and include examples in the prompt such that the robot would 1) confirm with the user by repeating the instruction it receives, 2) report error if it cannot produce bug-free code failed detect gesture (when it thinks the speech should accompany gesture), and 3) ask the user if it took correct first movement to decide whether to continue executing or abort. For example, before picking up anything, the robot will move to the object, "point" at the predicted target object with its gripper, and ask the human if it has located the correct target. If the human responds "no", then the robot will abort the picking up action. This level of transparency adds some overhead to the interaction but asserts that the robot will not make unexpected moves, which is important for ensuring safety when we do not have full control over what the LLM outputs.
4
Experiments
4.1 Can GIRAF enable instruction following robots to be more natural and efficient?
We hypothesize that incorporating gesture information, especially deictic pointing gestures, would make task specification easier and more natural for the human, outperforming prior language-only approaches to instruction following. Concretely, we make the following hypotheses:
H1. GIRAF preferred by human users compared to a language-only instruction following
method for specifying goals that are ambiguous hard describe.
H2. GIRAF is quantitatively more effective compared to the language instruction following method for specifying goals that are ambiguous hard to describe.
5

---

(eoscution) GIRAE (eoection))
GIRAB
EXI
Open Drawers
wieeet Ideniiedd
roccchel successfilly
eaey by communiicte
apain
fruetroned
SUCD0eS tnlee
undastood
12. Fetch Tools
Indeuction underwoond
roforent
excculed
ency to commanictte
trist
will use apain
Erustrated
SULTIIN raller
wussowfully
Figure 2: User Study Results. GIRAF is rated higher by users and enables better performance than baseline language-only method CaP [23]. Note that for performance we report both planning success rates (identifying the correct referent) and execution success rates.
To test H1 and H2, conduct
study with two table-top manipulation tasks where the user instructs 7-DoF Franka Panda robot arm to manipulate specific objects on the table while there multiple instances of semantically the same or similar objects in the scene. We compare our method with the baseline language-only method CaP [23]. Specifically, we designed two tasks shown on the left side of Fig. 2:
T1: Open Drawers In this task, there are 64 drawers in a cabinet and the needs to specify one for the robot to open. Here, we expect language to not be an efficient modality, as a language-only method would require the user to specifyy the exact row and column of the drawer.
T2: Fetch Tools In this task, a user asking the the robot the hand over a tool they would
like to use. These tools require the human to use specialized language to refer to them, and are additionally difficult for the VLM classify correctly.
We recruited 11 participants for the user study (6 female, 5 male). 6 out of the 11 participants are native English speakers. Each user participated in the two different task settings as described above. For each task, they select three different target objects, resulting in about 30 tests in total, and for each object, they have at most trials (speech recognition failures do not count towards trials). This study IRB approved. More details are included in Appendix and implementation the motion library is included in Appendix D.
Gestures provide an easier mode communication when interacting with robots. shown in Fig. 2, users rated GIRAF higher (except for frustration, which lower better) than the baseline across qualitative measures and GIRAF scored highest for "easy communicate" (supporting H1 with < .01). We also see that the baseline is rated lower for fetch tools task than in open drawers task as we expected. Most users could count and describe the position of drawers they want but lack the language to specify the tools.
Gestures enable higher success rate in instruction following. The rightmost plots of Fig. 2 show the success rates baseline method CaP and GIRAF for both tasks our user study (supporting with p < .01). We plot both the planning success rate and the execution success rate defined in Appendix C. GIRAF achieves higher success rates on both tasks, and we also find out one major limitation for the baseline identify the correct object purely from language lying in the fact that existing LLMs are not good geometric reasoning such understanding relative positions.
4.2 Can GIRAF reason about diverse types of gestures?
We evaluate the LLM planner's capability of reasoning about diverse types of unseen gestures in HRI settings in zero- few-shot manner. We curated the dataset GestureInstruct with 36 different

---

Label
Type
Language
Usage
pointing
deictic
"can you give me this"
ask for an objoct
gestures that Gesture Label Gesture Description
measuring with
konic
'move much to the left'
specify distance
two fingers
lighting up
iconic
"find this"
ask for object
OK
symbolic
signals "continue"
define ad-loc
fist
semaphoric
"come here"
language
Deictic
Iconic
Symbolic Semaphoric
Figure 3: Reasoning about diverse gestures. Representative gestures of each type (left). Success Rate of Gesture Reasoning with GIRAF on Iesturelnstruct using different gesture representations (right).
up coffee pod."
"Come over here."
up that mug."
pointing
pointing
"Bring
pointing
open
Figure 4: Long horizon interaction multiple gestures. visualize a long-horizon interaction using GIRAF that involves multiple different gestures unseen in the prompt, and pointing gestures that refer to object, location, direction depending on context.
speech-gesture task scenarios in human- nommn interaction settings (the complete dataset along with annotations is on our website). We generated each task scenario under four different communicative semaphoric, iconic and deictic and include both static and dyanmic gestures (see details Appendix B). Symbolic gestures conventional gestures used to convey fixed meaning and can replace language (e.g. thumbs-up). Semaphoric gestures human design for specific purposes and associate meaning by telling the robot what it means, then they would use them again like symbolic gestures (e.g. ask the robot to turn around on a snapping gesture). Deictic gestures are gestures used to physically refer to an object, a location, or direction (c.g. pointing with index finger). Iconic gestures are gestures that are used to represent objects, actions, intents, and abstract concepts (e.g. use two hands opening and closing represent book).
gesture sypes-symbonn
Each task scenario in GestureInstruct consists of image sequences of images showing the visual gesture, a high-level gesture label, an accompanying language instruction (except for symbolic gestures), an interaction context (e.g. robot state and the scene), and description of the human intent in the form of goal state. An example task scenario GestureInstruct is using iconic gesture "hammering", and ask the robot "give me the tool that does this" in the context of "robot sees screwdriver, a wirecutter and a hammer". The LLM task planner needs infer the correct type of gesture and call the corresponding function to handle detection and grounding of iconic gestures in order to find the correct tool. Additional examples of gesture-specch instruction-followee scenarios under each gesture type are shown in Fig. 3 (left) and the full dataset is on our website.
task
GIRAF able to reason about a diverse set of gestures in zero-shot manner. Our results show that GIRAF achieves 80.6% success rates across all task scenarios in GestureInstruct dataset. Fig. (right: dark orange bars) shows the results under each gesture type and we find that GIRAF performs better on object-related gestures (deictic and iconic) comparing to symbolic or semaphoric gestures. Our conjecture is that both symbolic and semaphoric gestures not accompanied by language, so the reasoning needs to purely rely on the gesture itself. Since LLM does not actually see the gesture,

---

its ability to reason about these gestures is limited. We also implement the full pipeline of GIRAF to show its reasoning capabilities for various gestures in a long horizon task. This long-horizon interaction is visualized in Fig. 4, demonstrating that GIRAF is able to reason about symbolic gestures, iconic gestures, and deictic gestures referring to different entities a single interaction. We provide additional example use cases in the Appendix.
4.3 Can GIRAF reason with different representations of gestures?
To understand what is the best way to ground gestures, we investigate what kind of representation enables high reasoning performances. Gestures can be represented differently with varying level of fidelity. We define three levels of representation fidelity for gestures with respect to hand pose and motion. Low-fidelity representations are gesture labels annotated by humans (e-g. "hammering"), while high-fidelity representations are numerical values of hand joint positions or motion trajectories (e.g. trajectory of the hand center in 3D space) that usually are output representations of automatic hand detection systems. Mid-fidelity representations are human textual descriptions of hand shapes hand motion (e.g. fist moving up and down). We evaluate LLM's ability to reason at different fidelity levels of gesture representation with a focus on textual representations (mid- and low-level fidelity, We initiliazed GestureInstruct using low-fidelity gesture class labels, and then populated them with mid- fideliilltty textual representations by describing the hand shape and motion of the gesture in detail.
GIRAF can reason about gestures with textual representations of different levels of fidelity. Fig. 3 shows the results of GIRAF reasoning with both gesture labels (low-fidelity) and gesture descriptions (mid- mnidyfddeitty). GIRAF able to achieve 75% success rates on deictic gestures and about 50% success rates on both iconic and semaphoric gesture. Semaphoric gestures are one-shot generalization scenarios (human defines what do in speech once and the robot needs learn to do that again next time without speech) so we expect LLM to perform the same with either type of gesture representation. Unexpectedly, the LLM planner completely fails to correctly reason about symbolic gestures when we use gesture descriptions while ChatGPT is able to describe these gestures in our preliminary experiments. Further investigation is needed to understand this behavior.
Discussion
Summary. In this work, we propose the GIRAF framework that incorporates gesture information for multimodal instruction following in human-robot interaction settings. We demonstrate instantiations of GIRAF enable more-favorable and better-performing robots (70% higher success rate) than language-only methods with a user study in two table-top manipulation tasks. We also evaluate GIRAF on the GestureInstruct dataset and show that it is able to reason about a diverse set of gestures.
Limitations and Future Work. In this work, the instantiation of GIRAF can only handle static gestures. While we show our reasoning module can handle dynamic gestures, we lack a model that can robustly detect dynamic gestures. One can build a specialized dynamic gesture detector through collecting supervised training data or leverage VLMs. While the VLMs we had access to do not take multiple images as input, we expect video-based VLMs can describe human dynamic gestures in the near future. As for the types of gesture descriptions, in our preliminary results we found that GIRAF cannot achieve non-zero performance with nigh-fidelity numerical representations of gestures (i.e. hand joint positions, 3D motion trajectories). Our conjecture is that existing L.MMs lack the level of geometric reasoning required to understand complex numerical representations of gestures. We leave full investigation of this for future work. For similar reasons, GIRAF as a framework still cannot solve tasks that require complex reasoning about the motion of the gestures, such as manipulative gestures (e.g. human user demonstrates a task and says "do this"), which is an interesting open problem for future work. Finally, we focus on hand gestures in this work while human full-body gestures could also be informative and useful for human-robot interactions, which we consider as promising future extensions of GIRAF.
8

---

Acknowledgments
This work is supported by NSF #2006388, #2132847, #2218760, ONR, AFOSR, and DARPA YFA.
References
[1] S. Waldherr, R. Romero, and S. Thrun. A gesture based interface for human-robot interaction. Autonomous Robots, 9:151-173, 2000.
[2] O. Mazhar, S. Ramdani, B. Navarro, R. Passama, and A. Cherubini. Towards real-time physical human-robot interaction using skeleton information and hand gestures. In 2018 IEFE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1-6. IEEE, 2018.
[3] Neto, M. Simão, N. Mendes, and M. Safeea. Gesture-based human-robot interaction for human assistance in manufacturing. The International Journal of Advanced Manufacturing Technology, 101:1 19-135, 2019.
[4] J. DelPreto and D) Rus. Plug-and-play gesture control using musele and motion sensors. In Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, pages 439 448, 2020.
[5] A. Madaan, S. Zhou, U. Alon, Y. Yang, and G. Neubig. Language models of code are few-shot commonsense learners. arXiv preprint arXiv:2210.07 2022.
[6] Wei, X. Wang, D. Schuurmans, M. Bosma, Chi, Q. Le, and Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022.
[7] A. Talmor, O. Yoran, R. L. Bras, C. Bhagavatula, Y. Goldberg, Y. Choi, and J. Berant. Common- senseqa 2.0: Exposing the limits of ai through gamification. arXiv preprint a2X1v:2201.05320, 2022.
[8] S. Dídac, S. Menon, and C. Vondrick. Vipergpt: Visual inference via python execution for reasoning. arXiv preprini arXiv:2303.08128 2023.
[9] I.. Jiang, J. D. Hwang, C. Bhagavatula, R. Le Bras, J. Liang, J. Dodge, K. Sakaguchi, M. Forbes, Borchardt, Gabriel, et al. Can machines learn morality? the delphi experiment. arXiv e e-prints, pages arXiv-2111 2021.
[10] Schramowski, C. Turan, N. Andersen, C. Rothkopf, and K. Kersting, Large pre-trained language models contain human-like biases of what is right and wrong do. Nature Machine Intelligence, 4(3):258 25, 2022.
[11] D. Hendrycks, C. Burns, S. Basart, A. Critch, J. Li, D. Song, and J. Steinhardt. Aligning ai with shared human values. In International Conference on Learning Representations.
[12] M. Kwon, S. M. Xie, K. Bullard, and D. Sadigh. Reward design with language models. arXiv preprint arXiv:2303.00001 2023.
[13] H. Hu and D. Sadigh. Language instructed reinforcement learning for human-ai coordination. In 40th International Conference on Machine Learning (ICML), 2023.
[14] H. Ha and S. Song. Semantic abstraction: Open-world scene understanding from 2D vision-language models. In Proceedings of the 2022 Conference on Robot Learning, 2022.
[15] T. Vuletic, A. Duffy, L. Hay, C. McTeague, G. Campbell, and M. Grealy. Systematic literature review of hand gestures used in human computer interaction interfaces. International Journal of Slaman-Compinn Studies, 129:74-94, 2019.

---

[16] C. Matussek, L. Bo, L. Zettlemoyer, and D. Fox. Learning from unseripted deictic gesture and language for human-robot interactions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 28, 2014.
[17] Y. Chen, Q. Li, D. Kong, Y. Kei, S.-C. Zhu, T. Gao, Y. Zhu, and S. Huang. Yourefit: Embodied reference understanding with language and gesture. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1385-1395, 2021.
[18] D. Whitney, M. Eldon, J. Oberlin, and S. Tellex. Interpreting multimodal referring expressions in real time. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 3331-3338. IEEE, 2016.
[19] W. Huang, Abbeel, D. Pathak, and Mordatch. Language models zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pages 911 9118-99 8-9177. PMLR, 2022.
[20] W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang. P. Florence, A. Zeng. J. Tompson, Mordatch, Chebotar, P. Sermanet, N. Brown, T. Jackson, L. Luu, Levine, K. Hausman, and Ichter. Inner monologue: Embodied reasoning through planning with language models. In arXiv preprint arXiv:2207.05608 2022.
[21] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakr- ishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Lu, L. Luu, C. Parada, P. Pastor, Quiambao, K. Rao, J. Rettinghouse, D. Reyes, P. Sermanet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, Xia, T. Xiao, P. Xu, S. Xu, M. Yan, and A. Zeng. Do as can and not say: Grounding language robotic affordances. In arXiv preprint arXiv:2204.01699 2022.
[22] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Text2motion: From natural language instructions to feasible plans. arXiv preprint arXiv:2303. 12153, 2023.
[23] Liang, Huang, Xia, P. Xu, K. Hausman, B. Ichter, Florence, and A. Zeng. Code as policies: Language model programs for embodied control. In arXiv preprint arXiv:2209.07553. 2022.
[24] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. arXiv preprint arXiv:2209.11300 2022.
[25] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. B. Girshick. Segment anything. ArXiv, abs/2304.02643, 2023.
[26] G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. Openclip, July 2021. URL
https://doi.. org/10. 110.1..5211/11 /281/2enod/ 5143773.
[27] Microsoft. Microsoft azure, 2010. URL /tttss///zzree.e mi www.ooooft com/en-us// Ac-
cessed June 4th, 2023.
[28] F. Zhang, V. Bazarevsky, A. Vakunov, A. Tkachenka, G. Sung, C.-L. Chang, and M. Grundmann. Mediapipe hands: On-device real-time hand tracking. ArXiv, abs/2006. 10214, 2020.
[29] Y. Zhang, C. Cao, Cheng, and H. Lu. Egogesture: a new dataset and benchmark for egocentric hand gesture recognition. IEEE Transactions on Multimedia, 20(5): 1050. 2018.
[30] P. Sundaresan, S. Belkhale, D. Sadigh, and J. Bohg. Kite: Keypoont-connditioned policies for semantic manipulation. 17Xiv:2306.1660, 2023.
10

---

[31] X. Chen, X. Wang, S. Changpinyo, A. J. Piergiovanni, P. Padlewski, D. M. Salz, S. Goodman, A. Gryener, B. Mustafa, I.. Beyer, A. Kolesnikov, J. Puigcerver, N. Ding, K. Rong, H. Akbari, G. Mishra, L. Xue, A. V. Thapliyal. J. Bradbury, W. Kuo, M. Seyedhosseini, C. Jia, B. K. Ayan, C. Riquelme, A. Steiner, A. Angelova, X. Zhai, N. Houlsby, and R. Soricut. Pali:
jointly-scaled multilingual language-image model. ArXiv, abs/2209.06794, 2022.
[32] W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi. In- structblip: Towards general-purpose vision-language models with instruction tuning. ArXiv, abs/2305.06500 2023.

---

Appendix
Table of Contents
A Gesture Types
12
B GestureInstruct Dataset: Task Scenarios
12
C User Study
15
C,I. Definition of Success
15
C.2 Instructions
15
C,3 Survey Questions
15
C.4 Rollouts
16
Motion Primitives
E Prompting LLM
F VLM for Grounding Gestures
G Realistic Use Cases
18
H Demos of Additional Use Cases
18
Failure Modes
18
Influence of Distance on Referent Detection
21
A Gesture Types
We reference the gesture classification in Vuletic al. [15] for gestures in HRI and adopt the gestures types that are most relevant and merged a few types that are tightly related and makes no difference for detecting purposes. E.g. we merged metaphoric gestures with iconic gestures as it is defined as iconic gestures for abstract concepts. We summarize the gesture classes in Table 1. Below, we list example gestures of each type in Table
Symbolic: thumbs-up for OK; rubbing index finger and thumb mean money.
Semaphoric: gestures used in sign language; gestures used for commanding animals.
Pictographic Iconic: gesture circle mean round object.
Spatiographie Iconic: gesture up and down and emphasize the object's 's location.
Kinematographic Iconic: rolling hand motion to refer rolling object.
Metaphoric Iconic: a cutting gesture to indicate a decision has been made.
Deictic: pointing gestures used indicate object; palm-open indicating desire to receive something.
B Gesturelnstruct Dataset: Task Scenarios
We generated total of 36 different gesture-speech instraction-following task scenarios and list them out in Table 2 and Table 3. We do not distinguish the sub-types of gestures under iconic gesture.
12

---

Table 1: Classification of Communicative Gestures
Speech Dependency
Gesture Type
Definition
represent object or concept. have conventional meaning and can directly translated into words used to trigger a predefined action, defined in formalized dictionary, developed for specific purpose represent meaning closely related the semantic content of the speech, illustrate what is being said
Symbolic
independent
Semaphoric
Pictographic
represent shape
Iconic
Spatiographic
represent spatial relation
Kinematographic
represent action an object
dependent
Mctaphoric
represent abstract concepts
Deictic
refer entity that is being said
To call that LLM reasons successfully task scenario, it needs to call the correct action primitive with the correct arguments in the generated plan. Take the fist gesture in semaphoric category as an example. The goal state is "robot moves to the hand position," so it is considered a successful trial if the LLM generated code looks like this:
Instruction 0: move over here Gesture: fist
target_pos detect_ pitrect_hadd_cente__pooo move.gripper ttctipppertt_p_se_e___ (target_pos)
Since Gesturelnstruct is designed solely for the purpose of evaluating if LLM has zero-shot reasoning capability for these types of gestures, took top-down approach of exemplifying gesture types when constructing dataset. However, a bottom-up approach of collecting suite of gesture-languae instructions from real applications would be preferred in practice.
Table 2: Gesturelnstruct Datasct part 1: Symbolic and Semaphoric Gestures
Gesture
gesture
label
description
gesture
languuge instruction
context
intent goal state
thumbs up
thumb extends out and points upward
robot gripper above cup hundlc robot gripper open and above cup hundle robot gripper open and ahove coffee pod moving towards the person
rohot picks up the cup
fingers
Symbolic
thumbs
ihumb extends out and points
robot asks human how to make corrections
down while other fingers curl inward
OK
thumb and index
forms
robot picks up coffee pod
circle while others extend out
stop
an open palm faces outward
robot stops
open palm inward,
backoning
and the whole hand moves
robot far away person
robox moves towards person
inward and outward repestedly
Bist
clussed palm
rabox hund position
Semuphoric
up
an open palm first faces upward.
fingers inward
grasp
robot picks up the coffee pod
closed palm first faces downward,
release
drop
robot coffee
all lingers extend out index finger exiends out while
cireling
harizontally
others curl inward, and the whole hand
turn around
robot turn around
circalor motion herizonrally
13

---

Table 3: Gesturefnstruct Dataset- part 2: Iconic and Deictic Gestures
Gesturr Type
gesturm lubel
geduue deseciption
Instruction
langnage
context
Intent/ goal state
hands circle
give bow) shaped
roundd bow!
nissuring with two fingers
thumb indes finger estend out while others curl inward
add this much water
robe is booding a water Jag
robor water cap
height simtlizz to the size
between humant's two fingers
pinching
thamh and index finger estend out while
others car insoert find and then thame-
do this
robor grtpper open and shove coffee pod
robot picks the ooffiee
index, laper
thunh
Lamicc
spresaling
touch other while others
iswand and then
do
robot huiling a coffie pool
robot velesacs the coffoe pad
and index figger seperate
teising
thumb and intex finger estend on
while others curl laward,
do this
robot gripper bottle of water
robot owass the bottle to open
und the while hand minecs
pushing
open
do
gripper wendeen black
block
lifling
and the whole hand moves upword
an open palm upward.
do this
nibor holding a bottle water
robor boole water off the ground
Iages gently car immrra wlnll dhmmb
squeesing
the other side liss, and thes
firgers mave clocr closed cloard pain.
do flis
and gripper abore
gobot bodting lemon
rubut squceuus the lemon
hammering
whole hund moves up down
looking for toot
and wirecuter on
robor over hammer
inlex und milili: linger exteral
cattngg
and indes and middle finger
while others cuil isward,
am looking for u tuel
nboo sces humnner, accewaivves and winecutter on the rible
niuoi hand over the winsculler
repearedly touch each other and separate
hammering
and
the
the tool that this
screwdrweer
roboc hand humner
lnner
cutting
and index, and moddle finger repearedly touch each other and separate
while others insurd,
give ilk tool that does this
néuut sccs hammeez anceddiverr and winecutte on the table
mobot hand ever the winccateer
azzwing
closed puin, and the whole hand retatess
give me the tool thai dasa this
nobor sees hummer, screediiver, and winccutter the tuble
robot hand over the screwdliver
cirllng varically
ethers
index firger extends while
again
just sguan: piece poper
rebet
equer
moves in ciecular motion vertically
smoking
Thunb and other tngers torn curved shape, and the whole hand is noar the mouis
bring aee this
robor varsous objects on the ubble
nubox hand over eiganese
purting beanie
sides head.
cloeal puine
hands
give
rohot saos horeic and exp
robot
downoord
opening a
two open palns face otter,
then they sooely move spart and bwm face apward
bring me
robot vations objocts on
mboo over buck
drinking
fngges gently cun immorrt wtiil thunhh on the other side, and the while hand five facos upwaed and then fila isvaad
need this
mbor sees varons objects on the kitches counter
zobot hand over a drink
geatly cur loward while side,
opening
while
can you help me with
robot opens
nutwark and rotutas
pointing
(obyect)
while whhers curl irwand
indes dinger extends out
pick up this water juy
robot gripecr is cappy
robot picks up the wakee jug
(becrion)
poiiting
while
index Bniger eslends our
ever hone
robot holding water
robor place dowa the water
locasion
Diectic
posstng
litk bit way
rolot bolding water jug
pointal
puinting
indes linger cxlends out
swaay
ninu mulliple objests
mbss sway the referma
(muttigle objects)
while wheer curl mwand
this, mis and this
on diny table
Heme inoo trast can
handorer
a6 ooen palm facss upward
plase it oive here
nobon has bool hand
robon hand over too)
wodhingg an object
and the whole hand
open palm laces dowpward.
oljest
pick up ttis
zubor gripper
robot picks up the louchad ebject
drawing
finger extends while others
deaw
robot hokding pencil
piezao poper
stops poper boboc trash into the trsh con
robre drows similor
circling
Index tinger exitods while
thro
robor gripper embpy
(dysomic pontting)
curl irwand and the whole hand moves
into the trach can
14

---

C User Study
C.1 Definition of Success
As shown in Fig. 2, we consider two types of success rates in this user study:
1. Success rate:
This refers to the planning success rate. We count a test successful if the robot finds the correct object (e.g. drawer, tool) that the user wants within trials. This measures how good the task planner and referent detection heuristic is, which the focus of this work.
2. Success rate (execution):
This refers the full task success rate. We count a test successful the robot can complete the task (e.g. open the correct drawer, fetch the correct tool) within 3 trials. This measures the performance of the whole system, including the robot execution.
C.2 Instructions
We provide the instructions we present to the user below:
In this study you will interact with robot arm to complete different table-top manipulation tasks and each in different scenarios. The study will take approximately 45 minutes complete. Please open the survey and fill out the first section before continuing.
Task: Open Cabinet Drawers In this task, you will ask the robot to open a few drawers of the cabinet for you. Pick 3 drawers (within the highlighted area) you want to ask the robot to open and mark them this image (mark the order you want to have them opened): <image> You will repeat this task in two different scenarios:
Scenario M: You use language to describe which drawer to open.
Scenario N: You use both language and pointing gestures for specifying the target drawer to open.
When you are ready to initiate a task, step once on the foot pedal and issue an instruction for the robot. The robot will confirm your instruction by repeating what it heard. You have trials for each task (speech detection failure does not count as trial). Please fill out the corresponding survey questions after each scenario.
Task: Fetch Tools In this task, you will ask the robot to fetch few tools for you. Pick 3 tools (randomly) for the robot to fetch and mark them with the order you'd 'd like to have them in this picture: <image> You will repeat this task two different scenarios:
Scenario M: You use language to describe which tool to fetch.
Scenario N: You use both language and pointing gestures for specifying the target tool to fetch.
When you are ready to initiate a task, step once on the foot pedal and issue an instruction for the robot. The robot will confirm your instruction by repeating what heard. The robot will try to hand the tool over to you. You have 3 trials for each task (speech detection failure does not count as trial). Please fill out the corresponding survey questions after each scenario.
C.3 Survey Questions
For each task, the user fill out the following survey for each scenario by rating how much the agrec with each statement:
The robot is able to understand my instruction.
The robot able to identify the correct item wanted.
The robot able to successfully execute the task.
It is easy to communicate with the robot what wanted.
I feel frustrated after interactions with the robot.
15

---

R: this the
Hi Open this drawer.
H: Yes.
T1. Open Drawers
H: Fetch that silver thing.
this the correct
R: Here you are.
H: Yes.
T2. Fetch Tools
X
Figure User Study Task Rollouts
I trust the robot for completing tasks specified for it.
I would use this robot again in the future for similar tasks.
C.4 Rollouts
Example trajectorics the two user study task are shown in Fig. 5. We record all the task specification language and RGB-D images the scene from 3 calibrated camera views and will release this data on our website well.
D Motion Primitives
We adopt the method from Sundaresan al. [30] and implement library of parameterized motion primitives for the tasks in our experiments. All of these motion primitives are parameterized with 3D position in the robot's reference frame. For example, the primitive for picking up object takes the center point of the object as parameter and the placing primitive takes the target location parameter. The depth cameras are fixed and calibrated so that the detected object positions can be mapped to the robot's reference frame. For simple pick-and-place motions, we hand code motion primitives that command gripper to move to the input target position with a fixed (upright) orientation. For complex manipulation motions such as opening the drawer and picking up non-concave objects (tools), use waypoint-bol collected supervised gripper pose data conditioned de-projected from 2D point annotated by the user. At test time, the detected object centers output from the scene descriptor will be used as input to these motion primitive models.
motion primitive models. To train each motion primitive model, pointcloud input and 3D point the object
E Prompting LLM
We prompt LLM for tasking planning, including reasoning about what to perceive. We also implement a language program to filter target objects. For our basline implementation, we also prompt the LLM to reason about spatial relationships between objects with examples. Full prompts used in our experiments can be found on our project website: tinyurl. www.coo///...o /mm/giraf23
16

---

Input Image with Ground Truth Label
5th 2nd column
4th 5th column
5th 5th column
Prompt
Counting from the top left corner, the drawer pointed by the user which and which column?
PaLI
1st row, 2nd column
1st row, 2nd column
1st row, 2nd column
InstructBLIP
right
right
right
The drawer being pointed by the the 4th and 6ith columnn counting from top left corner.
The drawer that the user is pointing the and 4th column counting from the top left corner
drawer being pointed by the is on the 5th and 8th solumn counting from the top left corner
ChatGPT4-V
T1. Open Drawers
Input Image with Ground Truth Label
middle screwdriver
right-most T-handle key
Prompt
Which tool is the user pointing to?
PaLI
screwdriver
peghoard
pegboard
InstructBLIP
hammer
hammer
hammer
ChatGPT4-
The user is pointing to screwdriver.
The user is pointing to the hammer the pegboard.
The user pointing to a pair of needle-nose pliers.
T2. Fetch Tools
Figure VLM referent detector. Examples of running referent detection with PaLI [31], InstructBLIP [32]. and chatGPT4-V demonstrate that existing VLMs still struggle with fine-grained geometric reasoning.
VLM for Grounding Gestures
We investigate VLM's 's capability of both 1) grounding gesture class and 2) detecting the referent of pointing gesture. We test 3 state-of-the-art models: PaLI [31], InstructBLIP [32]. and chatGPT4-V.
VLMS take the full scene image as input and therefore have more context reason about the gesture type than an end-to-end gesture classifier trained with only hand keypoints features and therefore has the potential to replace data-driven gesture classifiers. We prompt InstructBLIP [32] to classify static gestures in the GestureInstruct dataset and it can achieve around 80% zero-shot accuracy.
17

---

However, VLMs are not good at predicting referent of the pointing gesture, shown in Fig. 6: none of the three models can output the correct drawer instance the open drawers task as it requires complicated geometric reasoning: chatGPT4-V struggles to identify the exact location the drawer, Pall gets the answer format correct but outputs same answer for different input images, while InstructBLIP does not even output reasonable answers); all three models can predict reasonable output in the fetch tools task (chatGPT4-V and PaL. generating plausible responses in all cases) but lack the sensitivity for precise referent detection.
G Realistic Use Cases
While we designed the tasks for user study to highlight the benefits of combining gesture under- standing when following language instructions, believe the problems GIRAF addresses broadly exist in many realistic scenarios. For example, the Fetch-Tool task represents tasks where the human user does not know the name of the target item or the robot cannot detect the objects the human recognizes them. From the user's end, this is common when English is not their native language. From the robot's 's end, this may very well happen when it encounters rare objects or objects with unusual appearance. the user could realize the robot doesn't understand what they are referring to, they would want to point the target instead.
H Demos of Additional Use Cases
We show additional use cases of GIRAF including (1) using multiple gestures in single instruction: opening multiple drawers (Fig. 7), and picking and placing a tool (Fig. 8); (2) specifying grasp point on an object (Fig. 9); and (3) instructing the robot perform long-horizon sorting task (Fig. 10).
"Please open this and that "
pointing
"Can you open these two boxes?"
pointing
"'Oen this drawer, this one, and that one."
pointing
Figure
Open multiple drawers at once.
Failure Modes
We summarize error modes of each module the system below. We generally take two measures to account for different enror modes. The first one is to make the system as transparent as possible so we can catch the error before it propagates to the next level (e.g. robot will announce the action plans to take and confirm the target with the human user before it takes any action). The second measure is to catch exceptions in the code and detect simple failures (e.g. syntactic errors and grasp failures). For code error, the robot will announce its failure and ask the user to try again, for grasping
18

---

"Pick up this tool and place over here."
pointing
"Now pick up this wrench and place it here."
A
pointing
Figure 8: Fetch and place tool. "Pick up this toy over here."
pointing
"Use this point pick up the elephant."
pointing
Figure 9: Pick up object from a specific point.
failure the robot will adjust its motion primitive parameters and try again. While we do not directly tackle low-level action primitive failures, our system allows the user to fix low-level execution errors by incorporating directional pointing gestures. In of long-horizon tasks (Fig.4 in the paper), we show that the robot missed the grasping point of the mug handle by bit and the user point in the direction they want the robot to move in order to fix it.
1. Human description:
(a) Speech (Azure API [27]):
Misrecognized words:
This happens more often for non-native English speakers than the native as observed in user study. On average, this happens to native speakers 0.27 times per instruction and non-native speakers 0.46 times per instruction. When the user's instruction is misrecognized, we ask the user to give the instruction again, so this error would not propagate into the system.
(b) Gesture:
On average, gesture is not detected 0.07 times per instruction, and we further discuss different reasons of failure below.
Hand detector (MediaPipe [28]):
Hand not detected:
This happens more often when the image is taken from the side the hand instead of the back or front of the hand. When this happens, we ask the user to give the instruction again, so this error would not propagate into the system. Incorrect hand keypoints:
For example, might label the joints of the thumb as joints of the index finger. This happens more often when some hand joints occluded in the image, such as doing a fist gesture, and if this happens, we would still feed those keypoints
19

---

"Tick up that towel."
"Throw it into that basket."
pointing
pointing
OK
"Did find the basket?"
"Now pick that towel over there."
"Hand it me."
pointing
handover
Figure 10: Long horizon interaction with multiple gestures.
(a) More! Less Detected Objects.
(b) Incorrect Object Labels.
Figure 11: Scene Descriptor Failure Modes.
into the gesture classifier because don't 't have the groundtruth to verify whether the keypoints are correctly labeled. This kind of eπror might hurt the accuracy of the gesture classifier and referent detection module because both of them directly take hand keypoints as input instead of raw images.
ii Gesture classifier
Predict unknown gesture:
In this case, we would treat it as no gesture detected and ask the user to give the instruction again, so this error would not propagate.
Predict another gesture:
For example, the gesture classifier might predict pointing gesture as fist gesture. In this case, we would still feed the predicted gesture to the reasoning module as we don't have the groundtruth. This kind of error can hurt the performance the LLM task planning module, For example, when the user says "move over here" while the gesture classifier predicts pointing gesture as a fist gesture, LLM would tell the robot to to the hand center instead of moving to the pointing location.
2. Scene description:
More/ less detected objects:
As shown in the orange regions in Fig. 1aa the scene descriptor might predict more or less drawers than the actual number, but this generally not problematic in our experiments because users don't usually point those misclassified objects.
Incorrect object labels:
Another case of failure is when the scene descriptor gives objects wrong labels. This would hurt the performance of the referent detection module it would apply
20

---

semantic filter to just consider a subset of objects given object labels and the user's s instruction. For example, when the user says "pick up the leftmost screwdriver," the referent detection module would only consider the two orange points in Fig. 11b that are labeled as screwdrivers to be candidates that the user might point at. Therefore, even the user is pointing at the leftmost screwdriver, our system would think the middle one is the one being pointed at.
3. LLM task planning:
Bugs (semantic and logic errors generated code):
Bugs happen mainly because of its confusion about Python list and Numpy array. As result, the generated code might try to call Numpy array function on a Python list, resulting in an inexecutable piece of code. Bugs would also be generated when the language instruction is out-of-distribution. One example during our user study when the user said "on the third row take the third drawer." Since differs from the sentences put in the prompts too much, the generated code tries to feed more arguments into the action primitive. Specifically, the action primitive for opening a drawer takes one argument of the drawer's 3D position, but in this example, the generated code wants to feed the function with an additional 3D position corresponding to the third row. On average, bugs happen 0.02 times per instruction.
Incorrect gesture interpretation:
LLM might also fail understand the meaning of some gestures. For example, LLM fails call the correct function aorigate_folloving...u () given the gesture "beckoning."
4. Referent detection:
On average, referent detection fails 0.2 times per instruction. Besides being affected by accumulated errors from other components, we also observe another case of failure that is related to user's behavior.
User's 's different behavior (unreliable pointing):
We observed one interesting indicator of their attention and still rely on language to provide the accurate description. For example, one user gave the instruction "open the drawer on the second row and second column" while vaguely pointing toward that direction. Therefore, for this kind of users, our heuristic fails more often because their pointing gesture might not actually point at the object, but we also found out these cases happen more at the start of their studies they were able to adapt through a few trials.
where
tend gestures vague
system by pointing more accurately
Influence of Distance Referent Detection
Close
Neutral
Far
Compuici
Newtrol
Sparse
Figure 12: Setup Testing Referent Detection Accuracy.
21

---

Table 4: Referent Delection Accuracy under Different Setups.
Close
Neutral
Far
Compact 2/3
2/3
2/3
Neutral
2/3
3/3
3/3
Sparse
3/3
3/3
3/3
To further test how our referent detection heuristic works as the user standing further away. we ran small experiment varying the user's distance to the objects and how close the objects are. The setup shown in Fig. 12. As shown Table 4, our heuristic can still work even when the user points from further away, and as the distance between objects becomes larger, the accuracy of our heuristic improves because it is less likely be affected by noise in the detected hand keypoints.
22


Reviews:
[{"review_id": "j1urWvruiz", "reviewer": "Reviewer_beCJ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper’s goal is to enable robots to recognize gestures and contextualize them using real time language instructions from humans. The challenge here is that the intent behind a gesture strongly depends on the context (pointing at a cup can mean pour into the cup or pick up the cup, but you don’t know which one the human wants without more information than just the gesture). Similarly, language on its own can be ambiguous without gestures (\"open this drawer\" is very unclear). They propose a framework that leverages LLMs to contextualize the meaning of human gestures and infer human intent from them. The idea is to have a model for identifying the gesture, then prompt an LLM for reasoning about the gesture-language pair and generate a robot policy from that. The paper demonstrates that the proposed method, GIRAF, is preferred by users in a user study when compared to a language-only baseline, and also curates and publishes a dataset of gestures that can be useful for future research.", "summary_of_recommendation": "I really like this paper and I think it would be a wonderful contribution to CoRL! I think the ideas presented are novel, the multimodal framework could potentially be very useful in the embodied robotics community, and the gestures data set would be helpful for future researchers. The paper is somewhere between a weak and a strong accept for me, mostly because of the 2 issues above I would like to see addressed.\n", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "For the rebuttal, please discuss the gesture ambiguity issue. Can you either have a small experiment on that or discuss why the method should or should not work on it? Please also add a discussion on the situations where the method applies to demonstrate that this is not a niche problem. Lastly, please fix the grammatical errors; the paper is rife with them.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, I thought this paper was very well-written and enjoyable to read! The framework is sound, novel and clever; the experiments are thorough and compelling.\n\nI have 2 comments that I’d like to see addressed.\n\nFirst, a strong motivation for the paper is that gestures can be ambiguous (e.g. pointing at a cup could mean “pick up the cup” or “pour in this cup”), yet looking at Fig. 3 every gesture has only one corresponding language utterance. I would’ve liked to see language resolving gesture intent ambiguity, yet as far as I can tell that isn’t explored. Essentially, the user study shows that gestures are helpful for disambiguating language, but the intro also motivates the other direction: how language can be useful to disambiguate gesture intent. Have you tried an experiment like this, where you measure success rate for disambiguating between multiple meanings of the same gesture?\n\nMy second comment is that I’d like to see more discussion on the types of situations where the human knows how to use an object yet can’t name it. In the “open this drawer” it is clear why using gestures is important: the human clearly could say “open the 3rd drawer on the 2nd row”, but that is cumbersome and gesturing helps with that. But in a situation like “give me this tool”, when would a human not know how to name the tool yet know how to use it? What prevents the human from saying “give me the screwdriver”? I think it’s really important for the applicability of the method to have some discussion on when these kinds of situations emerge, otherwise the reader might be left feeling like this problem is a little bit manufactured.\n\nSmall comments:\n- Abstract typo: “deictib” should be “deictic”\n- Intro typo: “a pointing” should just be “pointing” \n- Fig. 3 caption “gestuers” → gestures\n- Discussion: “langauge” → language\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "TW8I_BslCyt", "reviewer": "Reviewer_i78B", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes GIRAF, an LLM-based framework to include gesture information in instruction following. GIRAF first uses several descriptors to describe scenes (objects and their 3D positions) as well as human speech and gestures. It then uses an LLM-based planner to generate the task plan by prompting with the instruction and the gesture description. To execute the plan, the referred object is identified by intersecting the pointing ray with the candidate objects. The experiment on a real robot demonstrates that GIRAF provides a more effective communication interface and is preferred by humans. To evaluate if the LLMs can understand diverse gestures described in different fidelity, this paper additionally proposes a GestureInstruct dataset for evaluation.", "summary_of_recommendation": "It is important to include gestures in the communication modality as it has been demonstrated as effective by user studies. However, the evaluation in its current form cannot justify the benefit of using LLMs in gesture reasoning.\n\n-----\nPost-rebuttal update:\nI would like to thank the authors for updating the paper to address my concerns. I agree with the authors that the LLM provides context for disambiguating the gesture but I didn't see an experiment that tests how well the LLM helps in disambiguation, for example, just extracting the verb from language and combining it with an object pointed by the gesture can be a good baseline for comparison. In general, I think this is an initial study and a good resource for the community, but I would encourage the authors to include a discussion about the provided information from language (as in the table presented in the rebuttal) in the paper and evaluation on how well LLMs help disambiguate. I'm updating my score accordingly.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "As discussed in the weaknesses, the paper didn’t describe the role of LLMs and have comparisons with baselines (that also consider gestures) to demonstrate the effectiveness to include LLMs in the instruction+gesture reasoning. It is hard to understand how LLMs contribute to the task in addition to gesture classification and task planning as in CaP.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths \n- Incorporating gestures into instruction following tasks is very useful for designing effective human-robot interaction.\n- The robot experiment conducted in this paper demonstrates that users consider using gestures as an easy and preferred way to communicate.\n\nWeaknesses \n- It is unclear what the role of LLMs is from the description of the paper. It seems to me that the LLMs can interpret the same pointing gesture differently based on the language instruction and the history. However, there is no experiment to evaluate this context-dependent reasoning which left a question that if we need an LLM here or if it is just to help to parse the instructions.\n- The perception API identifies the object/location the user is pointing to. Then a simple approach can be just extracting the action (i.e. verb) referred to in the instruction and then completing the action with the object intersecting with the pointing ray. How does the proposed LLM-based approach compare to a baseline like this or other prior works that also consider pointing gestures?\n- The evaluation using GestureInstruct only shows that LLMs can recognize gesture descriptions. However, it doesn’t mean that the classified gestures can be integrated into task planning as the pointing gesture. It will be helpful if the paper can demonstrate how an LLM can take diverse gestures into consideration for task plans.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "R0cXw27UDt", "reviewer": "Reviewer_zpDJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents an approach to prompt LLMs for robot planning conditioned on language instructions and gestures. It first uses off-the-shelf perception models to parse the scene and recognize human gestures, and then prompts LLMs with both the scene and gesture descriptors (as code/templates) as well as the language instructions to generate task plans. It then calls low-level controllers to execute the plans. Evaluation on a real robot shows promising results. ", "summary_of_recommendation": "This paper presents an interesting idea to use LLM for gesture & language-guided robot planning. There are concerns about safety and robustness to errors made by individual modules. My final rating will depend on how authors address these concerns.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Please address the two concerns in the review. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n1. The idea is technically solid and straightforward.\n\n2. The writing is clear and the videos are helpful.\n\n3. Analyses and discussions are quite informative.\n\nWeaknesses:\n1. The method involves combining multiple existing models. It would be good to estimate errors from each model and how errors can be accumulated in difficult cases. E.g., what will happen if perception models fail?\n\n2. It would be good to discuss what type of tasks could be solved by this system with or without gestures. I imagine that some tasks can be directly solved by instruction following (language only) without gestures, while others would have to rely on accurate gesture recognition. And there could be tasks that simple gestures are not sufficient. \n\n3. LLMs-based planning does not inherently have a safety guarantee. How would you address failures and potentially unsafe plans generated by LLMs?", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "7xp6lKUYb9", "reviewer": "Reviewer_h9cL", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a system investigating hand gestures for enhanced human-robot interaction by incorporating off-the-shelf components from prior work. The core contribution appears to be demonstrating the value of gesture information for HRI through the proposed system, rather than introducing novel technical methods.", "summary_of_recommendation": "The paper investigates an interesting use of gestures for human-robot interaction. However, the experiments conducted so far have been shown in contrived setups. It remains unclear how well the system will function when human subjects are far away from the object, where gestures are more important in practice. Additionally, the paper only has limited real-world experiments demonstrating how the system works with other gestures, and the limitations are not thoroughly discussed.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "* Regarding the drawer opening task, does the depth camera perform well for these small transparent drawers? I assume that the estimated positions of the drawers will have higher errors. If yes, how did the authors obtain good estimates?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Strengths**\n\n- The paper proposes a \"relatively\" complete system for identifying gestures and human language, and feeding this information into the LLM planner and some robot motion execution primitives.\n- A user study is included in the paper, making the evaluation more objective.\n- The whole system is pretty straightforward.\n\n**Weaknesses**\n\n- Although the paper reports quantitative data, there is not enough clear context provided for each number. For instance, what does the success rate represent? How many total tests were conducted? This lack of clarity applies to many parts of the paper, such as Figure 2, 3, or the success rate on the GestureInstruct dataset.\n- From a high-level perspective, humans typically use gestures when they are at a distance from objects. However, in the examples shown in the paper (drawer opening, tool fetching), the human operator is very close to the objects, to the point that there isn't really a need for robots to do the task. Humans can do them themselves. Therefore, these tasks appear somewhat contrived. To make the approach more compelling, it would be beneficial to show examples where human subjects stand far away from the object, and the robot can still identify the human gesture and where the fingers are pointing at, and complete the task. It might be the case that when humans are far away from the object, it would become more challenging to correctly identify the gesture and where the human is pointing at, and hence, the proposed pipeline might not work well.\n- The system has many components, and it is likely that each component can have its own failure modes. For instance, when does the heuristic-based referent searching fail? Unfortunately, there is limited discussion on these failure modes at present.\n- It seems that there are no physical experiments for gestures other than \"pointing\"?", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Summary of the paper:

The paper presents GIRAF, which enhances human-robot interaction by integrating gestures with language instructions. GIRAF leverages Large Language Models (LLMs) with off-the-shelf perception models to interpret scene descriptors, recognize human gestures, and generate task plans accordingly. Evaluation on a real robot reveals the potential benefits of this method. Additionally, the authors introduces the GestureInstruct dataset, designed to examine the ability of LLMs to understand and categorize diverse gesture descriptions.

Strengths:
- Novel Integration: The paper's main strength is combining gestures with language instructions to design effective human-robot interaction.
- Practical Implications: The robot experiment underlines that gestures are an intuitive and preferred method of communication for users, potentially reshaping future HRI designs.
- Comprehensive System Design: The manuscript presents a holistic system, encompassing gesture identification, language processing, LLM-based planning, and execution of robotic tasks. A user study adds weight to its practical evaluation.
- Dataset Contribution: The GestureInstruct dataset is a promising tool for further research in gesture-based robotic instructions.
- Clarity & Resources: Reviewers appreciated the clear writing, informative discussions, and the helpful videos accompanying the paper.

Weaknesses:
- Role & Necessity of LLMs: Multiple reviewers expressed concerns regarding the clear definition and the actual need for LLMs in the context of gesture reasoning. Questions were raised about the LLM's contribution beyond gesture classification and task planning.
- Comparative Analysis: There's a perceived gap in comparing the proposed LLM-based method with other baselines, especially those which already account for pointing gestures. This makes it challenging to pinpoint the actual advantages of GIRAF.
- Evaluation Ambiguity: The quantitative results lack clear context regarding the metrics' meanings and the total tests performed, making it hard to interpret the evaluation's significance.
- Experimentation Limitations: Some experiments, like those involving only "pointing" gestures or where the human is very close to the object, seem contrived and don't necessarily reflect real-world scenarios.
- System Robustness: The approach's reliance on several existing models introduces potential error propagation. There are concerns about the robustness and safety of plans generated by LLMs, especially when individual modules fail.
- Contextual Nuances: The paper suggests gestures can help clarify language instructions, but there is a limited exploration on how language can clarify ambiguous gestures. Also, some presented situations, like "give me this tool," might seem artificially constructed without sufficient real-world applicability discussions.

Rebuttal and Discussion:

The authors responded to the questions from the reviewers thoughtfully and addressed several of the concerns raised. Furthermore, they submitted a revised manuscript reflecting the results of the discussion, and the quality of the paper improved with detailed data added to the appendix. However, not all concerns have been completely resolved, and the revised manuscript exceeds the page limit. Therefore, further revisions are required.

Recommendation:

Given the novel integration of gestures and language for HRI, GIRAF's proposition is promising. However, for a clearer evaluation of its significance and a broader acceptance in the community, it is imperative to address the stated concerns, particularly around the precise role of LLMs, comparative analyses with baselines, and the system's robustness in diverse scenarios.
Decision: Accept (Poster)