Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agents
Authors: Wenlong Huang
Fei Xia
Dhruv Shah
Danny Driess
Andy Zeng
Yao Lu
Pete Florence
Igor Mordatch
Sergey Levine
Karol Hausman
brian ichter
Venue: NeurIPS 2023 poster (2023)
Keywords: robotics
language models
embodied agents
TL;DR: We formulate a token decoding procedure applying large language model to robotics settings. Tokens are selected based on likelihood under the language model and a set of grounded models, such as affordance, safety, and preference functions.

Abstract: Recent progress in large language models (LLMs) has demonstrated the ability to learn and leverage Internet-scale knowledge through pre-training with autoregressive models. Unfortunately, applying such models to settings with embodied agents, such as robots, is challenging due to their lack of experience with the physical world, inability to parse non-language observations, and ignorance of rewards or safety constraints that robots may require. On the other hand, language-conditioned robotic policies that learn from interaction data can provide the necessary grounding that allows the agent to be correctly situated in the real world, but such policies are limited by the lack of high-level semantic understanding due to the limited breadth of the interaction data available for training them. Thus, if we want to make use of the semantic knowledge in a language model while still situating it in an embodied setting, we must construct an action sequence that is both likely according to the language model and also realizable according to grounded models of the environment. We frame this as a problem similar to probabilistic filtering: decode a sequence that both has high probability under the language model and high probability under a set of grounded model objectives. We demonstrate how such grounded models can be obtained across three simulation and real-world domains, and that the proposed decoding strategy is able to solve complex, long-horizon embodiment tasks in a robotic setting by leveraging the knowledge of both models.

Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agents
Wenlong Huang'; Fei Xia", Dhruv Shah Danny Driess Andy Zeng", Yao Lu", Pete Florence", Igor Mordatch", Sergey Levine Karol Hausman', Brian Ichter Stanford University, Google Deepmind, 'UC Berkeley
Abstract
Recent progress in large language models (LLMs) has demonstrated the ability to learn and leverage Internet-scale knowledge through pre-training with autore- gressive models. Unfortunately, applying such models to settings with embodied agents, such as robots, is challenging due to their lack of experience with the physical world, inability to parse non-language observations, and ignorance of re- wards safety constraints that robots may require. On the other hand, language- conditioned robotic policies that learn from interaction data can provide the nec- essary grounding that allows the agent to be correctly situated in the real world, but such policies are limited by the lack of high-level semantic understanding due the limited breadth of the interaction data available for training them. Thus, if we want to make use of the semantic knowledge in a language model while action sequence
still situating it in embodied setting, we must construct that is both likely according to the language model and also realizable according grounded models of the environment. We frame this problem similar probabilistic filtering: decode a sequence that both has high probability under the language model and high probability under set of grounded model objectives. We demonstrate how such grounded models can be obtained across three simu- lation and real-world domains, and that the proposed decoding strategy is able solve complex, long-horizon embodiment tasks in robotic setting by leveraging the knowledge of both models.
Introduction
Recent works have demonstrated robots that are increasingly proficient at understanding and act- ing upon natural language, whether through planning or conditioned policies. Complementing such progress, the field of natural language processing has recently seen large language models (LLMs) become ubiquitously used as pre-trained or few-shot prompted models, due to their impressive few- These LLMs have efficiently learned from web-scale
shot performance and vast knowledge-bel data through autoregressively modeling the probability distribution over text tokens and thus gener- ate text. However, the nature this process such that applying such models to embodied settings remains challenge. They have not interacted with their environment, lack observability of non- language observation modalities (c.g., images), and may not know what is safe or possible for a particular embodiment.
Determining how execute long-horizon behaviors based on high-level verbal commands one particular area of robotics where the rich semantic knowledge in large language models can be es- pecially useful. This problem combines elements of semantic reasoning and planning: the robot
Work done as an intern at Google.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

User
Task Plan:
Place blocks in mismatchoe color bowls.
1. Put green block the red bowl blue biock the
Large
green
"green"
Language
Model
yellow orange
Grounded Decoding
Grounded
green
I'd like a drink and snack.
Co the green block
Models
blue
purple
Grounded Decoding solves robotic tasks by taking an instruction as input and selecting tokens that have high probability under a Large Language Model (LLM) and set of Grounded Models (GM). Thus, leverages the open- vee-bvocaylaay and semantic knowledge of LLMs while being grounded in the environment and the robot's capabilities. Furthermore, the whole process does require expensive fine-tuning of the LLM.
Figure I:
must understand the instruction, determine the steps needed to fulfill it, and also determine how to sequence those steps appropriately given capabilities and the current state of the environment. However, this is not problem that can be solved purely with semantics, as it requires sufficient grounding to understand how the task should be performed in context - for example, in the example in Figure the language model alone has way of knowing which block to pick up because this requires knowledge of which blocks are present, and also what manipulations the robot is capable of performing them. Thus, although language model assign probabilities for how likely vari- ous steps are to correspond to the desired task semantically, the constraints of the planning problem must also enter into the process. These constraints could themselves be represented ties that mirror the token probabilities generated by language model, reflecting their applicability the current environment rather than their semantic likelihood. We can frame this a problem similar to probabilistic filtering: decode a sequence (i.e., a task description) that both has a high probability under the language model and high probability under grounded model that predicts how applicable this sequence is to the current scene.
probabili-
Herein, present Grounded Decoding (GD), scalable, general approach to planning with LLMs embodied domains. Grounded Decoding jointly decodes the token probability an LLM and token probabilities from token-conditioned, robotic functions, such affordance functions capturing the abilities of a robot given its embodiment, safety functions, or more. By guiding the LLM directly at its output, Grounded Decoding enables general and flexible family of planning algorithms that combines LLM's strength of long-horizon and semantic reasoning and grounded models' strength of local and embodiment grounding.
Our contributions are as followed: 1) we present a robot-centric formulation for decoding lan- guage models to perform long-hov long-horizon robotic tasks with token-conditioned grounded models, 2) we demonstrate techniques for learning such grounded models, serving different purposes such as affordances and safety requirements, and 3) we show empirical evidence, across three simulation and real-world domains, that the proposed method performs strongly on a wide range of tasks while also significantly outperforming prior methods in efficiency.
2 Related Work
Guided Decoding for Language Models. Decoding strategies for large language models is an active area of research within natural language processing [77, 87, 25, 85, 38]. A number of recent works have focused on developing decoding heuristics for natural text generation [49, 35, 48, 18, 25, 6, 36, 63]. Another line of works use external classifiers for maximizing certain language utilities when decoding language models [71, 92, 26, 39, 37, 21, 38, 12, 23]. Most closely related to our work are classifier-guidde decoding methods developed for offline domains, such as image captioning [78, 74] and task-oriented dialog [72, 83). However, extensions to embodied domains, which we investigate exclusively in this work, remain non-trivial because grounding in embodied domains bounded by the abilities of the agent and by environment state transition as the agent actively interacts with the environment.

---

Please pack a picnic box for me.
peach cake
0.2
Large
0.2
Task Plan:
Language Model
salad
0.2
1.
Put the peach the brown box.
fruit
2.
Put the
cake
0.8
Affordance
pepsi
0.3
peach
salad
0.0
2.
Put the cake
red
0.3
Grounded Models
red
0.0
cake
0.2
Preference
Safety
Figure 2: Overview of Grounded Decoding. Given a ffee--orm language instruction, language model and grounded models jointly decide the next candidate token be decoded by combining their respective likeli- hood. Language model proposes likely tokens that produce goal goal-directed and coherent long- horizon behaviors, while grounded models connect them to the physical scene, through a flexible composition of multiple objective functions from multiple modalities, such as affordance, preferences, and safety.
Embodied and Multimodal Language Models. Training language models to understand em- bodiment is active area of research. Training multimodal models can enable some degree of embodied reasoning, such as understanding images and videos [9, 40, 76, 3]. Directly fine- tuning language models output actions has also been investigated [75, 55, 66]. Lastly, train- ing downstream models on language model embeddings shows promise [46, 52, 24, 94, 60, 41]. In this work, we investigate leveraging large frozen language models for embodied applica- tions [29, 2, 96, 8, 64, 30, 42, 70, 47, 27, 58, 73, 44, 43, 14, 16, 82, 93, 89, 45, 56], with grounded models to provide Jomain-specifee grounding during decoding process.
Comparison to SayCan. The most closely related work to our work is SayCan [2]. SayCan uses a large language model and value function select robotic skills among constrained set of primi- tives. This constrained set of primitives enables SayCan to use the so-called "scoring-mode" of the L.MM get the probability of skill being useful to high-level instruction. This requirement to consider only fixed and enumerated set primitives limits the applicability of SayCan in scenar- ios with many possible skills, such as open vocabulary or combinatorial tasks. Grounded Decoding on the other hand jointly decodes the LLM and the grounded model at the token level, allowing for expressive decoding with an open vocabulary. Furthermore, SayCan considers only grounding func- tions derived from RL-trained value functions for affordance grounding functions, while Grounded Decoding explores many types of grounding functions propose broad family of algorithms.
Task and Motion Planning. Task and motion planning [33] seeks to solve high-level instructions via sequencing tasks in dynamically feasible manner. Research within this area generally focuses on symbolic planning [19] or optimization-basel [79] approaches. Machine learning has increasingly been used to accelerate planning and enable new domains [91, 61, 69, 20, 17, 28, 90, 31, 1, 65, 51, 86, 88, 15]. However, planning constraints are often explicitly specified for TAMP methods. In contrast, we specify constraints (learned) probabilities, which are baked into the decoding process and provided by domain-specific grounded models.
Grounded Decoding
3.1 LLMs and Grounding Models
Large Language Models. LLMs are trained to predict the probability p(W) of a text sequence
W. represented as sequence of tokens W WI:N (wi...., WN). The tokens are elements
of a fixed vocabulary W. Typical neural architectures factorize the joint probability into p(W) =
IIW PLLM(W, [/ where PLLM is predicted by transformer network [81]. Given PLLM- generating text consisting of N-many tokens, the so-called decoding process, can be seen as the P.P.P Wtr.Mw.//w....... (m)m-)), which in practice solved, egg., using greedy search, beam search, or sampling strategies. To further ensure the LLM is solving desired task, one typically starts with a given text, the so-called prefix or prompt, that describes the task, and then the L.M completes this task in its decoding process.
optimization problem arg max eeW 1
3

---

Grounding Functions. We use the concept of grounding functions, Po s), which seek to model a probability of tokens Win given (potentially non-textual) state s S. This state intended to capture the embodiment of the robot and the environment, which may be an image, proprioception of the robot, or the environment state. Thus the grounding function models probabilities relevant to the robot embodiment and environment, such as whether the tokens are possible for the robot to exccute given the state (affordances), or other values like safety, cost, or user preferences.
3.2 Problem formulation.
Given an instruction in language L, we look at the problem of using an LLM to decode a language plan WI:N, which is typically done by finding the most likely tokens under the probability distribu- tion predicted by the LLM, PLLM ( ()), with & being the prefix. However, based on the instruction as the prefix alone, the LLM can easily generate text that is not grounded in the physical state of the environment, rendering such plans useless in the real world. In order to ground the language model in an actual physical embodiment, we propose Grounded Decoding (GD); The main idea of GD is to guide the generation of token sequences with grounding function(s) that conditioned on the embodiment the system.
Formally, let E S denote a representation of the state of the world. Then, GD attempts to generate text that is consistent with both the instruction and the physical state s:
WINN
arg
max PGD WWD(wi.lss,
: W.EW
To leverage the Internet-scale knowledge of LLMs, we factorize PGD ( (s, e) follows 2:
p(s, ([w1NN) p(W1.N)
PGD PGD MGD(WI.Nls, E)
p(s,
p(s|u1:N) ) ((((wrIN (P((N))))
(3)
p(s, E)
p(wL.N1))pp 10)(())(())))))) (spss))((ssss)
(4)
pls, (Os:))((NNN))(W).
p(wi:Nle) p(w):Nls)
(5)
p(w1:N)
x p(wI:N (0)p(wn.((s).
(6)
To decode autoregressively with the formulation, we factorize above into token decoding:
N
moowwNNls, x II (t.lww///1m111 2) (w(wi))))
(7)
n=1
The first term, token for the given instruction E appended previously decoded tokens Wim-- without the state s as input. The second term, Pa Pa(w [s). is the grounding function that is only conditioned on the state and judges whether the generated text win is consistent with the physical state. The core idea behind this factorization is that L.MSs exhibit long-term planning capabilities, while the grounding function guides the planning of the LLM to be possible in the concrete embodied physical world without needing to be informed or capable of reasoning over the long-horizon instruction.
f), can be modeled as the probability of the LLM predicting the
3.3 Grounded Decoding
This work investigates grounded decoding exclusively in the context of task planning for embodied agents. Figure visualizes single step of the simplest greedy search form of GD, and accompany- ing pseudo-code can be found in Algorithm
Given a high-level language instruction and history of executed actions, GD proceeds through process similar to probabilistic filtering by selecting to- kens iteratively that have high probability under the language model and the grounded model. After
We make three assumptions for this derivation: 1) and l are marginally independent (Line 3), 2) and conditionally independent given wiNN (Line 5), and 3) p(w1.)) is uniform responses (Line 6).

---

each token is selected, it is appended to the prefix. The process continues until a token in the termi- nal set Wiemn is selected, which could be period sign "." indicating the end of single-step skill (e.g., pick-and-place). Then the command wi; is sent to a that executes the action a conditioned on the environment state s. Crucially, this grounding func- tion must accept partial commands to enable grounding during decoding." Additionally, we note that GD, in its essence, provides grounded scoring function; thus, it can be easily extended to any search methods such as beam search, top-k sampling, etc.
policy π (a]s, wa:a)
Algorithm Grounded Decoding (GD) w/ Greedy Search
1: Given: state s, instruction terminal set Wiem
2: Initialize: w = {.. n = 0
3: while W/ & Wiem do
4: T1 =
5: Win arg max w. EW PLLM w/a 11m-1, E) pa(Win))s)
6: end while
7: Return: w
3.4 Techniques for Obtaining Grounded Models
Unlike language tasks, where single model is capable of performing general semantic reasoning. a singular grounded model remains an open problem. Indeed, each domain may impose varied environmental and embodiment constraints. Despite these challenges. we present several techniques for obtaining grounded models that can be leveraged in GD's ' formulation. and validate them in three domains in Section 4.
Token-Conditioned Value Functions. Assuming a robot acts with action a according to policy (als, Wim), that aims maximize certain utility and that the utility captures task objective, a natural choice that can provide "grounding score" is the action-value function Q(s, a a/wi..) as necessarily captures the embodiment of the robot. Additional objectives, such as task constraints, can also be encapsulated in Q(s, a alWi.)) ) to ensure grounding. Note that unlike the formulation fixed repertoire of token sequences. In practice, to obtain a (s,, a (1:n) that satisfies the requirements, one can train multi-task
proposed in [2], Wi:n cannot be restricted
agents, either through reinforcement learning (Section 4.2) or supervised learning (Section 4.1).
Multimodal Foundation Models. A general choice to ground LLMs is through using multimodal foundation models, such CLIP [57] open-vocabulary object detectors [22, 34, 50). Although these models can connect language to other grounded modalities (e.g., vision), they often lack the capability for complex long-horizon reasoning, and they do not consider embodiment constraints. As a result, to leverage them in the decoding process, they need to constrained to where they are the most applicable rather than always decoding jointly. To this end, we use prompt-based technique that allows LLMs to choose when to jointly decode (Section 4.3), which we find to be effective in most cases.
Rused Methods. Another source of grounding may come from features x <(win)) designed with expert domain knowledge, which then be used to map Win to a "grounding score" using pamametric or non-parametric functions fx) Such techniques may be most applicable when inter- pretability and enforeing hard constraints required, such safety-critical settings, when data are scarce, such as cases involving preferences of individual users (as shown in Section 4.1).
3.5 Comparisons to Prompt-based Methods
One alternative approach for grounding is including scene information as part of the prompt (e.g., object detection results [96]), which complements the grounding method proposed in this work.
As an example, an affordance ground function for a skill "pick up object", should emit a high probability for "pick" and "pick up" if any object is able to be picked and collapse to only feasible objects only once the object token is decoded.
'Emerging multimodal language models [16, 53] provide strong baselines, but they similarly cannot serve grounding functions because they are not conditioned on embodiment, except for cases where embodiment data from each individual domain can be used to finetune the LLM [16].
5

---

Tabletop Rearrangement
Step Pick Y and place on left side Step Pick up and place it right of Step Done
(Sim)
User
Spell as much of "oou" you can.
Step Go to the purple door and open it
MiniGrid 2D Maze (Sim)
Step 2: Go to the green door and open it
Step Go to the purple door and open
User
Step Go to the green door open it
Traverse the rooms to get to the goal.
Step
the red door open
Step Go to the goal
Thought: will bring you the [pepall and the [apple]
Kitchen Mobile Robot (Real)
apple Step Pick up the apple
Step
soda that coke,
Step 3: Bring you
Step 4: Put down
Language Model
Grouraled Muchel
Combaned Score
Figure 3: Example rollouts and likelihood of representative tokens under Grounded Decoding objective in three distinct domains: simulated tabletop rearrangement (top), Minigrid 2D Maze (middle), and real-world kitchen mobile manipulation (bottom). Each domain uses different prompts, grounded models, and low-level primitives. The GD formulation is shared across the domains, decoding a pre-trained langauge model with respect to domain-specific grounded models decompose open-ended instruction into actionable steps.
However, we note that prompting is often insufficent for grounding, as information about the scene and about the capabilities of the robot may not always be succinetly described in the prompt. Such examples include 1) a block stacking task, a block that has been stacked on cannot be picked, 2) in navigation task, to open door, one must have a key and that door must be reachable, and 3) in a mobile manipulation domain, an object may be visible but is out of reach of the manipulator. Therefore, Grounded Decoding more general and flexible grounding method that injects con- tinuous probabilities during decoding, which may even come from grounding functions from other modalities (e.g., vision).
Experiments
4.1 Long-Horizon Tabletop Manipulation
Herein experiment with simulated tabletop manipulation environment based on RAVENS [95]. We create a custom set of 20 tasks, with 10 seen tasks and 10 unseen tasks. Seen tasks are used for training (for supervised baseline) for few-shot prompting. They are grouped by following categories. Detailed breakdown can be found in Appendix A.2.
i. Letters: Rearranging alphabetical letters ("sort the letters in alphabetical order").
ii Blocks & Bowls: Rearranging combining blocks and bowls ("put blocks in matching bowls").
ii.. Box Packing: Sorting food items and utensils into boxes accordance with safety constraints and user preferences ("Can you pack the picnic box for me?").
Given only high-level language instructions and top-down visual observation of the environ- ment, Grounded Decoding decodes a sequence of text tokens representing the step command to be executed. Note that because GD generated grounded free-form free actions, does require each step to strictly map to a repertoire of skill as in [29, 2]. After complete command is generated, executed via a pre-trained multi-task language-conditioned CLIPort [67]. An example rollout is shown in Figure 4.To demonstrate the techniques proposed in Section 3.4 to obtain grounding functions, we study the composition following grounding functions (overall grounding score calculated PG = I P.) depending on the task categories. Refer to the Appendix A.2 for details.

---

CLIPort Short Long Ungrounded
+LLM
+GD (Ours)
Greedy Beam
+Skills
+LLM
+GD (Ours)
Seen Tasks
Letters
7%
40%
20%
43%
$7
PPO
HRL
Ungrounded
Greedy Beam
Blocks Bowls
2%
62%
35%
60%
77%
Box Packing*
15%
28%
11%
79 %
78%
Easy
28%
68%
96%
100%
100%
Medium
13%
48%
87%
93%
97%
Unseen Tasks
Letters
6%
10% 10%
19%
37%
41
Hard
6%
31%
54%
78%
88% %
Blocks & Bowls
6%
28%
44%
50%
Table I: Tabletop domain success rates.
Table 2: 2D Maze success rates.
Affordance Grounding Function (AF). As the primitive policy CLIPort [67] already acts as an action-value function over the pixel space, we directly leverage its predictions for affordance ground- ing. In particular, given scene image s and partiall--eccded instruction Win, CLIPort predicts unnormalized logits over the pixel space Upick: Uplace E 480x6640 respectively for the pick location and the place location. Therefore, for any given & and Wins we can calculate the affordance as PAr (Wr(Wi:n/ (s) (xa) (Upick (z, V) + Uplace y)).
Safety Grounding Funetion (S). To strictly enforce hard constraints such as safety requirements, we adopt the rule-based method proposed in Section 3,4. In particular, the features æ are indicator functions denoting whether knives and red boxes which we use as hazardous objects are involved in action, i.e., "["redd" or "knife" in Win]). We then use constant mappings to convert the features ps (W
& z x) to scores of or 1, where Z is the normalizing term and Re O is small value for ensuring the joint probability does not collapse to O.
Preference Grounding Function (P). We similarly use rule-based methods for preference ground- ing, as data of individual users may be scarce to learn a separate model. In particular, we choose two random objects (01, 02) as the preferred objects, i.e., x I[o or O2 in wi.]). Note that unlike safety functions, preferences often come in the form of "soft requirement". Therefore, the prefer- ence grounding function is implemented as pp (W/i:mm |s) 2 x 2 (1 2)), where we choose = 0.5 and / = 0.1 for our experiments.
We study two variants of GD using beam search and greedy search. We also compare to "No Grounding" baseline that decodes only according to language model likelihood. Furthermore, we compare to solitary method CLIPort [67] that directly take in the high-level language instructions without a planner. We consider two variants of CLIPort: "Short" that is trained with only single- step pick-and-place commands, and 2) "Long" that is trained on high-level instructions from the 10 training tasks, For more details, please refer to Appendix A.2.
Baselines.
Analysis. Results grouped by each task category are shown in Table 1 Please refer to the Ap- pendix for detailed breakdown. Each method is evaluated 20 episodes for each task within each task category. Supervised methods, such as CLIPort, are found to perform poorly on unseen tasks. Methods that leverage language model planner show better generalization short due to lack of grounding. Grounded Decoding achieves the best results by enabling the LLM plan actions using grounded information and is further improved with beam search.
unseen tasks but fall
4.2 2D Maze
We further evaluate the long-horizon reasoning of Grounded Decoding for 2D maze-solving on Minigrid [10]. The agent receives a top-down view the environment along with a natural language instruction. More details can be found in Appendix A.3. The tasks are grouped in three categories:
i. Easy: Simple tasks where the horizon is short (10-30 steps) and fully described by the textual instruction, e.g. OpenDoors and PutNext.
ii Medium: Short and long-horizon tasks (up to 80 steps) with step-by-step textual instructions,
e.g. LockedRoom.
ii. Hard: Complex, long-horizon instructions (over 100 steps) with ambiguous instructions that necessitate multi-step reasoning and efficient exploration, e.g. MultiRoom and BlockedUnlock.
$ Box Packing tasks are seen during training, but safety and preference requirements are only enforced during evaluation.
7

---

Stack all the blocks.
Step I: Pick up the zellone block and place it on the purple block.
Step Piik up the ned block and place it on the swllom block
Orour Mookettto-iitettseeeoo
Figure 4: Greedy decoding rollout with GD, where key decoded tokens are shown (yellow, Combined scores are normalized the maximum for visual clarity; others are normalized their sum.
Svellow.prpple, purple, red, yellow).
Affordance Grounding Function. Following the recipe from Section 3.3, we train token- conditioned affordance function to be used in GD. The difference is that the grounding function here is the value function from the goal-conditioned policy that is trained with PPO [62] instead of from demonstrations as in CLIPort [67]. The policy performs short-horizon skills such as "Go to red key" "Open the door" and are conditioned on CLIP embeddings of the skill and an image of the scene. Accordingly, the goal-conditioned value function evaluates the feasibility given the current observation and the (partially) decoded skill.
Baselines. We compare the two variants of GD - with greedy and beam search with 1) a solitary PPO policy [62], 2) hierarchical RL algorithm which plans over the low-level skills, and 3) hierarchical method that uses an ungrounded language model for planning [29].
Analysis. Table 2 reports the success rate, averaged across 100 episodes of randomly initialized environments. The "flat" RI agent performs poorly in all but the simplest environments, owing to difficulties with understanding the high-level instruction and reasoning over long horizons (often over 100 steps). Planning over low-level skills using hierarchical RL [5] improves this performance, since the high-level decision-makin problem is greatly simplified. However, the high-level RL agent still needs to reason over low-level (textual) skills by understanding their underlying capabili- ties and stitching them together. Using the planning capabilities of large language models to reason over textual skills significantly boosts this performance [29], since the language model can inherit the strong reasoning capabilities from its training data. This tends to be insufficient in challeng- ing environments, however, since the number potentially viable skills may be very large and the LLM has information about the robot's observations. GD can leverage the learned affordance function (in this case, the goal-conditioned value function) to inform the language model's 's plans, enabling successful long-horizon reasoning. We further find that beam search improves performance modestly, particularly in long-horizon tasks.
4.3 Mobile Manipulation in a Physical Kitchen
Our last environment is kitchen robot in the real world, and we follow the same implementations of the mobile manipulation platform and skills in SayCan [2]. We perform instruction following tasks, as in [2]. An example task is "Bring an apple", for which the robot needs to plan and execute sequence "1 Find an apple, 2. Pick up the apple, 3. Bring to you. 4. Put down, 5. Done". We split the tasks into two categories. Unambiguous means the instruction explicitly contains the object of interest, and Ambiguous means the instruction does not contain the object name. For example, when human asks "bring me the fruit", the robot needs to first determine available fruits. We assume all necessary objects are in the field of view. More details can be found in Appendix A.4.
Grounded Decoding with Chain-of-thought We demonstrate using multimodal foundation mod- els for Grounded Decoding, as proposed in Section 3.4. In particular, we use open- -pen-vocabulary object detector owl-vit [50]. Note that because these off- fhf-thh-sheff models are not trained on robot domain data, we find that it works best by constraining their influence on decoding. We achieve this by mak- ing slight modification to the SayCan algorithm [2]: before generating action plans, we prompt the I.LM to generate Aiuuall-groundde chain-of-thours [84] by giving L.LM the option of when to enable grounded decoding and disable grounded decoding, as visualized in Fig. 5. Specifically,
8

---

Romon: code shat ooter and feets
GD
SayCan
Joben plant 1. find on smale
sngpatziot tnapefroc
gparge
Orcanded racoting
Beang pea
Tasks
Planning Execution Planning Execution
Unambiguous
85%
57%
85%
57%
Ambiguous
58%
44%
33%
25%
Table 3: Success rates in kitchen environment.
GD (Greedy) GD (Beam) SayCan
Success Rate
50%
60%
64%
Token Count
1x
4.3x
113.7x
Cay Eaec Jien
Table By avoiding full enumcration skills, GD more efficient than SayCan while staying performant.
Figure 5: Example prompt and rollout in real- world kitchen environment.
LLMs can be prompted to generate a left bracket to start decoding jointly with grounded models and right bracket revert ungrounded decoding. After chain-of-thought, we SayCan scoring mode for decoding the action plans.
Analysis. Table 3 shows that GD recovers similar performance on Unambiguous tasks, and gain 25% in planning performance on Ambiguous tasks. This shows that GD with multimodal foundation models can effectively visually-groundee chain-of-thought to disambiguate abstract tasks.
Analysis
5.1 Comparison to SayCan
In this section, directly compare GD SayCan [2], which is related our method in that both combine language model knowledge and grounded model knowledge (discussed in more detail in Section 2). However, SayCan uses the language model score all pre-specified options, rendering it inefficient at dealing with large or combinatorial action spaces. In contrast, GD computation considers all possible language token in the autoregressive decoding process, which is independent of the size of the action space. Results shown in Table 4 demonstrate that GD is two orders of magnitude more efficient on our tasks, with comparable performance. Furthermore, by decoding at the most basic functioning unit of language, GD's formulation allows open- vocabulayy grounding beyond just affordances, e.g. safety, preferences, and multimodal embeddings such as CLIP.
5.2 Breakdown of Failure Reasons
Because all hierarchical approaches share an imperfect low-level policy for step execution, the re- sults reported in Table are compounded with both planning failures and low-level policy failure. In Figure 6, we provide failure breakdown analysis for Grounded Decoding and associated baselines. Note that the CLIPort baselines are solitary methods that do not use a planner, the failures solely composed of policy failures. As shown in Figure 6, while all planning-based methods use the same underlying low-level policy, Grounded Decoding significantly reduces planning failure by being able to incorporate grounded scene information into the decoding process. Moreover, we ob- serve that despite the shared affordance function across beam and greedy search, the beam search variant performs stronger by being aware of the full-length single-step instructions during decoding.
5.3 Grounded Action Manifold
A central goal of this work is to investigate the integration of grounded information into language model decoding to output instructions actionable by a policy. To investigate this, we use a t-SNE [80] plot to illustrate the extent to which grounded models help narrow down the search space for lan- guage models. Specifically, we first enumerate all meaningful instructions such as "pick x and place it on y," which are represented as dots in the figure. We then compute the affordance values with respect to four different scenes, where each color represents one scene. Finally, we group the dots using t-SNE and BERT embeddings [13]. Figure 7 shows that grounded
the tabletop domain,

---

Policy Faluure
Planner Fallur
Fallure
Languege bouce Creaniin dinn
(Bea))
(Greddy)
jlng!
Figure Visualization of actions colored by affordance ues in different scenes. Every dot represents a possible action in the tabletop domain, where the majority of the actions are infeasible. We show how grounded models identify the feasible actions specific scenes. Notably, these actions are not always clustered language space, requiring the ground- ing function to determine what action to perform.
100%
Figure 6: Failure breakdown tabletop do- main. GD achieves lowest planning failure among planning-bac based methods, among which beam search variant performs the best.
models can effectively identify achievable skills to produce an actionable manifold within the lan- guage space and that this grounding is required, language alone does not perfectly group action- possible skills for
able skills. It is worth noting that while we provide manual enumeration practical analysis, the full language space is much larger. This highlights the even more pronounced narrowing of the search in the language space.
Conclusions, Limitations, & Future Works
We presented Grounded Decoding (GD), approach for leveraging the knowledge and capabilities of large language models in embodied settings through grounding functions, which model the prob- abilities of tokens given embodiment. GD resembles probabilistic filtering, by decoding tokens that have high probabilities under the language model and under grounded model(s). By guiding the LLM'' decoding directly its output, GD a general, flexible, and expressive approach bodied tasks. This is demonstrated on three embodied domains, showing GD is capable of solving complex, long-horizon tasks.
Though quite general and flexible, GD has a few limitations. First, although we present several techniques for obtaining grounding functions in different domains, it remains question whether a capable and general grounding function can be obtained. We hope that recent progress large- scale robotics models (e.g. [7] and [59]) can remove this bottleneck, and note that the flexibility of GD allows such progress to be straightforwardly leveraged. Second, prompt engineering often required steer L.MSs the desired action space (e.g.. likely action verbs, likely present objects). Finally, while not requiring additional training, the joint decoding may be limiting compared to a single model capable of both grounding and language reasoning [3, 9, 16].
This work presented a family of algorithms for grounding LLMs embodiment, for which there are many avenues for future work. The flexibility of the approach enables many other grounding func- tions and ways to integrate grounding. Furthermore, the development and integration a foundation model for grounding would improve performance significantly. Finally, though GD's probabilistic filtering- based approach quite general, fusing grounding information to the language model after each token decoding may be limiting and future works investigate how such grounding can be elegantly integrated during decoding.
Acknowledgments
The authors would like acknowledge Pierre Sermanet, Carolina Parada, Jie Tan, Yevgen Chebotar, Vincent Vanhoucke, and Dorsa Sadigh for their feedback and contributions. This work is supported part by OpenAl academic access program, granted to Wenlong Huang.
10

---

References
[1] Christopher Agia, Toki Migimatsu, Jiajun Wu, and Jeannette Bohg. Stap: Sequencing task- agnostic policies. arXiv preprint 12Xiv:2210.122500 2022.
[2] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang. Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan. Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, and Mengyuan Yan. Do can and not as i say: Grounding language in robotic affordances. In arXiv preprint arXiv:2204.01699 2022.
[3] Jean-Baptiste Alayrac, Jeff Donahue. Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, al. Flamingo: visual language model for few-shot learning, arXiv preprint arXiv:2204. 14198, 2022.
[4] Ashutosh Baheti, Alan Ritter, Jiwei Li, and Bill Dolan. Generating more interesting responses in neural conversation models with distributional constraints. arXiv preprint arXiv:1809.01215. 2018.
[5] Andrew Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete Event Dynamic Systems, 13(1-2):41-77.jm 2003.
[6] Sourya Basu, Govardana Sachitanandam Ramachandran, Nitish Shirish Keskar, and Lav R Varshney. Mirostat: A neural text decoding algorithm that directly controls perplexity. arXiv preprint S.Ki::2007.14966 2020.
[7] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022.
[8] Boyuan Chen, Fei Xia, Brian Ichter, Kanishka Rao, Keerthana Gopalakrishnan, Michael S Ryoo, Austin Stone, and Daniel Kappler. Open- vocabulayyy queryable scene representations for real world planning. arXiv preprint www.::220.00974.... 2022.
[9] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794, 2022.
[10] Maxime Chevalier-Bei -Biisver,, Lucas Willems, and Suman Pal. Minimalistic gridworld environ- ment for gymnasium, 2018
[11] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.00 02311, 2022.
[12] Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. arXiv preprint arXiv:/9/2.02 164, 2019.
[13] Jacob Devlin, Ming-Wer Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1rXiv:/8/0.044 0404005, 2018.
[14] Yan Ding, Xiaohan Zhang, Chris Paxton, and Shiqi Zhang. Task and motion planning with large language models for object rearrangement. arXiv preprint arXiv:2303. 2023.
[15] Danny Driess, Ozgur Oguz, Jung-Su Ha, and Marc Toussaint. Deep visual heuristics: Learning feasibility of mixed-intedd mixed-integer programs for manipulation planning. In 2020 IEEE International Conference Robotics and Automation (ICRA), pages 9563-9569, IEEF, 2020.
11

---

[16] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint N3iv:2333..03378, 2023.
[17] Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridg- ing planning and reinforcement learning. Advances in Neural Information Processing Systems, 2019.
[18] Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. arXiv preprint arXiv: 04833, 2018.
[19] Richard E Fikes and Nils Nilsson. Strips: new approach to the application of theorem proving to problem solving. Artificial intelligence, 1971.
[20] Caelan Reed Garrett, Chris Paxton, Tomás Lozano-Pérez, Leslie Pack Kaelbling, and Dieter Fox. Online replanning in belief space for partially observable task and motion problems. In 2020 IEEE International Conference Robotics and Automation (ICRA), 2020.
[21] Marjan Ghazvininejad, Xing Shi, Jay Priyadarshi, and Kevin Knight. Hafez: an interactive poetry generation system. In Proceedings of ACL 2017, System Demonstrations, pages 43-48, 2017.
[22] Xiuye Gu, Tsung-Yo Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104. 13921, 2021.
[23] Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. Toxigen: A large-scale machine-generatte dataset for adversarial and implicit hate speech detection. arXiv preprint arXiv:2203.09509, 2022.
[24] Felix Hill, Sona Mokra, Nathaniel Wong, and Tim Harley. Human instraction-follooiing with deep reinforcement learning via transfer-learning from text. arXiv preprint arXiv:2005.09382, 2020.
[25] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint 17X7v 2019.
[26] Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bossclut, David Golub, and Yejin Choi. Learning to write with cooperative discriminators. arXiv preprint arXiv: 000807 2018.
[27] Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard. Visual language maps for robot navigation. arXiv preprint 1xXi:22210.05714, 2022.
[28] De-An Huang, Suraj Nair, Danfei Xu, Yuke Zhu, Animesh Garg, Li Fei-Fei, Silvio Savarese, and Juan Carlos Niebles. Neural task graphs: Generalizing to unseen tasks from single video demonstration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[29] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning. PMLR, 2022.
[30] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Em- bodied reasoning through planning with language models. In arXiv preprint arXiv:2207. 056088 2022.
[31] Brian Ichter, Pierre Sermanet, and Corey Lynch. Broadly-exploring, local-policy trees for long-horizon task planning. Conference on Robot Learning (CoRL.), 2021.
[32] Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Be-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pages 991-1002. PMLR, 2021.
12

---

[33] Leslie Pack Kaelbling and Tomás Lozano-Pérez Hierarchical planning in the now. In Work- shops at the Twenty-Fourth AAAI Conference on Artificial Intelligence, 2010.
[34] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780-1790, 2021.
[35] Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Dragomir Radev, Yejin Choi, and Noah A Smith. Beam decoding with controlled patience. arXiv preprint arXiv:2204.0544 2022.
[36] Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint 1909.00899 2019.
[37] Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. Gedi: Generative discriminator guided sequence generation. arXiv preprint arXiv:2009. 06367, 2020.
[38] Rémi Leblond, Jean-Bayg BaantBattstt Alayrac, Laurent Sifre, Miruma Pislar, Jean-Br -ann-Batistte Lespiau, loannis Antonoglou, Karen Simonyan, and Oriol Vinyals. Machine translation decoding be- yond beam search. arXiv preprint arXiv:2104.05334 2021.
[39] Jiwei Li, Will Monroe, and Dan Jurafsky. Learning to decode for future success. arXiv preprint arXiv:1701.065449 2017.
[40] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint htt.::/90.005577 2019.
[41] Shuang Li, Xavier Puig. Yilun Du, Clinton Wang. Ekin Akyurek, Antonio Torralba, Jacob Andreas, and Igor Mordatch. Pre-trained language models for interactive decision-making. arXiv preprint arXiv:2202.01777/ 2022.
[42] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. arXiv preprint 27Xiv:2209.07753, 2022.
[43] Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. Text2motion: From natural language instructions to feasible plans. arXiv preprint arXiv:2303.122 121153333....,.. 2023.
[44] Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Lim+ p: Empowering large language models with optimal planning proficiency, arXiv preprint arXiv:2304. 14777, 2023.
[45] Yujie Lu, Pan Lu, Zhiyu Chen, Wanrong Zhu, Xin Eric Wang, and William Yang Wang. Mul- timodal procedural planning via dual text-image prompting, arXiv preprint arXiv:2305.01795, 2023.
[46] Corey Lynch and Pierre Sermanet. Grounding language in play. 2020.
[47] Oier Mees, Jessica Borja-Diaz, and Wolfram Burgard. Grounding language with visual affor- dances over unstructured data. arXiv preprint arXiv:2210.01911,, 2022.
[48] Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. Typical decoding for natural language generation. arXiv preprint arXiv:2202.00666 2022.
[49] Clara Meister, Tim Vieira, and Ryan Cotterell. If beam search is the answer, what was the question? arXiv preprint 2rXiv:2010.02654 2020.
[50] Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. Simple apen-vocabulary object detection with vision transformers. arXiv preprint 26X22 2022.
13

---

[51] Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. In Advances in Neural Information Pro- cessing Systems, 2018.
[52] Suraj Nair, Eric Mitchell, Kevin Chen, Brian Ichter, Silvio Savarese, and Chelsea Finn. Learn- ing kanguage-conditione robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning, pages 1303-1315 PMLR, 2021.
[53] OpenAl. Gpt-4 technical report. arXiv, 2023.
[54] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02/55 2022.
[55] Alexander Pashevich, Cordelia Schmid, and Chen Sun. Episodic transformer for vision-and- language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[56] Dhruvesh Patel, Hamid Eghbalzadeh, Nitin Kamra, Michael Louis luzzolino, Unnat Jain, and Ruta Desai. Pretrained language models visual planners for human assistance. arXiv preprint 2rX4v: 19..0099999999 2023.
[57] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. Ma- rina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference Ma- chine Learning, volume 139 Proceedings of Machine Learning Research, pages PMLR, 18 24 Jul 2021.
[58] Shreyas Sundara Raman, Vanya Cohen, Eric Rosen, Iffrah Idrees, David Paulius, and Stefanic Tellex. Planning with large language models via corrective re-prompting. arXiv preprint vrXiv:2211.09933 2022.
[59] Scott Reed, Konrad ZoIna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175. 2022.
[60] Machel Reid, Yutaro Yamada, and Shixiang Shane Gu. Can wikipedia help offline reinforce- ment learning. arXiv preprint arXiv:2201. /2/22, 2022.
[61] Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi- ceri-parametric topological mem- ory for navigation. arXiv preprint arXiv: 00653, 2018.
[62] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 177.v:170. 2017.
[63] Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Sta- iano. Discriminative adversarial search for abstractive summarization. In International Con- ference on Machine Learning, pages 8555-8564 PMLR. 2020.
[64] Dhruv Shah, Blazej Osinski, Brian Ichter, and Sergey Levine. Lm-nav: Robotic nav- igation with large pre-trained models of language, vision, and action. arXiv preprint arXiv:2207.0442 2022.
[65] Dhruv Shah, Peng Xu, Yao Lu, Ted Xiao, Alexander Toshev, Sergey Levine, and Brian Ichter. Value function spaces: Skill-centric state abstractions for long-horizon reasoning. ICLR, 2022.
[66] Pratyusha Sharma, Antonio Torralba, and Jacob Andreas. Skill induction and planning with latent language. arXiv preprint arXiv:2 2021.
[67] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for
robotic manipulation. In Conference on Robot Learning, pages 894-906. PMLR, 2022.
[68] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: multi-task transformer for robotic manipulation. In Proceedings of the 6th Conference on Robot Learning (CoRI.), 2022.
14

---

[69] Tom Silver, Rohan Chitiss, Nishanth Kumar, Willie McClinton, Tomas Lozano-Perez Leslie Pack Kaelbling, and Joshua Tenenbaum. Inventing relational state and action abstrac- tions for effective and efficient bilevel planning. arXiv preprint wrXiv:2003096334 2022.
[70] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. arXiv preprint arXiv:2209.11302, 2022.
[71] Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural language generation with implicit language q learning. arXiv preprint arXiv:2206. 11771, 2022.
[72] Charlie Snell, Sherry Yang, Justin Fu, Yi Su, and Sergey Levine. Context-aware language modeling for goal-oriented dialogue systems. arXiv preprint arXiv:2204. 10198, 2022.
[73] Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. Lim-planner: Few-shot grounded planning for embodied agents with large language models. arXiv preprint arXiv: 2.04088, 2022.
[74] Yixuan Su, Tian Lan, Yahui Liu, Fangyu Liu, Dani Yogatama, Yan Wang, Lingpeng Kong, and Nigel Collier. Language models can see: plugging visual controls in text generation. arXiv preprint 2rXiv:2205.02655. 2022.
[75] Alessandro Suglia, Qiaozi Gao, Jesse Thomason, Govind Thattai, and Gaurav Sukhatme. Em- bodied bert: A transformer model for embodied, language-guis guided visual task completion. arXiv preprint vrXiv:2108.04927 2021.
[76] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
[77] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence sequence learning with neural networks. Advances in neural information processing systems, 27, 2014.
[78] Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf. Zero-shot image-to-text generation for visual-semantic arithmetic. arXiv preprint arXiv:2111.144477 2021.
[79] Mare Toussaint. bined task and motion planning. In Twenty-Fourts International Joint Conference on Artificial Intelligence, 2015.
programming: An optimization- approach to com-
[80] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911)), 2008.
[81] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[82] Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor, Design principles and model abilities. 2023, 2023.
Chatgpt for robotics:
[83] Siddharth Verma, Justin Fu, Mengjiao Yang, and Sergey Levine. Chai: A chatbot ai for task- oriented dialogue with offline reinforcement learning. arXiv preprint arXiv:2204.08-22 2022.
[84] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. /903, 2022.
[85] Sean Welleck, Ilia Kulikov, Jaedeok Kim, Richard Yuanzhe Pang, and Kyunghyun Cho. Con- sisteney of recurrent language model with respect incomplete decoding. arXiv preprint arXiv:2002.024999 2020.
[86] Bohan Wu, Suraj Nair, Li Fei-Fei, and Chelsea Finn. Example-driven model-based reinforce- ment learning for solving long-horizon visuomotor tasks. arXiv preprint arXiv:2109, 10312, 2021.
15

---

[87] Yonghui Wu, Mike Schuster. Zhifeng Chen, Quoc V Le. Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural ma- chine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv: 1609 08 44, 2016.
[88] Fei Xia, Chengshu Li, Roberto Martín-Martnn, Or Litany, Alexander Toshev, and Silvio Savarese. Relmogen: Integrating motion generation in reinforcement learning for mobile ma- nipulation. 2021 IEEE International Conference on Robotics and Automation (ICRA), 2021.
[89] Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, and Harold Soh. Translating natural language planning goals with large-language models. arXiv preprint arXiv:2302.05/28 2023.
[90] Danfei Xu, Ajay Mandlekar, Roberto MartinnMartin, Yuke Zhu, Silvio Savarese, and Li Fei- Fei. Deep affordance foresight: Planning through what can be done in the future. 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 6206-62111 6213. IEEE, 2021.
[91] Danfei Xu, Suraj Nair, Yuke Zhu, Julian Gao, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Neural task programming: Learning to generalize across hierarchical tasks. In 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018.
[92] Kevin Yang and Dan Klein. Fudge: Controlled text generation with future discriminators. arXiv preprint arXiv:2 204.02200, 2021.
[93] Haoqi Yuan, Chi Zhang. Hongcheng Wang, Feiyang Xie, Penglin Cai, Hao Dong, and Zongqing Lu. 4lan4mc: Skill reinforcement learning and planning for open-world minecraft tasks. arXiv preprint arXiv:2303. 16563, 2023.
[94] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. Advances in Neural Information Processing Systems, 2021.
[95] Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong. Vikas Sindhwani, and Johnny Lee. Transporter networks: Rearranging the visual world for robotic manipulation. Conference on Robot Learn- ing (CoRL., 2020.
[96] Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al. So- cratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint a0X0v:2204.00598, 2022.
16

---

A Appendix
A.1 Grounded Decoding Implementation Details
We study three different implementations of Grounded Decoding for each of the experimental do- mains. While each instantiation applied Grounded Decoding long-horizon planning and behavior synthesis, different components including language models and grounded models are used in each domain, as seen in Table 5. Grounded models used in these domains include Affordance Functions (AF), Safety Functions (S), Preference Functions (P), and Open- vreen-ooabullry Object Detectors (D).
Tabletop Rearrangement (Sim)
MiniGrid Maze (Sim)
Kitchen Mobile Manipulation (Real)
L.L
InsineiGPT [54)
InstraiGPP
InstrucGGPT PaLM[[
Primitives
CLiPort [67]
PPO [62]
KT-1
Grounded Mudels
S
AF
D
Table 5: Comparison between different versions of GD implemented in three different environments.
A.2
Implementation Details of Simulated Tabletop Rearrangement
A.2.1 Tasks
There are a total of 20 tasks (templates of language instructions), listed in Table 6, grouped into three task category: Letters, Blocks&Bowls, and Box Packing. Three categories share a total of 57 objects. For Letters category, the goals are to rearrange the alphabetical letter objects such that they satisfy certain orders specified by the language instructions. At the beginning of each episode, task-relevant objects and set of to randomly-sampled distractor objects (except for the Letters category) are initialized random positions is enforced between any two objects to avoid collision and penetration at initialization. To allow for automatic evaluations, a binary reward function is defined for each task using ground-truth state of the objects. Furthermore, we implement scripted policies for each task to collect demonstrations for training the CLIPort baseline. For certain tasks, also randomize the attributes mentioned in the given instructions, which can be found below:
the tabletop with fixed orientations. A minimum 15cm distance
i. (word): hi, world, left, right, top, down, love, you
ii (comer/side)): left side, top left corner, top side, top right corner, bottom right corner, bottom side, bottom left corner
A.2.2 Low-level Primitives
We use CLIPort [67] the low-level primitive that can be invoked by the LLM planner, as it shows promising results of generalization across free-form language instructions. Additionally, since the policy predicts per-pixel affordance, it can be repurposed to serve grounded models for planning for long-horizon tasks, which we leverage in this work. The single primitive policy is trained on 50,000 pre-collected demonstrations, across 10 training tasks, where each demonstration contains 1) language instruction of the format "pick up [x] and place it on [y]". top-down RGB-D observation of the current scene, 3) expert pick location expressed location expressed as pixel coordinates. The expert actions are obtained by accessing ground-truth object pose in the simulator. We further apply substring augmentation during training as we find helps with performance on partial commands (see Section
pixel coordinates, and 4) expert place
A.2.3 Language Model
We use InstructGPT [54] (text-davinci-002) accessed through OpenAl API.
A.2.4
CLIPort Baseline
CLiPort [67] already takes as input natural language instruction and capable of directly
outputting robot actions, it bears the question whether we need high-level planner for completing
17

---

long-horizon tasks. To this end, we additionally train two variants of multi-task CLIPort policy on 10 of the total 20 tasks as baselines (see Table 6 for the train/test split). One variant, which we referred as "CLIPort (Short)", is trained only on single-step pick-and-place instructions of the format "pick up [x] and place it on [y]" on the 10 training tasks. The decomposed pick-and-place instructions are obtained from scripted planners. At evaluation time, the policy is fed in only the high-level instructions without any planners. The other variant, which we referred as "CLIPort (Long)", is trained on the high-level instructions from the 10 training tasks (without decomposition from scripted planners). Similarly, at evaluation time, is fed in only the high-level instructions and evaluated on both seen and unseen instructions. Both variants are trained on 50,000 demonstrations, similar to the Grounded Decoding primitive. The goal of these baselines is to evaluate whether solitary language-conditinn conditioned policies can perform well on long-horizon tasks and generalize to new task instructions. Note that the CI Cort baselines are different from the primitive used Grounded Decoding, although they share the same architecture.
18

---

A.2.5 Full Experimental Results in Simulated Tabletop Domain
Below we show the full list of tasks and the full experimental results in the simulated tabletop domain. Each entry the average success rate across 20 rollouts. The tasks with blue-colored background are seen tasks and the tasks with orange-colored background are the unseen tasks. Seen tasks may used for training for supervised baselines (CLIPort) or may be used the prompt for methods using language model planner. Note that for the "Box Packing" task category, although all tasks were seen in training or the prompts, we enforce additional safety and preference constraints for evaluation only at test time.
+Grounded Decoding
Tasks
PG
Short Long Ungrounded Greedy
Beam
Letters
Put the letters in alphubetical order from left to right
AF
5%
20%
10%
20%
40%
Spell much of word
AF
10%
60%
30%
60%
65%
Separate the vowels from the remaining letters to the bottom side
AF
5%
40%
20%
50%
65%
Put letters reverse alphahetical order from left right
AF
15%
10%
15%
25%
25%
Correctly spell sport using the letters
10%
10%
30%
30%
geumetrically symmetrical Separate the consonants from the remaining letters bottom side
0%
0%
25%
25%
25%
Sort the letters loss than according to ASCII to bottom side
0%
20%
70%
75%
Blocks & Bowls
Stack blocks
90%
Put blocks on corner/side
Put all the blocks bowls with matching colors
0%
30%
70%
Put blocks in the bowls with mismalched coloes
AF
25%
30%
30%
55%
Put all the blocks different corners
AF
0%
5%
40%
50%
60%
Stack only the blocks of cool colors
AF
5%
5%
20%
70%
70%
Stack only the blocks of warm colors
AF
0%
10%
1.5%
45%
35%
Sort the primary color blocks to the left side
AF
0%
0%
20%
25%
30%
Box Packing
Pack the objects into the brown box
AF S
20%
40%
5%
100%
90%
Pack the objects into the boxes
AF S
10%
20%
5%
75%
70%
I'd like some snacks on the right side
AF + P
15%
20%
15%
40%
55%
Pack picnic box
S
15%
30%
20%
100%
95%
Table Full Experimental Results in Simulated Tabletop Rearrangement Tasks. tasks blue-colored background are seen tasks and the tasks with orange-colored background are the unseen tasks. *Box Packing tasks are all seen during training, but safety and preference requirements are only enforeed during evaluation.
19

---

A.3 Implementation Details of Minigrid 2D Maze
A.3.1
Environment Setup
We use the open-source gym-minie grid suite of environments to evaluate our method with one sim- ple change- instead of the default observation space which is a 7 x egocentric window, our agent has access to entire grid - that allows us to simplify the tasks by removing partial observability [65].
A.3.2 Tasks
The tasks grouped in three categories (please see Table for example instructions):
1. Easy: Simple tasks where the horizon is short (10-30 steps) and fully described by the textual instruction, e.g. OpenDoors and PutNext. The short horizon makes them relatively easy for wide range of HRL algorithms. The instructions for these tasks generally spell out each individual skill, making them particularly casy for high-level planners based on language modeling.
2. Medium:
Combination of short and long horizon tasks (up to 80 steps) with step-by-step textual instructions, e.g. LockedRoom. While being significantly longer, these tasks also tend to have instructions that spell out the low-level tasks (see Table 7).
3. Hard:
Complex, long horizon instructions (over 100 steps) with short, ambiguous instruc- tions that necessitate multi-step reasoning and efficient exploration, e.g. MultiRoom and BlockedUnlock. In addition to being long-horizon, the instructions in this case tend to be ambiguous and under-specified, e.g. "traverse through the rooms to get to the goal", which does not provide enough context for any blind planning agent.
Difficulty
Task Name
Example Instruction
OpenDoors
open door blue, then open door red move the red ball next the green box
Easy
PutNext
Medium
LockedRoom
the red key from the purple room, open the red door and go to the goal
MultiRoom
traverse the rooms to get to the goal
Hard
BlockedUnlock
pick up the blue box
Table Example Instructions in Minigrid
A.3.3 Language Model
We use InstrucIGPT [54] (text--dvvnnci-002) accessed through OpenAl API. The prompts used can be found in Section A.5.
We found the prompts to be generally sufficient for solving the "seen" tasks, as well as "unseen" tasks, i.e. tasks that do not have an example in the context. Empirically, we did not find any improvements by including more then 3 example tasks in the prompt we hypothesize that this is likely due to the shared low-level primitives across tasks. For all Minigrid experiments presented in this paper, we used the prompt shown in Section A.5.
A.3.4 Low-level Primitives
To train low-level primitives, we train an RL agent to solve a wide range of short-horizon sub- tasks (under 10 steps) that are shared across the various Minigrid tasks - go to <obj>, pick up <obj>, drop <obj>, open <obj>. Rather than training individual skills for each of them [65], we train a single multi-task policy that is conditioned on the CLIP embeddings [57] of the task strings. This scheme allows some robustness to synonyms and ambiguous task specifications, and has been widely used in learning ainguage-groundee policies [32, 68).
We train these primitives using PPO [62], as recommended by the environment developers [10]. Each of these skills trained with sparse outcome reward (+1 if trajectory is successful, 0
20

---

otherwise). In addition to these low-level skills, we perform a form of hindsight relabeling where "substrings" of the task strings are masked allow generalization to partial strings, e.g. "go to red" may be interpreted as "go to red key" or "go to red door", and our masking strategy allows the multi-task policy to execute tasks specified by partially complete strings, if necessary.
A.3.5 Additional Qualitative Results
Step g: to the purple door and open it
User
Step 2: go the green door and open
Traverse roams to get to the goal.
Step go
purple
open
Step
the green door and open it
Step S: go to the green door and open it
Step 6: go to the goal
1006
Get the green from the purple room, green door and go the goal.
Step go go purple door open Step
green key
Step Step 4: go to the green door and open it
the green key
Step St go the goal
Scep 1: go to the øbstacte
up obstacle
User
Step 3: place the obstacte
up the purple box.
Scep go
Step 5: pick up the blue key
Scep go
open
Step drop the blue key
go purple
Scep 10: pick up the purple box
Mecal
Grounded Modil (Allordance)
Cumb Score
Figure 8: Minigrid Domain
21

---

A.4
Implementation Details of Real-World Mobile Manipulation
A.4.1 Tasks
Instruction
put an energy bar and water bottle on the table bring a lime soda and bag of chips Can you throw away the apple and bring me a coke bring 7up and tea
move an multigrain chips to the table and an apple to the far counter move the lime soda, the sponge, and the water bottle the table bring me an apple, a coke, and water bottle
Table 8: List of unambiguous SayCan instructions
Instruction
want to wipe spill.
Bring me fruit
Bring me snack
Bring me bag of chips
Bring me bag of snack
Bring me bag of chips and something to wipe spill
Bring me bag of chips and something drink
Bring me bag of chips and soda
Human: I want soda that is not coke, and fruit. want fruit and soda
Table 9: List of ambiguous SayCan instructions
A.4.2 Language Model
For planning, we use PaLM [11]. 540B parameter language model trained on large datasets that include high-quality web documents, books, Wikipedia, conversations, and GitHub code. Be- fore planning, we use InstructGPT [54] (text-davinci-002). accessed through OpenAl API. to generate the (grounded) chain of thought.
We used square bracket to indicate grounded decoding, as illustrated in Fig. 5. The prompts shown in Listing 3.
A.4.3 Low-level Primitives
We use combination of learned and scripted control policies for navigation and manipulation, following the implementation described in SayCan [2] and RT-1 [7]. The manipulation policies for the picking action are learned using Behavior Cloning (BC) on 68000 demonstrations and 12000 autonomous successes that were collected over the course of 11 months using fleet of 10 robots. The demonstrations are collected by telcoperators using VR headset controllers to track the motion of their hand, which is then mapped onto the robot's 's end-effector pose. The navigation policies are scripted, based on a ground-truth map well as a learned perception module for collision avoidance and planning. The placing actions follow pre-computed motions only when preceded by navigation policy. The Value Functions used by SayCan for affordance grounding are provided by the nentwworks of trained RI. agents; we follow the RL training setup described in [2].
A.4.4 gpen-Vocabulary Detector Grounding Function
We use owl-vit [50] as our grounding model. takes an image and natural language query, and returns a list of bounding boxes with scores. We take the maximum score the grounding function.
22

---

More examples of object detection as a grounding function can be found in Fig. 9.
Bunan: vant to vipe some spill.
Ruzan: Bring mo bag of chips.
Robot thought: I will bring the [paper towel]
bring the [ketsie chipa]
Robot thought: will
Iusan: I want a soda that ja not coke, and a deuit.
Raboc thoughe: I Ms11 Zind the [pepes] and the [mpple]
Figure 9: Additional examples of using open-vocabulary object detection as a grounding function in Kitchen Mobile Manipulation Domain.
23

---

A.5
Prompts
Listing Grounded Decoding Prompt in Simulated Tabletop Rearrangement Domain
Task: Pack all letter objects en the brown box place Step pick up the and place it the brown box
Step pick up
Step done
Taak: Put the letters on the tables in alphabetical order
Step pick up
place it
Step 2: pick up the d and place it en the right of right place it
Step pick up
Step pick up the 1 and place it the right of
Step pick up the u and place it on the right of
Step done
Spell as much "blue" you
Step 1: pick up the 1 and place it on the botton left side right of Step pick the the and place it the right of
Step pick up the place
Step done
Task: Separate the vowels from the renaining letters place
Step pick up
Step 2: pick up the and place it the botton side
Step done
Task: Stack all the blocks
Step pick up the
block and it the pink block pick up the cyan block and place it on the brown block cyan black Step 4: pick up the gray black and place it on the orange black
Step 2:
Step pick up orange
place
Step done
Task: Put all the blocks on the botton left corner
Step pick up shita black place it
laft
pick up the yellov block and place it on the botton left corner Step pick up
Step 2:
green Step pick up the blue block and place it the botton left corner
place
corner
Step pick up the purple block and place it on the botton left corner
Step done
natching Stop 1: pick up the cyan black and place it on the cyan boul place it the purple Step pick up the brown block and place it the brown boul
Task: all the
Step pick up
purple
Step pick up the pink block and place it on the pick bowl
Step done
into any Step 1: pick up the donut stick and place it the red box brown Step pick up the peach and place it the brown box
Pack
Step pick up the pepai and place it
Step pick up the strasberry and place it on the red box
Step done
Pack
Step 1: pick up the knifa and place it the brown hox
Step pick up the plun and place on the brown box
Step pick up the pepsi and place it the brown box
Step pick up the cupcake and place it on the brown box
Step done
Teek: Pack
table the Step 1: pick up the i and place it en the brown box
box
Step pick up the green block and place it on the brown box Step pick up the and it the brown box
Step done
Teak: Can you put sone snacks on the right side for ne?
Step pick up plun place
right
Stop 2: done
Task: Can you
picnic box for mo?
Step 1: pick up the orange and place it on the picnic box
Step pick up diet pepai and place
picnic Step 3: pick up the knife and place it on the picnic box
Step
24

---

Listing 2: Grounded Decoding Prompt in MiniGrid 2D Maze Domain
You are 20 maze-solving agent with access to variety of lov-level skills such as picking dropping objects, navigating doors/keys/boxes, opaning/closing doors. example tasks:
Task: get grean key
purple roon, unlock grean go goal Step 1: go to the purple door and open it
Step go
green key Step 3: pick the key
Step go the goal.
green
open
Step
pick up purple
Step 1: go to the green obetacle
Step pick up
Step 3: place the obstacle
Step go
key
Step pick the blue key
Step go to the blue door and open it the blue
Step
Step 7: go to the purple box
Step pick up purple
Task: traverse the roona get to the goal
Step go the purple door and open it
Step go to the green door and open it
Step go the parple door and opan
Step 4: to the green door and open it open
Step go
green Step to the goal.
Now turn.
25

---

Listing 3: Grounded Decoding Prompt in Real- World Kitchen Mobile Manipulation Domain
The folloving objects are in the scene: Tup, apple, banana, nango, tea, multigrain chipe, kettle chipe, chips, coke, grapefruit soda, pepsi, redbull, energy bar, soda, spange, paper tovel, and water bottle.
jalapeno chips,
The following locations are in the scene: close counter, far counter, table, trash, boul. robot will always pat object
brackata Q
Bobot: robot
bring objects you.
Hunan: hungry.
thought:
[multigrain chipa].
Robot plan: 1. Find the multigrain chips
Pick up the multigrain chipe
Bring you
Put it down
robot can bring objecta to you. Hunan: Throw the fruit.
Robot:
Robot thought: I will find the [mango] and nove it the trash.
Robot plan: Find the
2. Pick up the mango
trash
4. Put. it down
Done
Robot: an robot that can bring objects to you.
Hunan: (inject instruction).
Bobot thought:
26


Reviews:
[{"review_id": "hTH8EX07ow", "reviewer": "Reviewer_pMcP", "summary": "This paper proposes Grounded Decoding (GD), a method that filters the decoded task descriptions by the probability of a grounded model, LLM-based task planning. This paper shows the framework can integrate different types of grounded models. Experiments across physical and simulated environments show that this decoding method outperforms prior models in instruction following tasks.", "questions": "- The results show the overall success rate. How does it perform given the safety and preference grounding?\n- There is a huge gap between planning and execution in the real robot experiment. What causes this big gap?\n- In Fig 5, why doesn’t include SayCan’s failure breakdown?\n- Fig 6 is quite confusing, the description says each dot is an instruction and they are colored by the affordance value computed against four different scenes. How are those scenes selected and why some instructions are infeasible if they are meaningful instructions for that domain?", "limitations": "The appear discussed the limitation of the proposed method well. The joint decoding may be limited as the grounding models are limited and most of them are object-centric grounding.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The introduction of grounding functions covers a family of methods that evaluates the likelihood of the generated text plans conditioned on the current states.\n- This paper presents the method clearly and it is easy to follow.\n- The inclusion of grounding in the decoding process reduces the planning failure.\n\n-----\n\nPost-rebuttal update\n\nThe authors' responses answered my questions. I agree that GD is a more flexible framework and can be more easily extended to include various grounding functions as opposed to SayCan. I would encourage the authors to include the rebuttal responses to the paper to improve clarity.", "weaknesses": "- While it is possible to include safety and preference in the grounding function, the current form to include safety and preference is quite preliminary. It only considers the presence of objects but usually, these should be represented as statements.\n- It is a good modification of SayCan to constrain the grounding in the brackets. It is unclear how reliable the ungrounded decoding and activate or revert grounding at the right time and how this modification fits the formulation of GD (Eq. 7).\n- The grounding function can be used as the affordance function in SayCan. If we consider the grounding function this way, the main advantage of GD is its efficiency not the introduction of the grounding function.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback. Please see our responses below.\n\n>*”the current form to include safety and preference is quite preliminary… these should be represented as statements”*\n- The main contribution of GD is presenting the general framework of guiding LLM-based planning with grounding functions, for which we extensively study affordance grounding. At the same time, we seek to show the generality of the concept and extend to additional use cases with safety and preference guidance, where we adopt the simplest methods for implementing these functions. We believe a more sophisticated implementation is very complementary to our work, and we leave this and its in depth study for future work.\n- We’d like to highlight though, that the framework of GD goes beyond putting statements in the prompt, because **continuous probabilities** are used to ground LLMs. This means that more complex safety or preference can be incorporated that leverage other modalities and are more expressive than prompting. For example, 1) continuous probabilities generated by safety or preference models from *other modalities* (e.g., images input into a CLIPort or PPO policy used in affordance grounding), or 2) continuous probabilities generated by another LLM by reading long documents of safety manuals or user history data.\n- We will add a section discussing grounding functions versus prompting with tradeoffs and examples to the updated paper.\n\n>*”It is unclear how reliable the ungrounded decoding and activate or revert grounding at the right time and how this modification fits the formulation of GD”*\n- We find the behaviors to be quite reliable given appropriate prompting – this was not a failure mode. We note that for less proficient language models, it may be less consistent. We will add a statement to this effect.\n- This modification fits into the formulation by setting the grounding function outside of the brackets to identity, as all statements are possible (and thus “grounded”) outside of the objects in the scene. We will clarify this in the paper.\n\n>*”The grounding function can be used as the affordance function in SayCan… the main advantage of GD is its efficiency not the introduction of the grounding function”*\n- The main distinction between GD and SayCan is that GD grounds LLMs in **autoregressive generation**, where SayCan grounds LLMs by **enumeration and ranking**, which requires a pre-defined set of skills and is restricted to output those skills only. Therefore, we believe GD presents a more general and flexible framework that is capable of incorporating a **larger** variety of grounding functions in a more **flexible** form, which especially scales well with multi-task policies. Though we agree that efficiency is a contribution of GD, as seen in Section 5.1, where GD is two orders of magnitude more efficient than SayCan. As capabilities of multi-task policies continue to scale, we expect this difference to be even more pronounced.\n\n>*”The results show the overall success rate. How does it perform given the safety and preference grounding?”*\n- The complete results of the safety and preference grounded tasks can be found in Table 6 in Appendix. For these tasks, the agent not only has to complete the task but also needs to follow the requirements to be considered “successful”. For example, GD attains a success rate of “100%” and “75%” on two tasks that require safety grounding, while the ungrounded baseline has only “5%” success rate. On one task that requires preference grounding, GD attains “55%” and ungrounded baseline has only “15%”. \n\n>*”There is a huge gap between planning and execution in the real robot experiment. What causes this big gap?”*\n- The gap comes from policy failure, where the plan generated by GD is correct but the underlying policy does not successfully complete the sub-task. \n\n>*”In Fig 5, why doesn’t include SayCan’s failure breakdown”*\n- We only show failure breakdown in the simulated tabletop domain, where it’s easier to quantitatively and statistically compare to different baselines over a large number of experiments. As real SayCan evals are quite time consuming to run, we feel we do not have enough data points to generate such a figure.\n\n>*”In Figure 6… How are those scenes selected and why some instructions are infeasible if they are meaningful instructions for that domain?”*\n- Thank you for pointing this out. Please see below for clarification, and we will also improve the clarity in the paper.\n- The scenes are manually selected to show as illustrative examples on why GD is efficient at identifying and grounding actions. More specifically, the domain refers to the general tabletop domain, where different initializations with different sets of objects are possible. We consider an instruction “meaningful” as long as they follow the pick&place action template and refer to possible objects in the full domain. However, depending on the specific initializations, most objects will not be present for each scene, thus most instructions are infeasible (even though they might be “meaningful” in some other scenes in the same domain)."}, {"review_id": "Np8Y6vtHqM", "reviewer": "Reviewer_VNBy", "summary": "This paper proposes Grounded Decoding, a methodology to ground the high level semantic plans output by an LLM into a realizable action sequence. Specifically, the authors construct a model which decodes a sequence that is likely to be realizable given the current state of the robot and environment, but also likely according to the language model. Experimental results in both simulation and the real world demonstrate the utility of the proposed method over baselines from prior work.", "questions": "- In Figure 3 (leftmost), why is the grounded model's score for _orange higher than that of _red, when the red block is visible but there is no orange block present? Similarly, why is the language model's score for _yellow so high in the rightmost plot?", "limitations": "The authors include a detailed section on the limitations of the work.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Well-written. The paper is well-written, and the figures aid in the understanding of the methodology.\n- Strong experimental results. Show the utility of using both the ungrounded LLM and their Grounded Decoding relative to prior work on a range of tasks both in simulation and the real world. Outperform SayCan in terms of efficiency by a non-trivial margin. The inclusion of the qualitative results is also useful.\n- Ablations. The experiments in simulation include ablations which isolate the contributions of the proposed grounded decoding module. Furthermore, the authors include ablations demonstrating the impact of the choice between greedy search and beam search, which is insightful.\n- Inclusion of failure mode analysis. The analysis on the failure modes of the system is insightful. ", "weaknesses": "- Limited experimental evaluation. The authors include results on just two environments in simulation and one in the real world. Showing results on a larger number of environments would make the claims more convincing.\n- Lack of comparisons between grounding functions. Section 3.4 proposes three different techniques for obtaining grounded models that can be leveraged in GD’s formulation. However, the three techniques do not seem to be directly compared in the experiments. An experiment of this form may provide insight about which formulation works best for the proposed algorithm. Furthermore, how do the different grounding functions (AF, S, P) impact performance in the experiments?\n- Ground truth map for navigation. The navigation module used in the mobile manipulation in a physical kitchen experiment is scripted and uses the ground truth map of the environment for navigation. This is a major assumption and limits deployability to only seen environments.\n- Prompt engineering. It is claimed that a certain amount of prompt engineering was done to steer the LLMs. This is an undesirable property of the proposed system. How sensitive is the system to such prompt engineering? Was the same effort also applied in prompt engineering for the baseline methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive remarks on the comprehensiveness of our experimental results as well as the presentation quality. Please see below as we provide more clarification to the raised points.\n\n>*”The authors include results on just two environments in simulation and one in the real world. Showing results on a larger number of environments would make the claims more convincing.”*\n- We hoped over these three domains to show diversity of environments for which GD is applicable. We would attempt to include more results on our environments if there is a specific concern for a aspect of the proposed method, and would be open to new domains if there are some the reviewer can propose that have an available low-level policy.\n\n>*”Lack of comparisons between grounding functions… which formulation works best for the proposed algorithm… how do the different grounding functions (AF, S, P) impact performance in the experiments?”*\n- The main contribution of GD is presenting the general framework of guiding LLM-based task planning with grounding functions. To this end, we show different implementations in different domains leveraging different grounding functions. However, it is difficult to directly compare them as they serve different purposes depending on the tasks in each domain.\n- Additionally, to the best of our knowledge, there doesn’t exist a single grounding function that works the best across all domains. This is also listed in the limitations section of the paper.\n- “AF, S, P” refers to affordance, safety, and preference respectively. Different tasks are designed to evaluate each of these grounding functions, where this is used as part of the success criterion. Please see the complete results in Table 6 in Appendix.\n\n>*”Ground truth map for navigation… This is a major assumption and limits deployability to only seen environments.”*\n- Thank you for pointing this out. We recognize this as a limitation for our mobile manipulation implementation. However, since navigation policies are not part of our main contributions, we believe implementation of a more advanced navigation module is orthogonal to our work. We note that GD may also be extended to learn/obtain better navigation policies by generating step-by-step plans to visit likely regions in an unseen environments, where likelihood can be guided via a grounding function.\n\n>*”How sensitive is the system to such prompt engineering? Was the same effort also applied in prompt engineering for the baseline methods?”*\n- We find that more advanced language models are less sensitive to prompt engineering. As language model capabilities continue to improve, we expect the method to be less dependent on the prompt engineering effort.\n- GD works by guiding the **continuous probabilities** in the decoding process. Therefore, same prompts are used for all baseline methods. This is a major advantage of GD: under the same prompt engineering effort, incorporating grounding functions in the decoding process should likely only help generate better task plans.\n\n>*”In Figure 3 (leftmost), why is the grounded model's score for _orange higher than that of _red, when the red block is visible but there is no orange block present?”*\n- This is likely due to the imperfectness of the grounded model, as they are only trained on limited data in the environment and may confuse the “orange” with “yellow” or other colors.\n\n>*”Similarly, why is the language model's score for _yellow so high in the rightmost plot?”*\n- The language model score is very high for this token because the task is “stack all the blocks” and the previous step already stacks the yellow block onto the purple block. Therefore, the next block must be stacked on top of the yellow block in order to complete the full instruction.\n- This highlights the particular strength of GD in that it considers both semantic likelihood from LLMs from their strong reasoning capabilities as well as grounded information from grounding functions.\n"}, {"review_id": "s4QnpLMThz", "reviewer": "Reviewer_WRL4", "summary": "This paper studies grounding Large Language Models (LLMs) for robot control. The planning tokens are generated by the joint probability of an LLM and grounded models sequentially. The grounded models can be RL value functions, multi-modal perception models or human rules. As an extension of SayCan, this method allows open-vocabulary control and combining multiple grounded models. Experimental results in two robotic environments and a maze environment demonstrate the effectiveness of the proposed method for solving long-horizon complex tasks.", "questions": "1. For the results in Table 3 and 4, does SayCan use the same LLM to your method?\n\n2. According to Table 4, SayCan costs a large number of tokens since it computes scores for all the low-level skills. In its setting where a fixed set of low-level skills are provided, can we use the LLM to output open-vocabulary tokens and match the skill names using text embedding's similarity, to decrease the token cost of SayCan?", "limitations": " The authors discussed the limitations in the paper. Some of these are addressed and others are left to future work. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written, with relevant references and easy-to-follow narration.\n\n2. The paper conducts experiments in various domains and tasks, showing that the proposed grounded decoding method can be widely applied in robotic tasks.", "weaknesses": "1. The contribution of this paper is mainly incremental to SayCan. Though it allows open-vocabulary planning, the key difference is that SayCan requires a fixed set of low-level skills while this paper requires a language-conditioned low-level policy.\n\n2. For the proposed 3 types of grounded models: token-conditioned value functions are similar to the skill affordance in SayCan. Multi-modal foundation models and rule-based methods are more like technical tricks in applications rather than a systematic method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive remarks on the comprehensiveness of our experimental results as well as the presentation quality. Please see below as we provide more clarification to the raised points.\n\n>*”The contribution of this paper is mainly incremental to SayCan”*\n- The main distinction between GD and SayCan is that GD grounds LLMs in *autoregressive generation*, which does not restrict it to output pre-defined skills only. Therefore, we believe GD presents a more general and flexible framework that is capable of incorporating a *larger* variety of grounding functions in a more *flexible* form that especially scales well with multi-task policies. As capabilities of multi-task policies continue to scale, we expect this difference to be even more pronounced.\n\n>*”For the proposed 3 types of grounded models: token-conditioned value functions are similar to the skill affordance in SayCan. Multi-modal foundation models and rule-based methods are more like technical tricks in applications rather than a systematic method.”*\n- Indeed. While our main contribution is presenting a general framework for grounding LLM-based planners in autoregressive generation, we do not find that there is a generalist grounded model that works well across different domains, which is also listed as a limitation in the paper. The main purpose of the presented grounded models is to serve as implementation examples that instantiates the proposed method in different domains depending on the task requirements.\n\n>*”For the results in Table 3 and 4, does SayCan use the same LLM to your method?”*\n- Yes.\n\n>*”In its setting where a fixed set of low-level skills are provided, can we use the LLM to output open-vocabulary tokens and match the skill names using text embedding's similarity, to decrease the token cost of SayCan?”*\n- Yes, this should be able to decrease the token cost of taking forward passes through the LLM. However, it has a few drawbacks. First, as the number of low-level skills combinatorially increases, computing the affordance for all of them may become intractable. Second, the LLM without the grounded context of the scene may output either too generic or incorrect skills. The embedding then needs to correct this, but without the full context of the query it may make mistakes. For instance in the tabletop domain, the LLM may output a skill to pick an orange object, but have to decide whether “yellow” or “blue” is closer. Whether the task is “pick up the lightest color object” or “pick up the darkest color object” changes which to correct the skills to. Instead, in GD the autoregressive grounding can handle this directly during decoding.\n"}, {"review_id": "DJJ3YJHUAn", "reviewer": "Reviewer_XxHZ", "summary": "The paper introduces Guiding Text Generation with Grounded Models for Robot Control. Previous works have shown that large pre-trained language models can generate long action plans for robots based on abstract instructions. However, these models struggle to handle unstructured environments with large action spaces. The project aims to enable language models to generate control commands in a grounded and efficient manner. \n\nThe paper explains how auto-regressive language models generate text by scoring all possible tokens based on their likelihood of completing the current action step. \n\nThe approach incorporates affordance, preference, and safety functions to create grounded models. The decision-making process of selecting tokens is illustrated using the example of stacking blocks. The work also demonstrates how the method can generalize to different tasks without additional training. The flexibility of grounded decoding allows for including objective functions and constraints on the fly. \n\nThe project concludes by discussing the potential applications of grounded decoding in other embodied tasks and the possibility of using advanced search methods for better performance.", "questions": "(1) What is the task size and the robot search space?\n\n(2) Why the 2D Maze experiment matters since it is unrelated to robot control", "limitations": "The limitations are mentioned;\nNo societal impacts found", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The work demonstrates its strength through plenty of experiments. By conducting numerous experiments (Embodied environment, maze, robot control in real space), the researchers provide a comprehensive evaluation of the proposed method and its performance in various scenarios. This enhances the robustness of the findings and allows for a better understanding of the method's capabilities and limitations.\n\nThe work takes into consideration the environmental conditions during planning. By incorporating affordance functions, preference functions, and safety functions, the grounded models consider the feasibility, preferences, and safety aspects of actions within the given environment. This ensures that the generated action plans are not only coherent with the high-level goals but also take into account the specific constraints and requirements of the environment, leading to more practical and context-aware robot control.\n\nThe key idea of the grounded model is well-illustrated. Through clear and concise explanations supported by visual representations and examples. By effectively communicating the core concept of the grounded model, it is easier to understand how the model works and how it addresses the challenges of generating control commands in a grounded and efficient manner. The illustrations help to visualize the decision-making process and the interaction between the language model and the affordance model, contributing to a more intuitive grasp of the proposed approach.", "weaknesses": "Even though the paper is titled with `Robot Control`, it lacks discussion of the robot motion planning considering the environment; i.e. how the robot motion planning changes when involving the grounded model. Especially, comparing the robot motion planning between Saycan and this work can be more persuasive.\n\nThe motivation for involving grounded information is not sufficient. It seems that the grounded model is a direct fusion of affordance map derived by CLIPort, while the whole system is equipped with template language instructions. A more straightforward approach would be using CV model to detect the object in the scene and then include the scene description in the prompt. For example: pick up the fruit in the scene, given `orange, apple, potato, ...`. \n\nThe key idea for considering the grounded map should be that the robot motion needs to think about its configuration, safety, environmental limitations, etc. However, those facts seemed to be disregarded.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the number of experiments in diverse domains. We appreciate the constructive feedback, please see our responses below.\n\n\n>*Lack of robot control*\n- We recognize the comment that the title may not best suit the work and intend to amend the name to “Grounded Decoding: Guiding Text Generation with Grounded Models for **Embodied Agents**”, which we feel reflects the algorithm and experimental domains (including the mentioned 2D Maze environment).\n\n\n>*”lacks discussion of the robot motion planning considering the environment; i.e. how the robot motion planning changes when involving the grounded model. Especially, comparing the robot motion planning between Saycan and this work can be more persuasive”*\n- The low-level motion generation between this work and SayCan is the same. We make the general assumption that there is a low-level policy that can convert state into low-level actions, instead focusing on generating grounded action plans for a given instruction, connecting to said policy – which in both grounded decoding and SayCan is kept fixed. As such the significant difference lies in high-level planning, which is summarized in table 3 and 4.\n\n\n>*“The motivation for involving grounded information is not sufficient.” and “The key idea for considering the grounded map should be that the robot motion needs to think about its configuration, safety, environmental limitations”*\n- We agree that the consideration of the robot’s configuration, safety, environmental limitations are crucial to a grounding function, as are the abilities of the robot’s underlying policies and the state of the environment. We state a version of this on L109 and indeed our grounding functions *do* consider configuration and the environment. As a few examples of where our grounding is more than detection: for the tabletop manipulation experiments CLIPort’s affordance understands that a block that has been stacked on may be visible but cannot be picked, for the 2D maze experiments the value function captures that to open a door one must have a key and that door must be navigable, and for the mobile manipulator experiments the value functions capture that seen object far out of reach cannot be picked. \n- We recognize this was not as clear as it should be and will amend the paper to make this much clearer, by adding the above relevant examples to each grounding function definition (L191, L231, L264) as well as to the discussion in Section 3.4.\n\n\n>*”A more straightforward approach would be using CV model to detect the object in the scene and then include the scene description in the prompt.”*\n- Though we agree that is a possible option, we note that such a detection is not equivalent to an affordance as described above. It would not capture the distance to objects for instance. \n- We will add a section discussing grounding functions versus prompting with tradeoffs and examples. \n- Importantly, the proposed formulation also complements what may be described in the prompt by guiding LLM generation with **continuous probabilities** from any grounding functions. This is thus more flexible, high fidelity, and general method to ground LLMs for embodied applications.\n\n\n>*”What is the task size and the robot search space?”*\n- The tasks are generally 5-10 steps long, and the search space varies depending on the tasks. Additionally, we note that for 2D maze and tabletop manipulation domains, because the policies are free-form language-conditioned, the full search space can be considered the full language space. Below we list the number of valid skills for each domain. We will include this information in the updated paper.\n- 2D Maze: 78 skills\n- Tabletop Manipulation: 3477 skills (57 pick objects, 61 place locations)\n- Mobile Manipulation: ~60 skills \n\n\n>*”Why the 2D Maze experiment matters since it is unrelated to robot control”*\n- While the 2D Maze environment doesn’t directly deal with control, it is a nice platform to validate the main contribution of the paper that studies guiding LLM generation with grounded models. The environment provides intuitive affordance functions and low-level policies that isolate the robotic system designs that are orthogonal to our contributions, while containing a large search space. The methodologies developed in the 2D Maze environment can be directly applied to other embodied environments. In addition, as mentioned above, we will change the name of the paper to better reflect its focus on “Embodied Agents”.\n"}, {"review_id": "HdBXLTlDDM", "reviewer": "Reviewer_TJzA", "summary": "The paper introduces a novel method to tackle a key challenge in utilizing Internet-scale knowledge from LLMs (Language Model Models) in robotics tasks. Specifically, the focus is on aligning the instructions generated by LLMs with the actual actions performed to manipulate objects. The paper endeavors to frame this issue in a manner analogous to probabilistic filtering. This framing aims to decode a sequence that exhibits both high probability according to the language model and high probability according to a predefined set of objectives or attainable skills in the grounded model.", "questions": "1. Would like to see comparison with other methods using detection to learn the labels of the object, and feeding those into the LLMs for grounded reasoning.\n2. Implementation on more general robot framework for reproducible. \n", "limitations": "The authors have address the limitations adequately. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper introduces a novel formulation focused on decoding language models in a robot-centric manner to facilitate long-horizon robotics tasks using token-conditioned grounded models. Additionally, the paper provides empirical evidence showcasing the efficacy of this approach in comparison to CLIPort, which utilizes CLIP as an open vocabulary knowledge-based grounding technique combined with the Transporter Network. Furthermore, the paper demonstrates the transfer-ability of this idea to various domains, including 2D mazes and mobile manipulation.", "weaknesses": "1.The paper could benefit from improvements in certain sections of the writing. For instance, there is an awkward expression in the phrasing of the sentence on lines 116-117.\n2.It would be valuable if the paper included a comparison with other techniques, such as Grounded SAM, which involves detecting objects in the scene and using them as input for LLMs to enable grounded reasoning. This alternative baseline would provide a more robust comparison for the proposed approach, as it still allows for the use of a planner to generate the task plan.\n3.The author's efforts to demonstrate the effectiveness of their approach across different domains are appreciated. However, it would be advantageous if the author conducted real-world experiments that are reproducible with a broader range of robots and baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the novelty, experimental evaluations, and applicability of the proposed method to various domains. We also appreciate the constructive feedback. Please see below for our responses.\n\n>*”The paper could benefit from improvements in certain sections of the writing. For instance, there is an awkward expression in the phrasing of the sentence on lines 116-117.”*\n- Thank you for pointing this out. We will improve the writing for the camera-ready version and rephrase the sentence on lines 116-117.\n\n>*”It would be valuable if the paper included a comparison with other techniques… which involves detecting objects in the scene.”*\n- Though we agree that is a possible option, we note that such a detection is not equivalent to an affordance as described above. It would not capture the distance to objects for instance. \n- We will add a section discussing grounding functions versus prompting with tradeoffs and examples. \n- Importantly, the proposed formulation also complements what may be described in the prompt by guiding LLM generation with **continuous probabilities** from any grounding functions. This is thus more flexible, high fidelity, and general method to ground LLMs for embodied applications.\n\n>*”it would be advantageous if the author conducted real-world experiments that are reproducible with a broader range of robots and baselines”*\n- We agree that more extensive real-world experiments across broader range of robots would further enhance the paper, but we would also like to emphasize that the main contribution of GD is to present a general framework for guiding LLM-based planners with grounding functions, which is agnostic to the type of robots used. Therefore, our main experimental results are focused on validating the proposed method, with statistically significant evaluations in simulation as well as demonstration on physical hardware.\n"}]

Author rebuttal:

Meta-review: This paper proposes Grounded Decoding (GD), a method to ground language model text generation for robot control by incorporating additional "grounding" information. GD decodes sequences that are likely under both a language model and grounded models like affordances or safety functions.

Several reviewers recommended acceptance, citing the novelty of the grounding formulation, strong experimental results across domains, efficiency over prior methods, and intuitive explanations. There were a few concerns mainly about the possibly incremental nature of improvement over prior work like SayCan. Reviewer XXHZ gave a borderline accept, while also commenting that 'robot control' might be a bit of a stretch in describing the paper. The authors acknowledged this comment with their proposal to rename it to 'embodied control', expanding the scope towards more conceptual environments such as maze2D. The reviewer-author discussion also highlighted the increased flexibility over SayCan, and the focus on high-level planning. 

Most of the other issues raised such as limited environments being tested, reliance on prompt engineering, etc. were addressed by the authors by highlighting flexibility of grounding functions, reduced prompt dependence of advanced LLMs, etc. I believe the proposed scheme is a novel enough formulation and the evaluations reasonably validate the approach, even if additional environments could strengthen claims. I recommend acceptance and encourage the authors sufficiently clarify novelty and scope in the camera ready (for example, adding the rebuttal discussion points in the paper). Expanding analysis of grounding choices would also be beneficial.
Decision: Accept (poster)