Title: Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agents
Venue: NeurIPS 2023
Accepted: yes
Keywords: robotics

language models

embodied agents
Abstract: Recent progress in large language models (LLMs) has demonstrated the ability to learn and leverage Internet-scale knowledge through pre-training with autoregressive models. Unfortunately, applying such models to settings with embodied agents, such as robots, is challenging due to their lack of experience with the physical world, inability to parse non-language observations, and ignorance of rewards or safety constraints that robots may require. On the other hand, language-conditioned robotic policies that learn from interaction data can provide the necessary grounding that allows the agent to be correctly situated in the real world, but such policies are limited by the lack of high-level semantic understanding due to the limited breadth of the interaction data available for training them. Thus, if we want to make use of the semantic knowledge in a language model while still situating it in an embodied setting, we must construct an action sequence that is both likely according to the language model and also realizable according to grounded models of the environment. We frame this as a problem similar to probabilistic filtering: decode a sequence that both has high probability under the language model and high probability under a set of grounded model objectives. We demonstrate how such grounded models can be obtained across three simulation and real-world domains, and that the proposed decoding strategy is able to solve complex, long-horizon embodiment tasks in a robotic setting by leveraging the knowledge of both models.

Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agents

Abstract

Recent progress in large language models (LLMs) has demonstrated the ability to learn and leverage Internet-scale knowledge through pre-training with autoregressive models. Unfortunately, applying such models to settings with embodied agents, such as robots, is challenging due to their lack of experience with the physical world, inability to parse non-language observations, and ignorance of rewards or safety constraints that robots may require. On the other hand, languageconditioned robotic policies that learn from interaction data can provide the necessary grounding that allows the agent to be correctly situated in the real world, but such policies are limited by the lack of high-level semantic understanding due to the limited breadth of the interaction data available for training them. Thus, if we want to make use of the semantic knowledge in a language model while still situating it in an embodied setting, we must construct an action sequence that is both likely according to the language model and also realizable according to grounded models of the environment. We frame this as a problem similar to probabilistic filtering: decode a sequence that both has high probability under the language model and high probability under a set of grounded model objectives. We demonstrate how such grounded models can be obtained across three simulation and real-world domains, and that the proposed decoding strategy is able to solve complex, long-horizon embodiment tasks in a robotic setting by leveraging the knowledge of both models.

1 Introduction

Recent works have demonstrated robots that are increasingly proficient at understanding and acting upon natural language, whether through planning or conditioned policies. Complementing such progress, the field of natural language processing has recently seen large language models (LLMs) become ubiquitously used as pre-trained or few-shot prompted models, due to their impressive fewshot performance and vast knowledge-base. These LLMs have efficiently learned from web-scale data through autoregressively modeling the probability distribution over text tokens and thus generate text. However, the nature of this process is such that applying such models to embodied settings remains a challenge. They have not interacted with their environment, lack observability of nonlanguage observation modalities (e.g., images), and may not know what is safe or possible for a particular embodiment.

Determining how to execute long-horizon behaviors based on high-level verbal commands is one particular area of robotics where the rich semantic knowledge in large language models can be especially useful. This problem combines elements of semantic reasoning and planning: the robot

Task Plan:

1. Put the green block in the red bowl.

2. Put the blue block in the

Grounded

Models

Large

Language

Model

red

green

blue

yellow

orange

red

green

blue

yellow

orange

"green"

Grounded

Decoding

User

Place the blocks in

mismatched color bowls.

User

I'd like a drink and a snack.

I'll bring a [pepsi] and an [apple].

User

Go to the green block.

1. Go to the purple door and open it.

must understand the instruction, determine the steps needed to fulfill it, and also determine how to sequence those steps appropriately given its capabilities and the current state of the environment. However, this is not a problem that can be solved purely with semantics, as it requires sufficient grounding to understand how the task should be performed in context - for example, in the example in Figure 1, the language model alone has no way of knowing which block to pick up because this requires knowledge of which blocks are present, and also what manipulations the robot is capable of performing on them. Thus, although a language model can assign probabilities for how likely various steps are to correspond to the desired task semantically , the constraints of the planning problem must also enter into the process. These constraints could themselves be represented as probabilities that mirror the token probabilities generated by a language model, reflecting their applicability to the current environment rather than their semantic likelihood. We can frame this as a problem similar to probabilistic filtering: decode a sequence (i.e., a task description) that both has a high probability under the language model and a high probability under a grounded model that predicts how applicable this sequence is to the current scene.

Herein, we present Grounded Decoding (GD) , a scalable, general approach to planning with LLMs embodied domains. Grounded Decoding jointly decodes the token probability of an LLM and token probabilities from token-conditioned, robotic functions, such as affordance functions capturing the abilities of a robot given its embodiment, safety functions, or more. By guiding the LLM directly at its output, Grounded Decoding enables a general and flexible family of planning algorithms that combines LLM's strength of long-horizon and semantic reasoning and grounded models' strength of local and embodiment grounding.

Our contributions are as followed: 1) we present a robot-centric formulation for decoding language models to perform long-horizon robotic tasks with token-conditioned grounded models, 2) we demonstrate techniques for learning such grounded models, serving different purposes such as affordances and safety requirements, and 3) we show empirical evidence, across three simulation and real-world domains, that the proposed method performs strongly on a wide range of tasks while also significantly outperforming prior methods in efficiency.

2 Related Work

Guided Decoding for Language Models. Decoding strategies for large language models is an active area of research within natural language processing [77, 87, 25, 85, 38]. A number of recent works have focused on developing decoding heuristics for natural text generation [49, 35, 48, 18, 25, 6, 36, 63]. Another line of works use external classifiers for maximizing certain language-space utilities when decoding language models [71, 92, 26, 39, 37, 21, 38, 4, 12, 23]. Most closely related to our work are classifier-guided decoding methods developed for offline domains, such as image captioning [78, 74] and task-oriented dialog [72, 83]. However, extensions to embodied domains, which we investigate exclusively in this work, remain non-trivial because grounding in embodied domains is bounded by the abilities of the agent and by environment state transition as the agent actively interacts with the environment.

User

2.     Put the cake

Task Plan:

1.

Put the peach in the brown box.

2.

Put the

Please pack a picnic box for me.

cake

salad

red

···

0.8

0.1

0.0

0.0

···

Large

Language

Model

cake

salad

fruit

···

0.2

0.2

0.2

0.1

···

pepsi

red

cake

A

···

0.3

0.3

0.2

0.2

···

Affordance

···

Preference

···

Safety

Grounded

Models

peach

peach

···

···

Embodied and Multimodal Language Models. Training language models to understand embodiment is an active area of research. Training multimodal models can enable some degree of embodied reasoning, such as understanding images and videos [9, 40, 76, 3]. Directly finetuning language models to output actions has also been investigated [75, 55, 66]. Lastly, training downstream models on language model embeddings shows promise [46, 52, 24, 94, 60, 41]. In this work, we investigate leveraging large frozen language models for embodied applications [29, 2, 96, 8, 64, 30, 42, 70, 47, 27, 58, 73, 44, 43, 14, 16, 82, 93, 89, 45, 56], with grounded models to provide domain-specific grounding during decoding process.

Comparison to SayCan. The most closely related work to our work is SayCan [2]. SayCan uses a large language model and a value function to select robotic skills among a constrained set of primitives. This constrained set of primitives enables SayCan to use the so-called 'scoring-mode' of the LLM to get the probability of a skill being useful to a high-level instruction. This requirement to consider only a fixed and enumerated set of primitives limits the applicability of SayCan in scenarios with many possible skills, such as open vocabulary or combinatorial tasks. Grounded Decoding on the other hand jointly decodes the LLM and the grounded model at the token level, allowing for expressive decoding with an open vocabulary. Furthermore, SayCan considers only grounding functions derived from RL-trained value functions for affordance grounding functions, while Grounded Decoding explores many types of grounding functions to propose a broad family of algorithms.

Task and Motion Planning. Task and motion planning [33] seeks to solve high-level instructions via sequencing tasks in dynamically feasible manner. Research within this area generally focuses on symbolic planning [19] or optimization-based [79] approaches. Machine learning has increasingly been used to accelerate planning and enable new domains [91, 61, 69, 20, 17, 28, 90, 31, 1, 65, 51, 86, 88, 15]. However, planning constraints are often explicitly specified for TAMP methods. In contrast, we specify constraints as (learned) probabilities, which are baked into the decoding process and provided by domain-specific grounded models.

3 Grounded Decoding

3.1 LLMs and Grounding Models

Large Language Models. LLMs are trained to predict the probability p ( W ) of a text sequence W , represented as a sequence of tokens W = w 1: N = ( w 1 , . . . , w N ) . The tokens are elements of a fixed vocabulary W . Typical neural architectures factorize the joint probability into p ( W ) = ∏ N n =1 p LLM ( w n | w 1: n -1 ) , where p LLM is predicted by a transformer network [81]. Given p LLM, generating a text consisting of N -many tokens, the so-called decoding process, can be seen as the optimization problem arg max w 1: N ∈W ∏ N n =1 p LLM ( w n | w 1: n -1 ) , which in practice is solved, e.g., using greedy search , beam search, or sampling strategies. To further ensure the LLM is solving a desired task, one typically starts with a given text, the so-called prefix or prompt , that describes the task, and then the LLM completes this task in its decoding process.

Grounding Functions. We use the concept of grounding functions, p G ( w 1: n | s ) , which seek to model a probability of tokens w 1: n given (potentially non-textual) state s ∈ S . This state is intended to capture the embodiment of the robot and the environment, which may be an image, proprioception of the robot, or the environment state. Thus the grounding function models probabilities relevant to the robot embodiment and environment, such as whether the tokens are possible for the robot to execute given the state (affordances), or other values like safety, cost, or user preferences.

3.2 Problem formulation.

Given an instruction in language glyph[lscript] , we look at the problem of using an LLM to decode a language plan w 1: N , which is typically done by finding the most likely tokens under the probability distribution predicted by the LLM, p LLM ( w 1: N | glyph[lscript] ) , with glyph[lscript] being the prefix. However, based on the instruction glyph[lscript] as the prefix alone, the LLM can easily generate text that is not grounded in the physical state of the environment, rendering such plans useless in the real world. In order to ground the language model in an actual physical embodiment, we propose Grounded Decoding (GD): The main idea of GDis to guide the generation of token sequences with grounding function(s) that are conditioned on the embodiment of the system.

Formally, let s ∈ S denote a representation of the state of the world. Then, GD attempts to generate text that is consistent with both the instruction glyph[lscript] and the physical state s :

To leverage the Internet-scale knowledge of LLMs, we factorize p GD ( w 1: N | s, glyph[lscript] ) as follows 2 :

To decode autoregressively with the formulation, we factorize above into token decoding:

The first term, p LLM ( w n | w 1: n -1 , glyph[lscript] ) , can be modeled as the probability of the LLM predicting the token for the given instruction glyph[lscript] appended previously decoded tokens w 1: n -1 without the state s as input. The second term, p G ( w 1: n | s ) , is the grounding function that is only conditioned on the state s and judges whether the generated text w 1: n is consistent with the physical state. The core idea behind this factorization is that LLMs exhibit long-term planning capabilities, while the grounding function guides the planning of the LLM to be possible in the concrete embodied physical world without needing to be informed or capable of reasoning over the long-horizon instruction.

3.3 Grounded Decoding

This work investigates grounded decoding exclusively in the context of task planning for embodied agents. Figure 2 visualizes a single step of the simplest greedy search form of GD, and accompanying pseudo-code can be found in Algorithm 1. Given a high-level language instruction and history of executed actions, GD proceeds through a process similar to probabilistic filtering by selecting tokens iteratively that have high probability under the language model and the grounded model. After

each token is selected, it is appended to the prefix. The process continues until a token in the terminal set W term is selected, which could be a period sign '.' indicating the end of a single-step skill (e.g., pick-and-place). Then the command w 1: i is sent to a language-conditioned policy π ( a | s, w 1: i ) that executes the action a conditioned on the environment state s . Crucially, this grounding function must accept partial commands to enable grounding during decoding. 3 Additionally, we note that GD, in its essence, provides a grounded scoring function; thus, it can be easily extended to any search methods such as beam search, top-k sampling, etc.

Algorithm 1 Grounded Decoding (GD) w/ Greedy Search

3.4 Techniques for Obtaining Grounded Models

Unlike language tasks, where a single model is capable of performing general semantic reasoning, a singular grounded model remains an open problem. Indeed, each domain may impose varied environmental and embodiment constraints. Despite these challenges, we present several techniques for obtaining grounded models that can be leveraged in GD's formulation, and validate them in three domains in Section 4.

Token-Conditioned Value Functions. Assuming a robot acts with action a according to policy π ( a | s, w 1: n ) , that aims to maximize certain a utility and that the utility captures a task objective, a natural choice that can provide 'grounding score' is the action-value function Q ( s, a | w 1: n ) as it necessarily captures the embodiment of the robot. Additional objectives, such as task constraints, can also be encapsulated in Q ( s, a | w 1: n ) to ensure grounding. Note that unlike the formulation proposed in [2], w 1: n cannot be restricted to a fixed repertoire of token sequences. In practice, to obtain a Q ( s, a | w 1: n ) that satisfies the requirements, one can train multi-task language-conditioned agents, either through reinforcement learning (Section 4.2) or supervised learning (Section 4.1).

Multimodal Foundation Models. A general choice to ground LLMs is through using multimodal foundation models, such as CLIP [57] or open-vocabulary object detectors [22, 34, 50]. Although these models can connect language to other grounded modalities (e.g., vision), they often lack the capability for complex or long-horizon reasoning, and they do not consider embodiment constraints. As a result, to leverage them in the decoding process, they need to constrained to where they are the most applicable rather than always decoding jointly. To this end, we use a prompt-based technique that allows LLMs to choose when to jointly decode (Section 4.3), which we find to be effective in most cases. 4 .

Rule-based Methods. Another source of grounding may come from features x = φ ( w 1: n ) designed with expert domain knowledge, which can then be used to map w 1: n to a 'grounding score' using pamametric or non-parametric functions f ( x ) . Such techniques may be most applicable when interpretability and enforcing hard constraints are required, such as safety-critical settings, or when data are scarce, such as cases involving preferences of individual users (as shown in Section 4.1).

3.5 Comparisons to Prompt-based Methods

One alternative approach for grounding is including scene information as part of the prompt (e.g., object detection results [96]), which complements the grounding method proposed in this work.

Grounded Models

Combined Score

Step 1: Go to the purple door and open it

Step 2: Go to the green door and open it

Step 3: Go to the

purple

door and open it

Step 4: Go to the green door and open it

Step 5: Go to the red door and open it

Step 6: Go to the goal

Step 1: Pick up

Y

and place it on left side

Step 2: Pick up O and place it on right of Y

Step 3: Done

User

Spell as much of

'you' as you can.

Tabletop Rearrangement

User

Traverse the rooms

to get to the goal.

MiniGrid 2D Maze

Thought: I will bring you the [

pepsi

] and

the [apple]

Plan: Step 1: Find an apple

Step 2: Pick up the apple

Step 3: Bring it to you

Step 4: Put it down

…

User

Bring me a soda that is

not coke, and a fruit.

(Real)

100%

80%

%09

40%

20%

0%

100%

80%

60%

山

40%

20%

0%

purple

green

vellow

100%

80%

%09

40%

20%

0%

.

(-3.539740621266, 2.5450411438941956, 'lime soda'),

(-14.8108290220984, 2.7466341853141785, 'grapefruit'),

(-13.0752280220984, 0.0, 'grapefruit Soda')]

However, we note that prompting is often insufficent for grounding, as information about the scene and about the capabilities of the robot may not always be succinctly described in the prompt. Such examples include 1) in a block stacking task, a block that has been stacked on cannot be picked, 2) in a navigation task, to open a door, one must have a key and that door must be reachable, and 3) in a mobile manipulation domain, an object may be visible but is out of reach of the manipulator. Therefore, Grounded Decoding is a more general and flexible grounding method that injects continuous probabilities during decoding, which may even come from grounding functions from other modalities (e.g., vision).

4 Experiments

4.1 Long-Horizon Tabletop Manipulation

Herein we experiment with a simulated tabletop manipulation environment based on RAVENS [95]. We create a custom set of 20 tasks, with 10 seen tasks and 10 unseen tasks. Seen tasks are used for training (for supervised baseline) or for few-shot prompting. They are grouped by following categories. Detailed breakdown can be found in Appendix A.2.

Letters : Rearranging alphabetical letters ('sort the letters in alphabetical order').

Blocks & Bowls : Rearranging or combining blocks and bowls ('put blocks in matching bowls').

Box Packing : Sorting food items and utensils into boxes in accordance with safety constraints and user preferences ('Can you pack the picnic box for me?').

Given only high-level language instructions and top-down visual observation of the environment, Grounded Decoding decodes a sequence of text tokens representing the step command to be executed. Note that because GD generated grounded free-form actions, it does not require each step to strictly map to a repertoire of skill as in [29, 2]. After a complete command is generated, it is executed via a pre-trained multi-task language-conditioned CLIPort [67]. An example rollout is shown in Figure 4.To demonstrate the techniques proposed in Section 3.4 to obtain grounding functions, we study the composition of following grounding functions (overall grounding score is calculated as p G = ∏ n i =1 p i ) depending on the task categories. Refer to the Appendix A.2 for details.

Affordance Grounding Function (AF). As the primitive policy CLIPort [67] already acts as an action-value function over the pixel space, we directly leverage its predictions for affordance grounding. In particular, given scene image s and partially-decoded instruction w 1: n , CLIPort predicts unnormalized logits over the pixel space u pick , u place ∈ R 480 × 640 , respectively for the pick location and the place location. Therefore, for any given s and w 1: n , we can calculate the affordance as p AF ( w 1: n | s ) = max ( x,y ) ∈ 480 × 640 ( u pick ( x, y ) + u place ( x, y )) .

Safety Grounding Function (S). To strictly enforce hard constraints such as safety requirements, we adopt the rule-based method proposed in Section 3.4. In particular, the features x are indicator functions denoting whether knives and red boxes which we use as hazardous objects are involved in an action, i.e., x = I [ 'red' or 'knife' in w 1: n ] . We then use constant mappings to convert the features p S ( w 1: n | s ) = glyph[epsilon1] Z x + 1 Z (1 -x ) to scores of 0 or 1, where Z is the normalizing term and glyph[epsilon1] ≈ 0 is a small value for ensuring the joint probability does not collapse to 0.

Preference Grounding Function (P). We similarly use rule-based methods for preference grounding, as data of individual users may be scarce to learn a separate model. In particular, we choose two random objects ( o 1 , o 2 ) as the preferred objects, i.e., x = I [ o 1 or o 2 in w 1: n ] . Note that unlike safety functions, preferences often come in the form of 'soft requirement'. Therefore, the preference grounding function is implemented as p P ( w 1: n | s ) = α Z x + β Z (1 -x ) , where we choose α = 0 . 5 and β = 0 . 1 for our experiments.

Baselines. We study two variants of GD using beam search and greedy search. We also compare to 'No Grounding' baseline that decodes only according to language model likelihood. Furthermore, we compare to solitary method CLIPort [67] that directly take in the high-level language instructions without a planner. We consider two variants of CLIPort: 1) 'Short' that is trained with only singlestep pick-and-place commands, and 2) 'Long' that is trained on high-level instructions from the 10 training tasks. For more details, please refer to Appendix A.2.

Analysis. Results grouped by each task category are shown in Table 1 5 . Please refer to the Appendix for detailed breakdown. Each method is evaluated on 20 episodes for each task within each task category. Supervised methods, such as CLIPort, are found to perform poorly on unseen tasks. Methods that leverage language model planner show better generalization to unseen tasks but fall short due to lack of grounding. Grounded Decoding achieves the best results by enabling the LLM to plan actions using grounded information and is further improved with beam search.

4.2 2D Maze

We further evaluate the long-horizon reasoning of Grounded Decoding for 2D maze-solving on Minigrid [10]. The agent receives a top-down view of the environment along with a natural language instruction. More details can be found in Appendix A.3. The tasks are grouped in three categories:

Easy : Simple tasks where the horizon is short (10-30 steps) and fully described by the textual instruction, e.g. OpenDoors and PutNext .

Medium : Short and long-horizon tasks (up to 80 steps) with step-by-step textual instructions, e.g. LockedRoom .

iii. Hard : Complex, long-horizon instructions (over 100 steps) with ambiguous instructions that necessitate multi-step reasoning and efficient exploration, e.g. MultiRoom and BlockedUnlock .

Step 1: Pick up the yellow block and place it on the purple block.

Step 2: Pick up the red block and place it on the yellow block.

User

Stack all the blocks.

Grounded Model (Affordance)

Combined Score

109%

100%

100%

00%

8

Affordance Grounding Function. Following the recipe from Section 3.3, we train tokenconditioned affordance function to be used in GD. The difference is that the grounding function here is the value function from the goal-conditioned policy that is trained with PPO [62] instead of from demonstrations as in CLIPort [67]. The policy performs short-horizon skills such as 'Go to red key' or 'Open the door' and are conditioned on CLIP embeddings of the skill and an image of the scene. Accordingly, the goal-conditioned value function evaluates the feasibility given the current observation and the (partially) decoded skill.

Baselines. We compare the two variants of GD - with greedy and beam search - with 1) a solitary PPO policy [62], 2) a hierarchical RL algorithm which plans over the low-level skills, and 3) a hierarchical method that uses an ungrounded language model for planning [29].

Analysis. Table 2 reports the success rate, averaged across 100 episodes of randomly initialized environments. The 'flat' RL agent performs poorly in all but the simplest environments, owing to difficulties with understanding the high-level instruction and reasoning over long horizons (often over 100 steps). Planning over low-level skills using hierarchical RL [5] improves this performance, since the high-level decision-making problem is greatly simplified. However, the high-level RL agent still needs to reason over low-level (textual) skills by understanding their underlying capabilities and stitching them together. Using the planning capabilities of large language models to reason over textual skills significantly boosts this performance [29], since the language model can inherit the strong reasoning capabilities from its training data. This tends to be insufficient in challenging environments, however, since the number of potentially viable skills may be very large and the LLM has no information about the robot's observations. GD can leverage the learned affordance function (in this case, the goal-conditioned value function) to inform the language model's plans, enabling successful long-horizon reasoning. We further find that beam search improves performance modestly, particularly in long-horizon tasks.

4.3 Mobile Manipulation in a Physical Kitchen

Our last environment is a kitchen robot in the real world, and we follow the same implementations of the mobile manipulation platform and skills in SayCan [2]. We perform instruction following tasks, as in [2]. An example task is 'Bring an apple', for which the robot needs to plan and execute a sequence of '1. Find an apple, 2. Pick up the apple, 3. Bring it to you. 4. Put it down, 5. Done'. We split the tasks into two categories. Unambiguous means the instruction explicitly contains the object of interest, and Ambiguous means the instruction does not contain the object name. For example, when human asks 'bring me the fruit', the robot needs to first determine available fruits. We assume all necessary objects are in the field of view. More details can be found in Appendix A.4.

Grounded Decoding with Chain-of-thought. We demonstrate using multimodal foundation models for Grounded Decoding, as proposed in Section 3.4. In particular, we use open-vocabulary object detector owl-vit [50]. Note that because these off-the-shelf models are not trained on robot domain data, we find that it works best by constraining their influence on decoding. We achieve this by making a slight modification to the SayCan algorithm [2]: before generating action plans, we prompt the LLM to generate visually-grounded chain-of-thought [84] by giving LLM the option of when to enable grounded decoding and disable grounded decoding, as visualized in Fig. 5. Specifically,

Robot: I am a robot that can bring objects to you.

Human: I want a soda that is not coke, and a fruit.

Robot thought: I will find the [pepsi] and the [apple]

Robot plan: 1. Find an apple

2. Pick up the apple

3. Bring it to you

4. ...

grapefruit

grapefruit Soda

grapefruit soda

lime soda

sponge

banana

Ungrounded decoding

Grounded decoding

SayCan scoring mode

1.Find an apple

2.Pick up the apple

5.Find a pepsi

6.Pick up the pepsi

8.Put it down

…

…

LLMs can be prompted to generate a left bracket to start decoding jointly with grounded models and a right bracket to revert to ungrounded decoding. After chain-of-thought, we use SayCan scoring mode for decoding the action plans.

Analysis. Table 3 shows that GD recovers similar performance on Unambiguous tasks, and gain 25% in planning performance on Ambiguous tasks. This shows that GD with multimodal foundation models can effectively use visually-grounded chain-of-thought to disambiguate abstract tasks.

5 Analysis

5.1 Comparison to SayCan

In this section, we directly compare GD to SayCan [2], which is related to our method in that both combine language model knowledge and grounded model knowledge (discussed in more detail in Section 2). However, SayCan uses the language model to score all pre-specified options, rendering it inefficient at dealing with large or combinatorial action spaces. In contrast, GD computation considers all possible language token in the autoregressive decoding process, which is independent of the size of the action space. Results shown in Table 4 demonstrate that GD is two orders of magnitude more efficient on our tasks, with comparable performance. Furthermore, by decoding at the most basic functioning unit of language, GD's formulation allows open-vocabulary grounding beyond just affordances, e.g. safety, preferences, and multimodal embeddings such as CLIP.

5.2 Breakdown of Failure Reasons

Because all hierarchical approaches share an imperfect low-level policy for step execution, the results reported in Table 1 are compounded with both planning failures and low-level policy failure. In Figure 6, we provide failure breakdown analysis for Grounded Decoding and associated baselines. Note that the CLIPort baselines are solitary methods that do not use a planner, so the failures are solely composed of policy failures. As shown in Figure 6, while all planning-based methods use the same underlying low-level policy, Grounded Decoding significantly reduces planning failure by being able to incorporate grounded scene information into the decoding process. Moreover, we observe that despite the shared affordance function across beam and greedy search, the beam search variant performs stronger by being aware of the full-length single-step instructions during decoding.

5.3 Grounded Action Manifold

A central goal of this work is to investigate the integration of grounded information into language model decoding to output instructions actionable by a policy. To investigate this, we use a t-SNE [80] plot to illustrate the extent to which grounded models help narrow down the search space for language models. Specifically, we first enumerate all meaningful instructions in the tabletop domain, such as 'pick x and place it on y,' which are represented as dots in the figure. We then compute the affordance values with respect to four different scenes, where each color represents one scene. Finally, we group the dots using t-SNE and BERT embeddings [13]. Figure 7 shows that grounded

GD (Beam)

GD (Greedy)

No Grounding

CLIPort (long)

CLIPort (short)

0%

25%

50%

75%

100%

Policy Failure

Planner Failure

No Failure

Pick up the white block and

Put the apple in the box

place it in the white bowl

Put the A to the right of the W

models can effectively identify achievable skills to produce an actionable manifold within the language space and that this grounding is required, as language alone does not perfectly group actionable skills. It is worth noting that while we provide manual enumeration of all possible skills for practical analysis, the full language space is much larger. This highlights the even more pronounced narrowing of the search in the language space.

6 Conclusions, Limitations, & Future Works

We presented Grounded Decoding (GD), an approach for leveraging the knowledge and capabilities of large language models in embodied settings through grounding functions, which model the probabilities of tokens given an embodiment. GD resembles probabilistic filtering, by decoding tokens that have high probabilities under the language model and under grounded model(s). By guiding the LLM's decoding directly at its output, GD is a general, flexible, and expressive approach to embodied tasks. This is demonstrated on three embodied domains, showing GD is capable of solving complex, long-horizon tasks.

Though quite general and flexible, GD has a few limitations. First, although we present several techniques for obtaining grounding functions in different domains, it remains a question whether a capable and general grounding function can be obtained. We hope that recent progress in largescale robotics models (e.g. [7] and [59]) can remove this bottleneck, and note that the flexibility of GD allows such progress to be straightforwardly leveraged. Second, prompt engineering is often required to steer LLMs to the desired action space (e.g., likely action verbs, likely present objects). Finally, while not requiring additional training, the joint decoding may be limiting compared to a single model capable of both grounding and language reasoning [3, 9, 16].

This work presented a family of algorithms for grounding LLMs in embodiment, for which there are many avenues for future work. The flexibility of the approach enables many other grounding functions and ways to integrate grounding. Furthermore, the development and integration of a foundation model for grounding would improve performance significantly. Finally, though GD's probabilistic filtering-based approach is quite general, fusing grounding information to the language model after each token decoding may be limiting and future works can investigate how such grounding can be elegantly integrated during decoding.

Acknowledgments

The authors would like to acknowledge Pierre Sermanet, Carolina Parada, Jie Tan, Yevgen Chebotar, Vincent Vanhoucke, and Dorsa Sadigh for their feedback and contributions. This work is supported in part by OpenAI academic access program, granted to Wenlong Huang.

References

Christopher Agia, Toki Migimatsu, Jiajun Wu, and Jeannette Bohg. Stap: Sequencing taskagnostic policies. arXiv preprint arXiv:2210.12250 , 2022.

Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, and Mengyuan Yan. Do as i can and not as i say: Grounding language in robotic affordances. In arXiv preprint arXiv:2204.01691 , 2022.

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198 , 2022.

Ashutosh Baheti, Alan Ritter, Jiwei Li, and Bill Dolan. Generating more interesting responses in neural conversation models with distributional constraints. arXiv preprint arXiv:1809.01215 , 2018.

Andrew G. Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete Event Dynamic Systems , 13(1-2):41-77, jan 2003.

Sourya Basu, Govardana Sachitanandam Ramachandran, Nitish Shirish Keskar, and Lav R Varshney. Mirostat: A neural text decoding algorithm that directly controls perplexity. arXiv preprint arXiv:2007.14966 , 2020.

Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817 , 2022.

Boyuan Chen, Fei Xia, Brian Ichter, Kanishka Rao, Keerthana Gopalakrishnan, Michael S Ryoo, Austin Stone, and Daniel Kappler. Open-vocabulary queryable scene representations for real world planning. arXiv preprint arXiv:2209.09874 , 2022.

Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794 , 2022.

Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for gymnasium, 2018.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022.

Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. arXiv preprint arXiv:1912.02164 , 2019.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Yan Ding, Xiaohan Zhang, Chris Paxton, and Shiqi Zhang. Task and motion planning with large language models for object rearrangement. arXiv preprint arXiv:2303.06247 , 2023.

Danny Driess, Ozgur Oguz, Jung-Su Ha, and Marc Toussaint. Deep visual heuristics: Learning feasibility of mixed-integer programs for manipulation planning. In 2020 IEEE International Conference on Robotics and Automation (ICRA) , pages 9563-9569. IEEE, 2020.

Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 , 2023.

Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement learning. Advances in Neural Information Processing Systems , 2019.

Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. arXiv preprint arXiv:1805.04833 , 2018.

Richard E Fikes and Nils J Nilsson. Strips: A new approach to the application of theorem proving to problem solving. Artificial intelligence , 1971.

Caelan Reed Garrett, Chris Paxton, Tom´ as Lozano-P´ erez, Leslie Pack Kaelbling, and Dieter Fox. Online replanning in belief space for partially observable task and motion problems. In 2020 IEEE International Conference on Robotics and Automation (ICRA) , 2020.

Marjan Ghazvininejad, Xing Shi, Jay Priyadarshi, and Kevin Knight. Hafez: an interactive poetry generation system. In Proceedings of ACL 2017, System Demonstrations , pages 43-48, 2017.

Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.13921 , 2021.

Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. arXiv preprint arXiv:2203.09509 , 2022.

Felix Hill, Sona Mokra, Nathaniel Wong, and Tim Harley. Human instruction-following with deep reinforcement learning via transfer-learning from text. arXiv preprint arXiv:2005.09382 , 2020.

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751 , 2019.

Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. Learning to write with cooperative discriminators. arXiv preprint arXiv:1805.06087 , 2018.

Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard. Visual language maps for robot navigation. arXiv preprint arXiv:2210.05714 , 2022.

De-An Huang, Suraj Nair, Danfei Xu, Yuke Zhu, Animesh Garg, Li Fei-Fei, Silvio Savarese, and Juan Carlos Niebles. Neural task graphs: Generalizing to unseen tasks from a single video demonstration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019.

Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning . PMLR, 2022.

Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models. In arXiv preprint arXiv:2207.05608 , 2022.

Brian Ichter, Pierre Sermanet, and Corey Lynch. Broadly-exploring, local-policy trees for long-horizon task planning. Conference on Robot Learning (CoRL) , 2021.

Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning , pages 991-1002. PMLR, 2021.

Leslie Pack Kaelbling and Tom´ as Lozano-P´ erez. Hierarchical planning in the now. In Workshops at the Twenty-Fourth AAAI Conference on Artificial Intelligence , 2010.

Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1780-1790, 2021.

Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Dragomir Radev, Yejin Choi, and Noah A Smith. Beam decoding with controlled patience. arXiv preprint arXiv:2204.05424 , 2022.

Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858 , 2019.

Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. Gedi: Generative discriminator guided sequence generation. arXiv preprint arXiv:2009.06367 , 2020.

R´ emi Leblond, Jean-Baptiste Alayrac, Laurent Sifre, Miruna Pislar, Jean-Baptiste Lespiau, Ioannis Antonoglou, Karen Simonyan, and Oriol Vinyals. Machine translation decoding beyond beam search. arXiv preprint arXiv:2104.05336 , 2021.

Jiwei Li, Will Monroe, and Dan Jurafsky. Learning to decode for future success. arXiv preprint arXiv:1701.06549 , 2017.

Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 , 2019.

Shuang Li, Xavier Puig, Yilun Du, Clinton Wang, Ekin Akyurek, Antonio Torralba, Jacob Andreas, and Igor Mordatch. Pre-trained language models for interactive decision-making. arXiv preprint arXiv:2202.01771 , 2022.

Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:2209.07753 , 2022.

Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. Text2motion: From natural language instructions to feasible plans. arXiv preprint arXiv:2303.12153 , 2023.

Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Llm+ p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304.11477 , 2023.

Yujie Lu, Pan Lu, Zhiyu Chen, Wanrong Zhu, Xin Eric Wang, and William Yang Wang. Multimodal procedural planning via dual text-image prompting. arXiv preprint arXiv:2305.01795 , 2023.

Corey Lynch and Pierre Sermanet. Grounding language in play. 2020.

Oier Mees, Jessica Borja-Diaz, and Wolfram Burgard. Grounding language with visual affordances over unstructured data. arXiv preprint arXiv:2210.01911 , 2022.

Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. Typical decoding for natural language generation. arXiv preprint arXiv:2202.00666 , 2022.

Clara Meister, Tim Vieira, and Ryan Cotterell. If beam search is the answer, what was the question? arXiv preprint arXiv:2010.02650 , 2020.

Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. Simple open-vocabulary object detection with vision transformers. arXiv preprint arXiv:2205.06230 , 2022.

Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. In Advances in Neural Information Processing Systems , 2018.

Suraj Nair, Eric Mitchell, Kevin Chen, Brian Ichter, Silvio Savarese, and Chelsea Finn. Learning language-conditioned robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning , pages 1303-1315. PMLR, 2021.

OpenAI. Gpt-4 technical report. arXiv , 2023.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 , 2022.

Alexander Pashevich, Cordelia Schmid, and Chen Sun. Episodic transformer for vision-andlanguage navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021.

Dhruvesh Patel, Hamid Eghbalzadeh, Nitin Kamra, Michael Louis Iuzzolino, Unnat Jain, and Ruta Desai. Pretrained language models as visual planners for human assistance. arXiv preprint arXiv:2304.09179 , 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 8748-8763. PMLR, 18-24 Jul 2021.

Shreyas Sundara Raman, Vanya Cohen, Eric Rosen, Ifrah Idrees, David Paulius, and Stefanie Tellex. Planning with large language models via corrective re-prompting. arXiv preprint arXiv:2211.09935 , 2022.

Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175 , 2022.

Machel Reid, Yutaro Yamada, and Shixiang Shane Gu. Can wikipedia help offline reinforcement learning. arXiv preprint arXiv:2201.12122 , 2022.

Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi-parametric topological memory for navigation. arXiv preprint arXiv:1803.00653 , 2018.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. Discriminative adversarial search for abstractive summarization. In International Conference on Machine Learning , pages 8555-8564. PMLR, 2020.

[64]

Blazej igation

with large

Osinski, pre-trained

arXiv:2207.04429

, 2022.

Dhruv Shah, Peng Xu, Yao Lu, Ted Xiao, Alexander Toshev, Sergey Levine, and Brian Ichter. Value function spaces: Skill-centric state abstractions for long-horizon reasoning. ICLR , 2022.

Pratyusha Sharma, Antonio Torralba, and Jacob Andreas. Skill induction and planning with latent language. arXiv preprint arXiv:2110.01517 , 2021.

Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning , pages 894-906. PMLR, 2022.

Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Proceedings of the 6th Conference on Robot Learning (CoRL) , 2022.

Ichter, models

of and

Sergey language,

vision, and

Lm-nav:

action.

Robotic arXiv

nav- preprint

Tom Silver, Rohan Chitnis, Nishanth Kumar, Willie McClinton, Tomas Lozano-Perez, Leslie Pack Kaelbling, and Joshua Tenenbaum. Inventing relational state and action abstractions for effective and efficient bilevel planning. arXiv preprint arXiv:2203.09634 , 2022.

Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. arXiv preprint arXiv:2209.11302 , 2022.

Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural language generation with implicit language q learning. arXiv preprint arXiv:2206.11871 , 2022.

Charlie Snell, Sherry Yang, Justin Fu, Yi Su, and Sergey Levine. Context-aware language modeling for goal-oriented dialogue systems. arXiv preprint arXiv:2204.10198 , 2022.

Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. Llm-planner: Few-shot grounded planning for embodied agents with large language models. arXiv preprint arXiv:2212.04088 , 2022.

Yixuan Su, Tian Lan, Yahui Liu, Fangyu Liu, Dani Yogatama, Yan Wang, Lingpeng Kong, and Nigel Collier. Language models can see: plugging visual controls in text generation. arXiv preprint arXiv:2205.02655 , 2022.

Alessandro Suglia, Qiaozi Gao, Jesse Thomason, Govind Thattai, and Gaurav Sukhatme. Embodied bert: A transformer model for embodied, language-guided visual task completion. arXiv preprint arXiv:2108.04927 , 2021.

Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2019.

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems , 27, 2014.

Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf. Zero-shot image-to-text generation for visual-semantic arithmetic. arXiv preprint arXiv:2111.14447 , 2021.

Marc Toussaint. Logic-geometric programming: An optimization-based approach to combined task and motion planning. In Twenty-Fourth International Joint Conference on Artificial Intelligence , 2015.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. Chatgpt for robotics: Design principles and model abilities. 2023 , 2023.

Siddharth Verma, Justin Fu, Mengjiao Yang, and Sergey Levine. Chai: A chatbot ai for taskoriented dialogue with offline reinforcement learning. arXiv preprint arXiv:2204.08426 , 2022.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 , 2022.

Sean Welleck, Ilia Kulikov, Jaedeok Kim, Richard Yuanzhe Pang, and Kyunghyun Cho. Consistency of a recurrent language model with respect to incomplete decoding. arXiv preprint arXiv:2002.02492 , 2020.

Bohan Wu, Suraj Nair, Li Fei-Fei, and Chelsea Finn. Example-driven model-based reinforcement learning for solving long-horizon visuomotor tasks. arXiv preprint arXiv:2109.10312 , 2021.

Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 , 2016.

Fei Xia, Chengshu Li, Roberto Mart´ ın-Mart´ ın, Or Litany, Alexander Toshev, and Silvio Savarese. Relmogen: Integrating motion generation in reinforcement learning for mobile manipulation. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , 2021.

Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, and Harold Soh. Translating natural language to planning goals with large-language models. arXiv preprint arXiv:2302.05128 , 2023.

Danfei Xu, Ajay Mandlekar, Roberto Mart´ ın-Mart´ ın, Yuke Zhu, Silvio Savarese, and Li FeiFei. Deep affordance foresight: Planning through what can be done in the future. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , pages 6206-6213. IEEE, 2021.

Danfei Xu, Suraj Nair, Yuke Zhu, Julian Gao, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Neural task programming: Learning to generalize across hierarchical tasks. In 2018 IEEE International Conference on Robotics and Automation (ICRA) , 2018.

Kevin Yang and Dan Klein. Fudge: Controlled text generation with future discriminators. arXiv preprint arXiv:2104.05218 , 2021.

Haoqi Yuan, Chi Zhang, Hongcheng Wang, Feiyang Xie, Penglin Cai, Hao Dong, and Zongqing Lu. Plan4mc: Skill reinforcement learning and planning for open-world minecraft tasks. arXiv preprint arXiv:2303.16563 , 2023.

Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. Advances in Neural Information Processing Systems , 2021.

Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, and Johnny Lee. Transporter networks: Rearranging the visual world for robotic manipulation. Conference on Robot Learning (CoRL) , 2020.

Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598 , 2022.

A Appendix

A.1 Grounded Decoding Implementation Details

We study three different implementations of Grounded Decoding for each of the experimental domains. While each instantiation applied Grounded Decoding to long-horizon planning and behavior synthesis, different components including language models and grounded models are used in each domain, as seen in Table 5. Grounded models used in these domains include Affordance Functions (AF), Safety Functions (S), Preference Functions (P), and Open-Vocabulary Object Detectors (D).

A.2 Implementation Details of Simulated Tabletop Rearrangement

A.2.1 Tasks

There are a total of 20 tasks (templates of language instructions), listed in Table 6, grouped into three task category: Letters, Blocks&Bowls, and Box Packing. Three categories share a total of 57 objects. For Letters category, the goals are to rearrange the alphabetical letter objects such that they satisfy certain orders specified by the language instructions. At the beginning of each episode, task-relevant objects and a set of 1 to 3 randomly-sampled distractor objects (except for the Letters category) are initialized at random positions on the tabletop with fixed orientations. A minimum 15cm distance is enforced between any two objects to avoid collision and penetration at initialization. To allow for automatic evaluations, a binary reward function is defined for each task using ground-truth state of the objects. Furthermore, we implement scripted policies for each task to collect demonstrations for training the CLIPort baseline. For certain tasks, we also randomize the attributes mentioned in the given instructions, which can be found below:

i. 〈 word 〉 : hi, world, left, right, top, down, love, you ii. 〈 corner/side 〉 : left side, top left corner, top side, top right corner, bottom right corner, bottom side, bottom left corner

A.2.2 Low-level Primitives

We use CLIPort [67] as the low-level primitive that can be invoked by the LLM planner, as it shows promising results of generalization across free-form language instructions. Additionally, since the policy predicts per-pixel affordance, it can be repurposed to serve as grounded models for planning for long-horizon tasks, which we leverage in this work. The single primitive policy is trained on 50,000 pre-collected demonstrations, across 10 training tasks, where each demonstration contains 1) language instruction of the format 'pick up [x] and place it on [y]', 2) top-down RGB-D observation of the current scene, 3) expert pick location expressed as pixel coordinates, and 4) expert place location expressed as pixel coordinates. The expert actions are obtained by accessing ground-truth object pose in the simulator. We further apply substring augmentation during training as we find it helps with performance on partial commands (see Section

A.2.3 Language Model

We use InstructGPT [54] ( text-davinci-002 ), accessed through OpenAI API.

A.2.4 CLIPort Baseline

As CLIPort [67] already takes as input a natural language instruction and is capable of directly outputting robot actions, it bears the question whether we need a high-level planner for completing

long-horizon tasks. To this end, we additionally train two variants of multi-task CLIPort policy on 10 of the total 20 tasks as baselines (see Table 6 for the train/test split). One variant, which we referred as 'CLIPort (Short)', is trained only on single-step pick-and-place instructions of the format 'pick up [x] and place it on [y]' on the 10 training tasks. The decomposed pick-and-place instructions are obtained from scripted planners. At evaluation time, the policy is fed in only the high-level instructions without any planners. The other variant, which we referred as 'CLIPort (Long)', is trained on the high-level instructions from the 10 training tasks (without decomposition from scripted planners). Similarly, at evaluation time, it is fed in only the high-level instructions and evaluated on both seen and unseen instructions. Both variants are trained on 50,000 demonstrations, similar to the Grounded Decoding primitive. The goal of these baselines is to evaluate whether solitary language-conditioned policies can perform well on long-horizon tasks and generalize to new task instructions. Note that the CLIPort baselines are different from the primitive used in Grounded Decoding, although they share the same architecture.

A.2.5 Full Experimental Results in Simulated Tabletop Domain

Below we show the full list of tasks and the full experimental results in the simulated tabletop domain. Each entry is the average success rate across 20 rollouts. The tasks with blue-colored background are seen tasks and the tasks with orange-colored background are the unseen tasks. Seen tasks may be used for training for supervised baselines (CLIPort) or may be used in the prompt for methods using language model planner. Note that for the 'Box Packing' task category, although all tasks were seen in training or the prompts, we enforce additional safety and preference constraints for evaluation only at test time.

A.3 Implementation Details of Minigrid 2D Maze

A.3.1 Environment Setup

We use the open-source gym-minigrid suite of environments to evaluate our method with one simple change - instead of the default observation space which is a 7 × 7 egocentric window, our agent has access to entire grid -that allows us to simplify the tasks by removing partial observability [65].

A.3.2 Tasks

The tasks are grouped in three categories (please see Table 7 for example instructions):

Easy : Simple tasks where the horizon is short (10-30 steps) and fully described by the textual instruction, e.g. OpenDoors and PutNext . The short horizon makes them relatively easy for a wide range of HRL algorithms. The instructions for these tasks generally spell out each individual skill, making them particularly easy for high-level planners based on language modeling.

Medium : Combination of short and long horizon tasks (up to 80 steps) with step-by-step textual instructions, e.g. LockedRoom . While being significantly longer, these tasks also tend to have instructions that spell out the low-level tasks (see Table 7).

Hard : Complex, long horizon instructions (over 100 steps) with short, ambiguous instructions that necessitate multi-step reasoning and efficient exploration, e.g. MultiRoom and BlockedUnlock . In addition to being long-horizon, the instructions in this case tend to be ambiguous and under-specified, e.g. 'traverse through the rooms to get to the goal', which does not provide enough context for any blind planning agent.

A.3.3 Language Model

We use InstructGPT [54] ( text-davinci-002 ), accessed through OpenAI API. The prompts used can be found in Section A.5.

We found the prompts to be generally sufficient for solving the 'seen' tasks, as well as 'unseen' tasks, i.e. tasks that do not have an example in the context. Empirically, we did not find any improvements by including more then 3 example tasks in the prompt - we hypothesize that this is likely due to the shared low-level primitives across tasks. For all Minigrid experiments presented in this paper, we used the prompt shown in Section A.5.

A.3.4 Low-level Primitives

To train low-level primitives, we train an RL agent to solve a wide range of short-horizon subtasks (under 10 steps) that are shared across the various Minigrid tasks go to <obj> , pick up <obj> , drop <obj> , open <obj> . Rather than training individual skills for each of them [65], we train a single multi-task policy that is conditioned on the CLIP embeddings [57] of the task strings. This scheme allows some robustness to synonyms and ambiguous task specifications, and has been widely used in learning language-grounded policies [32, 68].

We train these primitives using PPO [62], as recommended by the environment developers [10]. Each of these skills are trained with a sparse outcome reward (+1 if a trajectory is successful, 0

otherwise). In addition to these low-level skills, we perform a form of hindsight relabeling where 'substrings' of the task strings are masked to allow generalization to partial strings, e.g. 'go to red' may be interpreted as 'go to red key' or 'go to red door', and our masking strategy allows the multi-task policy to execute tasks specified by partially complete strings, if necessary.

A.3.5 Additional Qualitative Results

Step 1: go to the purple door and open it

Step 2: go to the green door and open it

Step 3: go to the

purple

door and open it

Step 4: go to the green door and open it

Step 5: go to the green door and open it

Step 6: go to the goal

User

Traverse the

rooms to get

to the goal.

Grounded Model (Affordance)

Combined Score

Step 1: go to the purple door and open it

Step 2: go to the

green

key

Step 3: pick up the green key

Step 4: go to the green door and open it

Step 5: go to the goal

Get the green

key from the

purple room,

unlock the

green door and

go to the goal.

Step 1: go to the

obstacle

Step 2: pick up the obstacle

Step 3: place the obstacle

Step 4: go to the blue key

Step 5: pick up the blue key

Step 6: go to the blue door and open it

Step 8: drop the blue key

Step 9: go to the purple box

Step 10: pick up the purple box

Pick up the

purple box.

User

User

100%

80% -

60%

1

40%

20%

0%

purple

greenred

yellow

100%

80%

60%

40%

20%

0%

qreenpurpledoorroomgoal

100%

80% -

%09

40%

20%

0%

blue

purple

doo

A.4 Implementation Details of Real-World Mobile Manipulation

A.4.1 Tasks

Instruction

put an energy bar and water bottle on the table

bring me a lime soda and a bag of chips

Can you throw away the apple and bring me a coke

bring me a 7up can and a tea

move an multigrain chips to the table and an apple to the far counter

move the lime soda, the sponge, and the water bottle to the table

bring me an apple, a coke, and water bottle

A.4.2 Language Model

For planning, we use PaLM [11], a 540B parameter language model trained on a large datasets that include high-quality web documents, books, Wikipedia, conversations, and GitHub code. Before planning, we use InstructGPT [54] ( text-davinci-002 ), accessed through OpenAI API. to generate the (grounded) chain of thought.

We used square bracket to indicate grounded decoding, as illustrated in Fig. 5. The prompts are shown in Listing 3.

A.4.3 Low-level Primitives

We use a combination of learned and scripted control policies for navigation and manipulation, following the implementation described in SayCan [2] and RT-1 [7]. The manipulation policies for the picking action are learned using Behavior Cloning (BC) on 68000 demonstrations and 12000 autonomous successes that were collected over the course of 11 months using a fleet of 10 robots. The demonstrations are collected by teleoperators using VR headset controllers to track the motion of their hand, which is then mapped onto the robot's end-effector pose. The navigation policies are scripted, based on a ground-truth map as well as a learned perception module for collision avoidance and planning. The placing actions follow pre-computed motions only when preceded by a navigation policy. The Value Functions used by SayCan for affordance grounding are provided by the Q -networks of trained RL agents; we follow the RL training setup described in [2].

A.4.4 Open-Vocabulary Detector Grounding Function

We use owl-vit [50] as our grounding model. It takes in an image and a natural language query, and returns a list of bounding boxes with scores. We take the maximum score a the grounding function.

More examples of object detection as a grounding function can be found in Fig. 9.

papertowel.16

Human: I want a soda that is not coke, and a fruit. Robot thought: I will find the [pepsi] and the [apple]

A.5 Prompts

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel method to tackle a key challenge in utilizing Internet-scale knowledge from LLMs (Language Model Models) in robotics tasks. Specifically, the focus is on aligning the instructions generated by LLMs with the actual actions performed to manipulate objects. The paper endeavors to frame this issue in a manner analogous to probabilistic filtering. This framing aims to decode a sequence that exhibits both high probability according to the language model and high probability according to a predefined set of objectives or attainable skills in the grounded model.\n\nSTRENGTHS:\nThe paper introduces a novel formulation focused on decoding language models in a robot-centric manner to facilitate long-horizon robotics tasks using token-conditioned grounded models. Additionally, the paper provides empirical evidence showcasing the efficacy of this approach in comparison to CLIPort, which utilizes CLIP as an open vocabulary knowledge-based grounding technique combined with the Transporter Network. Furthermore, the paper demonstrates the transfer-ability of this idea to various domains, including 2D mazes and mobile manipulation.\n\nWEAKNESSES:\n1.The paper could benefit from improvements in certain sections of the writing. For instance, there is an awkward expression in the phrasing of the sentence on lines 116-117.\n2.It would be valuable if the paper included a comparison with other techniques, such as Grounded SAM, which involves detecting objects in the scene and using them as input for LLMs to enable grounded reasoning. This alternative baseline would provide a more robust comparison for the proposed approach, as it still allows for the use of a planner to generate the task plan.\n3.The author's efforts to demonstrate the effectiveness of their approach across different domains are appreciated. However, it would be advantageous if the author conducted real-world experiments that are reproducible with a broader range of robots and baselines.\n\nQUESTIONS:\n1. Would like to see comparison with other methods using detection to learn the labels of the object, and feeding those into the LLMs for grounded reasoning.\n2. Implementation on more general robot framework for reproducible.\n\nLIMITATIONS:\nThe authors have address the limitations adequately.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Guiding Text Generation with Grounded Models for Robot Control. Previous works have shown that large pre-trained language models can generate long action plans for robots based on abstract instructions. However, these models struggle to handle unstructured environments with large action spaces. The project aims to enable language models to generate control commands in a grounded and efficient manner. \n\nThe paper explains how auto-regressive language models generate text by scoring all possible tokens based on their likelihood of completing the current action step. \n\nThe approach incorporates affordance, preference, and safety functions to create grounded models. The decision-making process of selecting tokens is illustrated using the example of stacking blocks. The work also demonstrates how the method can generalize to different tasks without additional training. The flexibility of grounded decoding allows for including objective functions and constraints on the fly. \n\nThe project concludes by discussing the potential applications of grounded decoding in other embodied tasks and the possibility of using advanced search methods for better performance.\n\nSTRENGTHS:\nThe work demonstrates its strength through plenty of experiments. By conducting numerous experiments (Embodied environment, maze, robot control in real space), the researchers provide a comprehensive evaluation of the proposed method and its performance in various scenarios. This enhances the robustness of the findings and allows for a better understanding of the method's capabilities and limitations.\n\nThe work takes into consideration the environmental conditions during planning. By incorporating affordance functions, preference functions, and safety functions, the grounded models consider the feasibility, preferences, and safety aspects of actions within the given environment. This ensures that the generated action plans are not only coherent with the high-level goals but also take into account the specific constraints and requirements of the environment, leading to more practical and context-aware robot control.\n\nThe key idea of the grounded model is well-illustrated. Through clear and concise explanations supported by visual representations and examples. By effectively communicating the core concept of the grounded model, it is easier to understand how the model works and how it addresses the challenges of generating control commands in a grounded and efficient manner. The illustrations help to visualize the decision-making process and the interaction between the language model and the affordance model, contributing to a more intuitive grasp of the proposed approach.\n\nWEAKNESSES:\nEven though the paper is titled with `Robot Control`, it lacks discussion of the robot motion planning considering the environment; i.e. how the robot motion planning changes when involving the grounded model. Especially, comparing the robot motion planning between Saycan and this work can be more persuasive.\n\nThe motivation for involving grounded information is not sufficient. It seems that the grounded model is a direct fusion of affordance map derived by CLIPort, while the whole system is equipped with template language instructions. A more straightforward approach would be using CV model to detect the object in the scene and then include the scene description in the prompt. For example: pick up the fruit in the scene, given `orange, apple, potato, ...`. \n\nThe key idea for considering the grounded map should be that the robot motion needs to think about its configuration, safety, environmental limitations, etc. However, those facts seemed to be disregarded.\n\nQUESTIONS:\n(1) What is the task size and the robot search space?\n\n(2) Why the 2D Maze experiment matters since it is unrelated to robot control\n\nLIMITATIONS:\nThe limitations are mentioned;\nNo societal impacts found", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies grounding Large Language Models (LLMs) for robot control. The planning tokens are generated by the joint probability of an LLM and grounded models sequentially. The grounded models can be RL value functions, multi-modal perception models or human rules. As an extension of SayCan, this method allows open-vocabulary control and combining multiple grounded models. Experimental results in two robotic environments and a maze environment demonstrate the effectiveness of the proposed method for solving long-horizon complex tasks.\n\nSTRENGTHS:\n1. The paper is well-written, with relevant references and easy-to-follow narration.\n\n2. The paper conducts experiments in various domains and tasks, showing that the proposed grounded decoding method can be widely applied in robotic tasks.\n\nWEAKNESSES:\n1. The contribution of this paper is mainly incremental to SayCan. Though it allows open-vocabulary planning, the key difference is that SayCan requires a fixed set of low-level skills while this paper requires a language-conditioned low-level policy.\n\n2. For the proposed 3 types of grounded models: token-conditioned value functions are similar to the skill affordance in SayCan. Multi-modal foundation models and rule-based methods are more like technical tricks in applications rather than a systematic method.\n\nQUESTIONS:\n1. For the results in Table 3 and 4, does SayCan use the same LLM to your method?\n\n2. According to Table 4, SayCan costs a large number of tokens since it computes scores for all the low-level skills. In its setting where a fixed set of low-level skills are provided, can we use the LLM to output open-vocabulary tokens and match the skill names using text embedding's similarity, to decrease the token cost of SayCan?\n\nLIMITATIONS:\nThe authors discussed the limitations in the paper. Some of these are addressed and others are left to future work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Grounded Decoding, a methodology to ground the high level semantic plans output by an LLM into a realizable action sequence. Specifically, the authors construct a model which decodes a sequence that is likely to be realizable given the current state of the robot and environment, but also likely according to the language model. Experimental results in both simulation and the real world demonstrate the utility of the proposed method over baselines from prior work.\n\nSTRENGTHS:\n- Well-written. The paper is well-written, and the figures aid in the understanding of the methodology.\n- Strong experimental results. Show the utility of using both the ungrounded LLM and their Grounded Decoding relative to prior work on a range of tasks both in simulation and the real world. Outperform SayCan in terms of efficiency by a non-trivial margin. The inclusion of the qualitative results is also useful.\n- Ablations. The experiments in simulation include ablations which isolate the contributions of the proposed grounded decoding module. Furthermore, the authors include ablations demonstrating the impact of the choice between greedy search and beam search, which is insightful.\n- Inclusion of failure mode analysis. The analysis on the failure modes of the system is insightful.\n\nWEAKNESSES:\n- Limited experimental evaluation. The authors include results on just two environments in simulation and one in the real world. Showing results on a larger number of environments would make the claims more convincing.\n- Lack of comparisons between grounding functions. Section 3.4 proposes three different techniques for obtaining grounded models that can be leveraged in GD’s formulation. However, the three techniques do not seem to be directly compared in the experiments. An experiment of this form may provide insight about which formulation works best for the proposed algorithm. Furthermore, how do the different grounding functions (AF, S, P) impact performance in the experiments?\n- Ground truth map for navigation. The navigation module used in the mobile manipulation in a physical kitchen experiment is scripted and uses the ground truth map of the environment for navigation. This is a major assumption and limits deployability to only seen environments.\n- Prompt engineering. It is claimed that a certain amount of prompt engineering was done to steer the LLMs. This is an undesirable property of the proposed system. How sensitive is the system to such prompt engineering? Was the same effort also applied in prompt engineering for the baseline methods?\n\nQUESTIONS:\n- In Figure 3 (leftmost), why is the grounded model's score for _orange higher than that of _red, when the red block is visible but there is no orange block present? Similarly, why is the language model's score for _yellow so high in the rightmost plot?\n\nLIMITATIONS:\nThe authors include a detailed section on the limitations of the work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Grounded Decoding (GD), a method that filters the decoded task descriptions by the probability of a grounded model, LLM-based task planning. This paper shows the framework can integrate different types of grounded models. Experiments across physical and simulated environments show that this decoding method outperforms prior models in instruction following tasks.\n\nSTRENGTHS:\n- The introduction of grounding functions covers a family of methods that evaluates the likelihood of the generated text plans conditioned on the current states.\n- This paper presents the method clearly and it is easy to follow.\n- The inclusion of grounding in the decoding process reduces the planning failure.\n\n-----\n\nPost-rebuttal update\n\nThe authors' responses answered my questions. I agree that GD is a more flexible framework and can be more easily extended to include various grounding functions as opposed to SayCan. I would encourage the authors to include the rebuttal responses to the paper to improve clarity.\n\nWEAKNESSES:\n- While it is possible to include safety and preference in the grounding function, the current form to include safety and preference is quite preliminary. It only considers the presence of objects but usually, these should be represented as statements.\n- It is a good modification of SayCan to constrain the grounding in the brackets. It is unclear how reliable the ungrounded decoding and activate or revert grounding at the right time and how this modification fits the formulation of GD (Eq. 7).\n- The grounding function can be used as the affordance function in SayCan. If we consider the grounding function this way, the main advantage of GD is its efficiency not the introduction of the grounding function.\n\nQUESTIONS:\n- The results show the overall success rate. How does it perform given the safety and preference grounding?\n- There is a huge gap between planning and execution in the real robot experiment. What causes this big gap?\n- In Fig 5, why doesn’t include SayCan’s failure breakdown?\n- Fig 6 is quite confusing, the description says each dot is an instruction and they are colored by the affordance value computed against four different scenes. How are those scenes selected and why some instructions are infeasible if they are meaningful instructions for that domain?\n\nLIMITATIONS:\nThe appear discussed the limitation of the proposed method well. The joint decoding may be limited as the grounding models are limited and most of them are object-centric grounding.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}