SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Wenlong Huang $^{1*}$ , Fei Xia $^{2}$ , Dhruv Shah $^{3}$ , Danny Driess $^{2}$ , Andy Zeng $^{2}$ , Yao Lu $^{2}$ , Pete Florence $^{2}$ , Igor Mordatch $^{2}$ , Sergey Levine $^{2,3}$ , Karol Hausman $^{2}$ , Brian Ichter $^{2}$ $^{1}$ Stanford University,  $^{2}$ Google Deepmind,  $^{3}$ UC Berkeley grounded-decoding.github.io
Recent progress in large language models (LLMs) has demonstrated the ability to learn and leverage Internet-scale knowledge through pre-training with autoregressive models. Unfortunately, applying such models to settings with embodied agents, such as robots, is challenging due to their lack of experience with the physical world, inability to parse non-language observations, and ignorance of rewards or safety constraints that robots may require. On the other hand, language-conditioned robotic policies that learn from interaction data can provide the necessary grounding that allows the agent to be correctly situated in the real world, but such policies are limited by the lack of high-level semantic understanding due to the limited breadth of the interaction data available for training them. Thus, if we want to make use of the semantic knowledge in a language model while still situating it in an embodied setting, we must construct an action sequence that is both likely according to the language model and also realizable according to grounded models of the environment. We frame this as a problem similar to probabilistic filtering: decode a sequence that both has high probability under the language model and high probability under a set of grounded model objectives. We demonstrate how such grounded models can be obtained across three simulation and real-world domains, and that the proposed decoding strategy is able to solve complex, long-horizon embodiment tasks in a robotic setting by leveraging the knowledge of both models.
Recent works have demonstrated robots that are increasingly proficient at understanding and acting upon natural language, whether through planning or conditioned policies. Complementing such progress, the field of natural language processing has recently seen large language models (LLMs) become ubiquitously used as pre-trained or few-shot prompted models, due to their impressive few-shot performance and vast knowledge-base. These LLMs have efficiently learned from web-scale data through autoregressively modeling the probability distribution over text tokens and thus generate text. However, the nature of this process is such that applying such models to embodied settings remains a challenge. They have not interacted with their environment, lack observability of non-language observation modalities (e.g., images), and may not know what is safe or possible for a particular embodiment.
Determining how to execute long-horizon behaviors based on high-level verbal commands is one particular area of robotics where the rich semantic knowledge in large language models can be especially useful. This problem combines elements of semantic reasoning and planning: the robot
*Work done as an intern at Google.
Figure 1: Grounded Decoding solves robotic tasks by taking an instruction as input and selecting tokens that have high probability under a Large Language Model (LLM) and a set of Grounded Models (GM). Thus, it leverages the open-vocabulary and semantic knowledge of LLMs while being grounded in the environment and in the robot's capabilities. Furthermore, the whole process does not require expensive fine-tuning of the LLM.
must understand the instruction, determine the steps needed to fulfill it, and also determine how to sequence those steps appropriately given its capabilities and the current state of the environment. However, this is not a problem that can be solved purely with semantics, as it requires sufficient grounding to understand how the task should be performed in context – for example, in the example in Figure 1, the language model alone has no way of knowing which block to pick up because this requires knowledge of which blocks are present, and also what manipulations the robot is capable of performing on them. Thus, although a language model can assign probabilities for how likely various steps are to correspond to the desired task semantically, the constraints of the planning problem must also enter into the process. These constraints could themselves be represented as probabilities that mirror the token probabilities generated by a language model, reflecting their applicability to the current environment rather than their semantic likelihood. We can frame this as a problem similar to probabilistic filtering: decode a sequence (i.e., a task description) that both has a high probability under the language model and a high probability under a grounded model that predicts how applicable this sequence is to the current scene.
Herein, we present Grounded Decoding (GD), a scalable, general approach to planning with LLMs embodied domains. Grounded Decoding jointly decodes the token probability of an LLM and token probabilities from token-conditioned, robotic functions, such as affordance functions capturing the abilities of a robot given its embodiment, safety functions, or more. By guiding the LLM directly at its output, Grounded Decoding enables a general and flexible family of planning algorithms that combines LLM's strength of long-horizon and semantic reasoning and grounded models' strength of local and embodiment grounding.
Our contributions are as followed: 1) we present a robot-centric formulation for decoding language models to perform long-horizon robotic tasks with token-conditioned grounded models, 2) we demonstrate techniques for learning such grounded models, serving different purposes such as affordances and safety requirements, and 3) we show empirical evidence, across three simulation and real-world domains, that the proposed method performs strongly on a wide range of tasks while also significantly outperforming prior methods in efficiency.
Guided Decoding for Language Models. Decoding strategies for large language models is an active area of research within natural language processing [77, 87, 25, 85, 38]. A number of recent works have focused on developing decoding heuristics for natural text generation [49, 35, 48, 18, 25, 6, 36, 63]. Another line of works use external classifiers for maximizing certain language-space utilities when decoding language models [71, 92, 26, 39, 37, 21, 38, 4, 12, 23]. Most closely related to our work are classifier-guided decoding methods developed for offline domains, such as image captioning [78, 74] and task-oriented dialog [72, 83]. However, extensions to embodied domains, which we investigate exclusively in this work, remain non-trivial because grounding in embodied domains is bounded by the abilities of the agent and by environment state transition as the agent actively interacts with the environment.
Figure 2: Overview of Grounded Decoding. Given a free-form language instruction, a language model and grounded models jointly decide the next candidate token to be decoded by combining their respective likelihood. Language model proposes likely tokens that produce goal-directed and coherent long-horizon behaviors, while grounded models connect them to the physical scene, through a flexible composition of multiple objective functions from multiple modalities, such as affordance, preferences, and safety.
Embodied and Multimodal Language Models. Training language models to understand embodiment is an active area of research. Training multimodal models can enable some degree of embodied reasoning, such as understanding images and videos [9, 40, 76, 3]. Directly finetuning language models to output actions has also been investigated [75, 55, 66]. Lastly, training downstream models on language model embeddings shows promise [46, 52, 24, 94, 60, 41]. In this work, we investigate leveraging large frozen language models for embodied applications [29, 2, 96, 8, 64, 30, 42, 70, 47, 27, 58, 44, 43, 14, 16, 82, 93, 89, 45, 56], with grounded models to provide domain-specific grounding during decoding process.
Comparison to SayCan. The most closely related work to our work is SayCan [2]. SayCan uses a large language model and a value function to select robotic skills among a constrained set of primitives. This constrained set of primitives enables SayCan to use the so-called "scoring-mode" of the LLM to get the probability of a skill being useful to a high-level instruction. This requirement to consider only a fixed and enumerated set of primitives limits the applicability of SayCan in scenarios with many possible skills, such as open vocabulary or combinatorial tasks. Grounded Decoding on the other hand jointly decodes the LLM and the grounded model at the token level, allowing for expressive decoding with an open vocabulary. Furthermore, SayCan considers only grounding functions derived from RL-trained value functions for affordance grounding functions, while Grounded Decoding explores many types of grounding functions to propose a broad family of algorithms.
Task and Motion Planning. Task and motion planning [33] seeks to solve high-level instructions via sequencing tasks in dynamically feasible manner. Research within this area generally focuses on symbolic planning [19] or optimization-based [79] approaches. Machine learning has increasingly been used to accelerate planning and enable new domains [91, 61, 69, 20, 17, 28, 90, 31, 1, 65, 51, 86, 88, 15]. However, planning constraints are often explicitly specified for TAMP methods. In contrast, we specify constraints as (learned) probabilities, which are baked into the decoding process and provided by domain-specific grounded models.
3 Grounded Decoding
3.1 LLMs and Grounding Models
Large Language Models. LLMs are trained to predict the probability  $p(W)$  of a text sequence  $W$ , represented as a sequence of tokens  $W = w_{1:N} = (w_1, \dots, w_N)$ . The tokens are elements of a fixed vocabulary  $\mathcal{W}$ . Typical neural architectures factorize the joint probability into  $p(W) = \prod_{n=1}^{N} p_{\mathrm{LLM}}(w_n | w_{1:n-1})$ , where  $p_{\mathrm{LLM}}$  is predicted by a transformer network [81]. Given  $p_{\mathrm{LLM}}$ , generating a text consisting of  $N$ -many tokens, the so-called decoding process, can be seen as the optimization problem  $\arg \max_{w_{1:N} \in \mathcal{W}} \prod_{n=1}^{N} p_{\mathrm{LLM}}(w_n | w_{1:n-1})$ , which in practice is solved, e.g., using greedy search, beam search, or sampling strategies. To further ensure the LLM is solving a desired task, one typically starts with a given text, the so-called prefix or prompt, that describes the task, and then the LLM completes this task in its decoding process.
Grounding Functions. We use the concept of grounding functions,  $p_{\mathrm{G}}(w_{1:n}|s)$ , which seek to model a probability of tokens  $w_{1:n}$  given (potentially non-textual) state  $s \in S$ . This state is intended to capture the embodiment of the robot and the environment, which may be an image, proprioception of the robot, or the environment state. Thus the grounding function models probabilities relevant to the robot embodiment and environment, such as whether the tokens are possible for the robot to execute given the state (affordances), or other values like safety, cost, or user preferences.
3.2 Problem formulation.
Given an instruction in language  $\ell$ , we look at the problem of using an LLM to decode a language plan  $w_{1:N}$ , which is typically done by finding the most likely tokens under the probability distribution predicted by the LLM,  $p_{\mathrm{LLM}}(w_{1:N}|\ell)$ , with  $\ell$  being the prefix. However, based on the instruction  $\ell$  as the prefix alone, the LLM can easily generate text that is not grounded in the physical state of the environment, rendering such plans useless in the real world. In order to ground the language model in an actual physical embodiment, we propose Grounded Decoding (GD): The main idea of GD is to guide the generation of token sequences with grounding function(s) that are conditioned on the embodiment of the system.
Formally, let  $s \in S$  denote a representation of the state of the world. Then, GD attempts to generate text that is consistent with both the instruction  $\ell$  and the physical state  $s$ :
$$
w _ {1: N} ^ {*} = \arg \max  _ {w _ {1: N}, w _ {n} \in \mathcal {W}} p _ {\mathrm {G D}} \left(w _ {1: N} \mid s, \ell\right) \tag {1}
$$
To leverage the Internet-scale knowledge of LLMs, we factorize  $p_{\mathrm{GD}}(w_{1:N}|s,\ell)$  as follows:
$$
\begin{array}{l} p _ {\mathrm {G D}} \left(w _ {1: N} \mid s, \ell\right) = \frac {p (s , \ell \mid w _ {1 : N}) p \left(w _ {1 : N}\right)}{p (s , \ell)} (2) \\ = \frac {p (s \mid w _ {1 : N}) p (\ell \mid w _ {1 : N}) p (w _ {1 : N})}{p (s , \ell)} (3) \\ = \frac {p \left(w _ {1 : N} \mid \ell\right) p (\ell) p \left(w _ {1 : N} \mid s\right) p (s) p \left(w _ {1 : N}\right)}{p (s , \ell) p \left(w _ {1 : N}\right) p \left(w _ {1 : N}\right)} (4) \\ \propto \frac {p \left(w _ {1 : N} \mid \ell\right)}{p \left(w _ {1 : N}\right)} p \left(w _ {1: N} \mid s\right) (5) \\ \propto p \left(w _ {1: N} | \ell\right) p \left(w _ {1: N} \mid s\right). (6) \\ \end{array}
$$
To decode autoregressively with the formulation, we factorize above into token decoding:
$$
p _ {\mathrm {G D}} \left(w _ {1: N} \mid s, \ell\right) \propto \prod_ {n = 1} ^ {N} p _ {\mathrm {L L M}} \left(w _ {n} \mid w _ {1: n - 1}, \ell\right) p _ {\mathrm {G}} \left(w _ {1: n} \mid s\right). \tag {7}
$$
The first term,  $p_{\mathrm{LLM}}(w_n|w_{1:n-1}, \ell)$ , can be modeled as the probability of the LLM predicting the token for the given instruction  $\ell$  appended previously decoded tokens  $w_{1:n-1}$  without the state  $s$  as input. The second term,  $p_{\mathrm{G}}(w_{1:n}|s)$ , is the grounding function that is only conditioned on the state  $s$  and judges whether the generated text  $w_{1:n}$  is consistent with the physical state. The core idea behind this factorization is that LLMs exhibit long-term planning capabilities, while the grounding function guides the planning of the LLM to be possible in the concrete embodied physical world without needing to be informed or capable of reasoning over the long-horizon instruction.
3.3 Grounded Decoding
This work investigates grounded decoding exclusively in the context of task planning for embodied agents. Figure 2 visualizes a single step of the simplest greedy search form of GD, and accompanying pseudo-code can be found in Algorithm 1. Given a high-level language instruction and history of executed actions, GD proceeds through a process similar to probabilistic filtering by selecting tokens iteratively that have high probability under the language model and the grounded model. After
2We make three assumptions for this derivation: 1)  $s$  and  $\ell$  are marginally independent (Line 3), 2)  $s$  and  $\ell$  are conditionally independent given  $w_{1:N}$  (Line 5), and 3)  $p(w_{1:N})$  is uniform over responses (Line 6).
each token is selected, it is appended to the prefix. The process continues until a token in the terminal set  $\mathcal{W}_{\mathrm{term}}$  is selected, which could be a period sign “.” indicating the end of a single-step skill (e.g., pick-and-place). Then the command  $w_{1:i}$  is sent to a language-conditioned policy  $\pi(a|s, w_{1:i})$  that executes the action  $a$  conditioned on the environment state  $s$ . Crucially, this grounding function must accept partial commands to enable grounding during decoding. $^3$  Additionally, we note that GD, in its essence, provides a grounded scoring function; thus, it can be easily extended to any search methods such as beam search, top-k sampling, etc.
Algorithm 1 Grounded Decoding (GD) w/ Greedy Search
1: Given: state  $s$ , instruction  $\ell$ , terminal set  $\mathcal{W}_{\mathrm{term}}$   
2: Initialize:  $w = \{\}$ ,  $n = 0$   
3: while  $w_{n} \notin \mathcal{W}_{\mathrm{term}}$  do  
4:  $n = n + 1$   
5:  $w_{n} = \arg \max_{w_{n} \in \mathcal{W}} p_{\mathrm{LLM}}(w_{n}|w_{1:n - 1}, \ell) p_{\mathrm{G}}(w_{1:n}|s)$   
6: end while  
7: Return:  $w$
3.4 Techniques for Obtaining Grounded Models
Unlike language tasks, where a single model is capable of performing general semantic reasoning, a singular grounded model remains an open problem. Indeed, each domain may impose varied environmental and embodiment constraints. Despite these challenges, we present several techniques for obtaining grounded models that can be leveraged in GD's formulation, and validate them in three domains in Section 4.
Token-Conditioned Value Functions. Assuming a robot acts with action  $a$  according to policy  $\pi(a|s, w_{1:n})$ , that aims to maximize certain a utility and that the utility captures a task objective, a natural choice that can provide "grounding score" is the action-value function  $Q(s, a|w_{1:n})$  as it necessarily captures the embodiment of the robot. Additional objectives, such as task constraints, can also be encapsulated in  $Q(s, a|w_{1:n})$  to ensure grounding. Note that unlike the formulation proposed in [2],  $w_{1:n}$  cannot be restricted to a fixed repertoire of token sequences. In practice, to obtain a  $Q(s, a|w_{1:n})$  that satisfies the requirements, one can train multi-task language-conditioned agents, either through reinforcement learning (Section 4.2) or supervised learning (Section 4.1).
Multimodal Foundation Models. A general choice to ground LLMs is through using multimodal foundation models, such as CLIP [57] or open-vocabulary object detectors [22, 34, 50]. Although these models can connect language to other grounded modalities (e.g., vision), they often lack the capability for complex or long-horizon reasoning, and they do not consider embodiment constraints. As a result, to leverage them in the decoding process, they need to constrained to where they are the most applicable rather than always decoding jointly. To this end, we use a prompt-based technique that allows LLMs to choose when to jointly decode (Section 4.3), which we find to be effective in most cases. $^{4}$
Rule-based Methods. Another source of grounding may come from features  $x = \phi(w_{1:n})$  designed with expert domain knowledge, which can then be used to map  $w_{1:n}$  to a "grounding score" using parametric or non-parametric functions  $f(x)$ . Such techniques may be most applicable when interpretability and enforcing hard constraints are required, such as safety-critical settings, or when data are scarce, such as cases involving preferences of individual users (as shown in Section 4.1).
3.5 Comparisons to Prompt-based Methods
One alternative approach for grounding is including scene information as part of the prompt (e.g., object detection results [96]), which complements the grounding method proposed in this work.
<sup>3</sup>As an example, an affordance ground function for a skill "pick up object", should emit a high probability for "pick" and "pick up" if any object is able to be picked and collapse to only feasible objects only once the object token is decoded.
4Emerging multimodal language models [16, 53] provide strong baselines, but they similarly cannot serve general-purpose grounding functions because they are not conditioned on embodiment, except for cases where embodiment data from each individual domain can be used to finetune the LLM [16].
Figure 3: Example rollouts and likelihood of representative tokens under Grounded Decoding objective in three distinct domains: simulated tabletop rearrangement (top), Minigrid 2D Maze (middle), and real-world kitchen mobile manipulation (bottom). Each domain uses different prompts, grounded models, and low-level primitives. The GD formulation is shared across the domains, decoding a pre-trained language model with respect to domain-specific grounded models to decompose a open-ended instruction into actionable steps.
However, we note that prompting is often insufficient for grounding, as information about the scene and about the capabilities of the robot may not always be succinctly described in the prompt. Such examples include 1) in a block stacking task, a block that has been stacked on cannot be picked, 2) in a navigation task, to open a door, one must have a key and that door must be reachable, and 3) in a mobile manipulation domain, an object may be visible but is out of reach of the manipulator. Therefore, Grounded Decoding is a more general and flexible grounding method that injects continuous probabilities during decoding, which may even come from grounding functions from other modalities (e.g., vision).
4.1 Long-Horizon Tabletop Manipulation
Herein we experiment with a simulated tabletop manipulation environment based on RAVENS [95]. We create a custom set of 20 tasks, with 10 seen tasks and 10 unseen tasks. Seen tasks are used for training (for supervised baseline) or for few-shot prompting. They are grouped by following categories. Detailed breakdown can be found in Appendix A.2.
i. Letters: Rearranging alphabetical letters ("sort the letters in alphabetical order").
ii. Blocks & Bowls: Rearranging or combining blocks and bowls ("put blocks in matching bowls").
iii. Box Packing: Sorting food items and utensils into boxes in accordance with safety constraints and user preferences ("Can you pack the picnic box for me?").
Given only high-level language instructions and top-down visual observation of the environment, Grounded Decoding decodes a sequence of text tokens representing the step command to be executed. Note that because GD generated grounded free-form actions, it does not require each step to strictly map to a repertoire of skill as in [29, 2]. After a complete command is generated, it is executed via a pre-trained multi-task language-conditioned CLIPort [67]. An example rollout is shown in Figure 4. To demonstrate the techniques proposed in Section 3.4 to obtain grounding functions, we study the composition of following grounding functions (overall grounding score is calculated as  $p_{\mathrm{G}} = \prod_{i=1}^{n} p_{i}$ ) depending on the task categories. Refer to the Appendix A.2 for details.
CLIPort	+LLM	+GD (Ours)
Short	Long	Ungrounded	Greedy	Beam
Seen Tasks
Letters	7%	40%	20%	43%	57%
Blocks & Bowls	2%	62%	35%	60%	77%
Box Packing*	15%	28%	11%	79%	78%
Unseen Tasks
Letters	6%	10%	19%	37%	41%
Blocks & Bowls	6%	10%	28%	44%	50%
Table 1: Tabletop domain success rates.
PPO	+Skills HRL	+LLM Ungrounded	+GD (Ours)
Greedy	Beam
Easy	28%	68%	96%	100%	100%
Medium	13%	48%	87%	93%	97%
Hard	6%	31%	54%	78%	88%
Table 2: 2D Maze success rates.
Affordance Grounding Function (AF). As the primitive policy CLIPort [67] already acts as an action-value function over the pixel space, we directly leverage its predictions for affordance grounding. In particular, given scene image  $s$  and partially-decoded instruction  $w_{1:n}$ , CLIPort predicts unnormalized logits over the pixel space  $\mathbf{u}_{\mathrm{pick}}$ ,  $\mathbf{u}_{\mathrm{place}} \in \mathbb{R}^{480 \times 640}$ , respectively for the pick location and the place location. Therefore, for any given  $s$  and  $w_{1:n}$ , we can calculate the affordance as  $p_{\mathrm{AF}}(w_{1:n}|s) = \max_{(x,y) \in 480 \times 640} (\mathbf{u}_{\mathrm{pick}}(x,y) + \mathbf{u}_{\mathrm{place}}(x,y))$ .
Safety Grounding Function (S). To strictly enforce hard constraints such as safety requirements, we adopt the rule-based method proposed in Section 3.4. In particular, the features  $x$  are indicator functions denoting whether knives and red boxes which we use as hazardous objects are involved in an action, i.e.,  $x = \mathbb{I}[\text{"red"}$  or "knife" in  $w_{1:n}$ ]. We then use constant mappings to convert the features  $p_{\mathrm{S}}(w_{1:n}|s) = \frac{\epsilon}{Z}x + \frac{1}{Z}(1 - x)$  to scores of 0 or 1, where  $Z$  is the normalizing term and  $\epsilon \approx 0$  is a small value for ensuring the joint probability does not collapse to 0.