SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Preference Grounding Function (P). We similarly use rule-based methods for preference grounding, as data of individual users may be scarce to learn a separate model. In particular, we choose two random objects  $(o_1,o_2)$  as the preferred objects, i.e.,  $x = \mathbb{I}[o_1$  or  $o_2$  in  $w_{1:n}]$ . Note that unlike safety functions, preferences often come in the form of "soft requirement". Therefore, the preference grounding function is implemented as  $p_{\mathrm{P}}(w_{1:n}|s) = \frac{\alpha}{Z} x + \frac{\beta}{Z} (1 - x)$ , where we choose  $\alpha = 0.5$  and  $\beta = 0.1$  for our experiments.
Baselines. We study two variants of GD using beam search and greedy search. We also compare to "No Grounding" baseline that decodes only according to language model likelihood. Furthermore, we compare to solitary method CLIPort [67] that directly take in the high-level language instructions without a planner. We consider two variants of CLIPort: 1) "Short" that is trained with only single-step pick-and-place commands, and 2) "Long" that is trained on high-level instructions from the 10 training tasks. For more details, please refer to Appendix A.2.
Analysis. Results grouped by each task category are shown in Table 1<sup>5</sup>. Please refer to the Appendix for detailed breakdown. Each method is evaluated on 20 episodes for each task within each task category. Supervised methods, such as CLIPort, are found to perform poorly on unseen tasks. Methods that leverage language model planner show better generalization to unseen tasks but fall short due to lack of grounding. Grounded Decoding achieves the best results by enabling the LLM to plan actions using grounded information and is further improved with beam search.
4.2 2D Maze
We further evaluate the long-horizon reasoning of Grounded Decoding for 2D maze-solving on Minigrid [10]. The agent receives a top-down view of the environment along with a natural language instruction. More details can be found in Appendix A.3. The tasks are grouped in three categories:
i. Easy: Simple tasks where the horizon is short (10-30 steps) and fully described by the textual instruction, e.g. OpenDoors and PutNext.
ii. Medium: Short and long-horizon tasks (up to 80 steps) with step-by-step textual instructions, e.g. LockedRoom.
iii. Hard: Complex, long-horizon instructions (over 100 steps) with ambiguous instructions that necessitate multi-step reasoning and efficient exploration, e.g. MultiRoom and BlockedUnlock.
5Box Packing tasks are seen during training, but safety and preference requirements are only enforced during evaluation.
Figure 4: Greedy decoding rollout with GD, where key decoded tokens are shown (yellow, purple, red, yellow). Combined scores are normalized to the maximum for visual clarity; others are normalized to their sum.
Affordance Grounding Function. Following the recipe from Section 3.3, we train token-conditioned affordance function to be used in GD. The difference is that the grounding function here is the value function from the goal-conditioned policy that is trained with PPO [62] instead of from demonstrations as in CLIPort [67]. The policy performs short-horizon skills such as "Go to red key" or "Open the door" and are conditioned on CLIP embeddings of the skill and an image of the scene. Accordingly, the goal-conditioned value function evaluates the feasibility given the current observation and the (partially) decoded skill.
Baselines. We compare the two variants of GD - with greedy and beam search - with 1) a solitary PPO policy [62], 2) a hierarchical RL algorithm which plans over the low-level skills, and 3) a hierarchical method that uses an ungrounded language model for planning [29].
Analysis. Table 2 reports the success rate, averaged across 100 episodes of randomly initialized environments. The "flat" RL agent performs poorly in all but the simplest environments, owing to difficulties with understanding the high-level instruction and reasoning over long horizons (often over 100 steps). Planning over low-level skills using hierarchical RL [5] improves this performance, since the high-level decision-making problem is greatly simplified. However, the high-level RL agent still needs to reason over low-level (textual) skills by understanding their underlying capabilities and stitching them together. Using the planning capabilities of large language models to reason over textual skills significantly boosts this performance [29], since the language model can inherit the strong reasoning capabilities from its training data. This tends to be insufficient in challenging environments, however, since the number of potentially viable skills may be very large and the LLM has no information about the robot's observations. GD can leverage the learned affordance function (in this case, the goal-conditioned value function) to inform the language model's plans, enabling successful long-horizon reasoning. We further find that beam search improves performance modestly, particularly in long-horizon tasks.
4.3 Mobile Manipulation in a Physical Kitchen
Our last environment is a kitchen robot in the real world, and we follow the same implementations of the mobile manipulation platform and skills in SayCan [2]. We perform instruction following tasks, as in [2]. An example task is "Bring an apple", for which the robot needs to plan and execute a sequence of "1. Find an apple, 2. Pick up the apple, 3. Bring it to you. 4. Put it down, 5. Done". We split the tasks into two categories. Unambiguous means the instruction explicitly contains the object of interest, and Ambiguous means the instruction does not contain the object name. For example, when human asks "bring me the fruit", the robot needs to first determine available fruits. We assume all necessary objects are in the field of view. More details can be found in Appendix A.4.
Grounded Decoding with Chain-of-thought. We demonstrate using multimodal foundation models for Grounded Decoding, as proposed in Section 3.4. In particular, we use open-vocabulary object detector owl-vit [50]. Note that because these off-the-shelf models are not trained on robot domain data, we find that it works best by constraining their influence on decoding. We achieve this by making a slight modification to the SayCan algorithm [2]: before generating action plans, we prompt the LLM to generate visually-grounded chain-of-thought [84] by giving LLM the option of when to enable grounded decoding and disable grounded decoding, as visualized in Fig. 5. Specifically,
Figure 5: Example prompt and rollout in real-world kitchen environment.
Tasks	GD	SayCan
Planning Execution	Planning Execution
Unambiguous	85%	57%	85%	57%
Ambiguous	58%	44%	33%	25%
Table 3: Success rates in kitchen environment.
GD (Greedy)	GD (Beam)	SayCan
Success Rate	50%	60%	64%
Token Count	1x	4.3x	113.7x
Table 4: By avoiding full enumeration of skills, GD is able to be efficient than SayCan while staying performant.
LLMs can be prompted to generate a left bracket to start decoding jointly with grounded models and a right bracket to revert to ungrounded decoding. After chain-of-thought, we use SayCan scoring mode for decoding the action plans.
Analysis. Table 3 shows that GD recovers similar performance on Unambiguous tasks, and gain  $25\%$  in planning performance on Ambiguous tasks. This shows that GD with multimodal foundation models can effectively use visually-grounded chain-of-thought to disambiguate abstract tasks.
5.1 Comparison to SayCan
In this section, we directly compare GD to SayCan [2], which is related to our method in that both combine language model knowledge and grounded model knowledge (discussed in more detail in Section 2). However, SayCan uses the language model to score all pre-specified options, rendering it inefficient at dealing with large or combinatorial action spaces. In contrast, GD computation considers all possible language token in the autoregressive decoding process, which is independent of the size of the action space. Results shown in Table 4 demonstrate that GD is two orders of magnitude more efficient on our tasks, with comparable performance. Furthermore, by decoding at the most basic functioning unit of language, GD's formulation allows open-vocabulary grounding beyond just affordances, e.g. safety, preferences, and multimodal embeddings such as CLIP.
5.2 Breakdown of Failure Reasons
Because all hierarchical approaches share an imperfect low-level policy for step execution, the results reported in Table 1 are compounded with both planning failures and low-level policy failure. In Figure 6, we provide failure breakdown analysis for Grounded Decoding and associated baselines. Note that the CLIPort baselines are solitary methods that do not use a planner, so the failures are solely composed of policy failures. As shown in Figure 6, while all planning-based methods use the same underlying low-level policy, Grounded Decoding significantly reduces planning failure by being able to incorporate grounded scene information into the decoding process. Moreover, we observe that despite the shared affordance function across beam and greedy search, the beam search variant performs stronger by being aware of the full-length single-step instructions during decoding.
5.3 Grounded Action Manifold
A central goal of this work is to investigate the integration of grounded information into language model decoding to output instructions actionable by a policy. To investigate this, we use a t-SNE [80] plot to illustrate the extent to which grounded models help narrow down the search space for language models. Specifically, we first enumerate all meaningful instructions in the tabletop domain, such as "pick x and place it on y," which are represented as dots in the figure. We then compute the affordance values with respect to four different scenes, where each color represents one scene. Finally, we group the dots using t-SNE and BERT embeddings [13]. Figure 7 shows that grounded
Figure 6: Failure breakdown on tabletop domain. GD achieves lowest planning failure among planning-based methods, among which beam search variant performs the best.
Figure 7: Visualization of actions colored by affordance values in different scenes. Every dot represents a possible action in the tabletop domain, where the majority of the actions are infeasible. We show how grounded models can identify the feasible actions for specific scenes. Notably, these actions are not always clustered in language space, requiring the grounding function to determine what action to perform.
models can effectively identify achievable skills to produce an actionable manifold within the language space and that this grounding is required, as language alone does not perfectly group actionable skills. It is worth noting that while we provide manual enumeration of all possible skills for practical analysis, the full language space is much larger. This highlights the even more pronounced narrowing of the search in the language space.
6 Conclusions, Limitations, & Future Works
We presented Grounded Decoding (GD), an approach for leveraging the knowledge and capabilities of large language models in embodied settings through grounding functions, which model the probabilities of tokens given an embodiment. GD resembles probabilistic filtering, by decoding tokens that have high probabilities under the language model and under grounded model(s). By guiding the LLM's decoding directly at its output, GD is a general, flexible, and expressive approach to embodied tasks. This is demonstrated on three embodied domains, showing GD is capable of solving complex, long-horizon tasks.
Though quite general and flexible, GD has a few limitations. First, although we present several techniques for obtaining grounding functions in different domains, it remains a question whether a capable and general grounding function can be obtained. We hope that recent progress in large-scale robotics models (e.g. [7] and [59]) can remove this bottleneck, and note that the flexibility of GD allows such progress to be straightforwardly leveraged. Second, prompt engineering is often required to steer LLMs to the desired action space (e.g., likely action verbs, likely present objects). Finally, while not requiring additional training, the joint decoding may be limiting compared to a single model capable of both grounding and language reasoning [3, 9, 16].
This work presented a family of algorithms for grounding LLMs in embodiment, for which there are many avenues for future work. The flexibility of the approach enables many other grounding functions and ways to integrate grounding. Furthermore, the development and integration of a foundation model for grounding would improve performance significantly. Finally, though GD's probabilistic filtering-based approach is quite general, fusing grounding information to the language model after each token decoding may be limiting and future works can investigate how such grounding can be elegantly integrated during decoding.
The authors would like to acknowledge Pierre Sermanet, Carolina Parada, Jie Tan, Yevgen Chebotar, Vincent Vanhoucke, and Dorsa Sadigh for their feedback and contributions. This work is supported in part by OpenAI academic access program, granted to Wenlong Huang.
[87] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016.
[88] Fei Xia, Chengshu Li, Roberto Martin-Martin, Or Litany, Alexander Toshev, and Silvio Savarese. Relmogen: Integrating motion generation in reinforcement learning for mobile manipulation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), 2021.
[89] Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, and Harold Soh. Translating natural language to planning goals with large-language models. arXiv preprint arXiv:2302.05128, 2023.
[90] Danfei Xu, Ajay Mandlekar, Roberto Martin-Martin, Yuke Zhu, Silvio Savarese, and Li Fei-Fei. Deep affordance foresight: Planning through what can be done in the future. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 6206–6213. IEEE, 2021.
[91] Danfei Xu, Suraj Nair, Yuke Zhu, Julian Gao, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Neural task programming: Learning to generalize across hierarchical tasks. In 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018.
[92] Kevin Yang and Dan Klein. Fudge: Controlled text generation with future discriminators. arXiv preprint arXiv:2104.05218, 2021.
[93] Haoqi Yuan, Chi Zhang, Hongcheng Wang, Feiyang Xie, Penglin Cai, Hao Dong, and Zongqing Lu. Plan4mc: Skill reinforcement learning and planning for open-world northwest tasks. arXiv preprint arXiv:2303.16563, 2023.
[94] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. Advances in Neural Information Processing Systems, 2021.
[95] Andy Zeng, Pete Florence, Jonathan Thompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, and Johnny Lee. Transporter networks: Rearranging the visual world for robotic manipulation. Conference on Robot Learning (CoRL), 2020.
[96] Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598, 2022.
A.1 Grounded Decoding Implementation Details
We study three different implementations of Grounded Decoding for each of the experimental domains. While each instantiation applied Grounded Decoding to long-horizon planning and behavior synthesis, different components including language models and grounded models are used in each domain, as seen in Table 5. Grounded models used in these domains include Affordance Functions (AF), Safety Functions (S), Preference Functions (P), and Open-Vocabulary Object Detectors (D).
Tabletop Rearrangement (Sim)	MiniGrid 2D Maze (Sim)	Kitchen Mobile Manipulation (Real)
LLM	InstructGPT [54]	InstructGPT	InstructGPT + PaLM[11]
Primitives	CLIPort [67]	PPO [62]	RT-1 [7]
Grounded Models	AF + S + P	AF	D
Table 5: Comparison between different versions of GD implemented in three different environments.
A.2 Implementation Details of Simulated Tabletop Rearrangement
There are a total of 20 tasks (templates of language instructions), listed in Table 6, grouped into three task category: Letters, Blocks&Bowls, and Box Packing. Three categories share a total of 57 objects. For Letters category, the goals are to rearrange the alphabetical letter objects such that they satisfy certain orders specified by the language instructions. At the beginning of each episode, task-relevant objects and a set of 1 to 3 randomly-sampled distractor objects (except for the Letters category) are initialized at random positions on the tabletop with fixed orientations. A minimum  $15\mathrm{cm}$  distance is enforced between any two objects to avoid collision and penetration at initialization. To allow for automatic evaluations, a binary reward function is defined for each task using ground-truth state of the objects. Furthermore, we implement scripted policies for each task to collect demonstrations for training the CLIPort baseline. For certain tasks, we also randomize the attributes mentioned in the given instructions, which can be found below:
i.  $\langle \mathrm{word}\rangle$  : hi, world, left, right, top, down, love, you
ii.  $\langle$  corner/side  $\rangle$ : left side, top left corner, top side, top right corner, bottom right corner, bottom side, bottom left corner
A.2.2 Low-level Primitives
We use CLIPort [67] as the low-level primitive that can be invoked by the LLM planner, as it shows promising results of generalization across free-form language instructions. Additionally, since the policy predicts per-pixel affordance, it can be repurposed to serve as grounded models for planning for long-horizon tasks, which we leverage in this work. The single primitive policy is trained on 50,000 pre-collected demonstrations, across 10 training tasks, where each demonstration contains 1) language instruction of the format "pick up [x] and place it on [y]", 2) top-down RGB-D observation of the current scene, 3) expert pick location expressed as pixel coordinates, and 4) expert place location expressed as pixel coordinates. The expert actions are obtained by accessing ground-truth object pose in the simulator. We further apply substring augmentation during training as we find it helps with performance on partial commands (see Section
A.2.3 Language Model
We use InstructGPT [54] (text-davinci-002), accessed through OpenAI API.
A.2.4 CLIPort Baseline
As CLIPort [67] already takes as input a natural language instruction and is capable of directly outputting robot actions, it bears the question whether we need a high-level planner for completing
long-horizon tasks. To this end, we additionally train two variants of multi-task CLIPort policy on 10 of the total 20 tasks as baselines (see Table 6 for the train/test split). One variant, which we referred as "CLIPort (Short)", is trained only on single-step pick-and-place instructions of the format "pick up [x] and place it on [y]" on the 10 training tasks. The decomposed pick-and-place instructions are obtained from scripted planners. At evaluation time, the policy is fed in only the high-level instructions without any planners. The other variant, which we referred as "CLIPort (Long)", is trained on the high-level instructions from the 10 training tasks (without decomposition from scripted planners). Similarly, at evaluation time, it is fed in only the high-level instructions and evaluated on both seen and unseen instructions. Both variants are trained on 50,000 demonstrations, similar to the Grounded Decoding primitive. The goal of these baselines is to evaluate whether solitary language-conditioned policies can perform well on long-horizon tasks and generalize to new task instructions. Note that the CLIPort baselines are different from the primitive used in Grounded Decoding, although they share the same architecture.
A.2.5 Full Experimental Results in Simulated Tabletop Domain
Below we show the full list of tasks and the full experimental results in the simulated tabletop domain. Each entry is the average success rate across 20 rollouts. The tasks with blue-colored background are seen tasks and the tasks with orange-colored background are the unseen tasks. Seen tasks may be used for training for supervised baselines (CLIPort) or may be used in the prompt for methods using language model planner. Note that for the "Box Packing" task category, although all tasks were seen in training or the prompts, we enforce additional safety and preference constraints for evaluation only at test time.
Tasks	pG	CLIPort	+LLM	+Grounded Decoding
Short	Long	Ungrounded	Greedy	Beam
Letters
Put the letters in alphabetical order from left to right	AF	5%	20%	10%	20%	40%
Spell as much of word as you can	AF	10%	60%	30%	60%	65%
Separate the vowels from the remaining letters to the bottom side	AF	5%	40%	20%	50%	65%
Put the letters in reverse alphabetical order from left to right	AF	15%	10%	15%	25%	25%
Correctly spell out a sport using the present letters	AF	10%	10%	5%	30%	30%
Sort the geometrically symmetrical letters to the bottom side	AF	5%	10%	15%	35%	50%
Separate the consonants from the remaining letters to the bottom side	AF	0%	0%	25%	25%	25%
Sort the letters less than "D" according to ASCII to the bottom side	AF	0%	20%	35%	70%	75%
Blocks & Bowls
Stack all the blocks	AF	5%	90%	30%	75%	90%
Put all the blocks on the corner/side	AF	0%	65%	50%	45%	70%
Put all the blocks in the bowls with matching colors	AF	0%	30%	25%	60%	70%
Put the blocks in the bowls with mismatched colors	AF	25%	30%	45%	30%	55%
Put all the blocks in different corners	AF	0%	5%	40%	50%	60%
Stack only the blocks of cool colors	AF	5%	5%	20%	70%	70%
Stack only the blocks of warm colors	AF	0%	10%	15%	45%	35%
Sort the primary color blocks to the left side	AF	0%	0%	20%	25%	30%
Box Packing*
Pack the objects into the brown box	AF + S	20%	40%	5%	100%	90%
Pack the objects into the boxes	AF + S	10%	20%	5%	75%	70%
I'd like some snacks on the right side	AF + P	15%	20%	15%	40%	55%
Pack me a picnic box	AF + S + P	15%	30%	20%	100%	95%
Table 6: Full Experimental Results in Simulated Tabletop Rearrangement Tasks. The tasks with blue-colored background are seen tasks and the tasks with orange-colored background are the unseen tasks. *Box Packing tasks are all seen during training, but safety and preference requirements are only enforced during evaluation.
A.3 Implementation Details of Minigrid 2D Maze
A.3.1 Environment Setup
We use the open-source gym-minigrid suite of environments to evaluate our method with one simple change — instead of the default observation space which is a  $7 \times 7$  egocentric window, our agent has access to entire grid — that allows us to simplify the tasks by removing partial observability [65].
A.3.2 Tasks
The tasks are grouped in three categories (please see Table 7 for example instructions):