Compositional Foundation Models for Hierarchical Planning
Authors: Anurag Ajay
Seungwook Han
Yilun Du
Shuang Li
Abhi Gupta
Tommi S. Jaakkola
Joshua B. Tenenbaum
Leslie Pack Kaelbling
Akash Srivastava
Pulkit Agrawal
Venue: NeurIPS 2023 poster (2023)
Keywords: Foundation Models
Composition
Hierarchical Planning
TL;DR: A foundation model which leverages multiple expert foundation model, trained individually on language, vision and action data, jointly together to solve long-horizon tasks

Abstract: To make effective decisions in novel environments with long-horizon goals, it is crucial to engage in hierarchical reasoning across spatial and temporal scales. This entails planning abstract subgoal sequences, visually reasoning about the underlying plans, and executing actions in accordance with the devised plan through visual-motor control. We propose Compositional Foundation Models for Hierarchical Planning (HiP), a foundation model which leverages multiple expert foundation model trained on language, vision and action data individually jointly together to solve long-horizon tasks. We use a large language model to construct symbolic plans that are grounded in the environment through a large video diffusion model. Generated video plans are then grounded to visual-motor control, through an inverse dynamics model that infers actions from generated videos. To enable effective reasoning within this hierarchy, we enforce consistency between the models via iterative refinement. We illustrate the efficacy and adaptability of our approach in three different long-horizon table-top manipulation tasks.

Compositional Foundation Models for Hierarchical Planning
Anurag Ajay" 18, Seungwook Han * Yilun Du *, Shuang Li
Abhi Gupta $, Tommi Jaakkola 8, Josh Tenenbaum" Leslie Kaelbling
Akash Srivastava +, Pulkit Agrawal
Improbable AI Labt MIT-IBM Watson Al Lab² Massachusetts Institute Technology
https://hierarchiee
fonng--foonnaioo-nnnnnen gi tithhb..o. io/
Abstract
To make effective decisions in novel environments with long-horizon goals, it is crucial to engage hierarchical reasoning across spatial and temporal scales. This entails planning abstract subgoal sequences, visually reasoning about the under- lying plans, and executing actions accordance with the devised plan through visual-motor control. We propose Compositional Foundation Models for Hierarchi- cal Planning (HiP), a foundation model which leverages multiple expert foundation model trained language, vision and action data individually jointly together solve long-horizon tasks. We use large language model to construct symbolic plans that grounded in the environment through large video diffusion model. Generated video plans are then grounded to visual-m mesuul-motor&&o control, through an in- verse dynamics model that infers actions from generated videos. To enable effective reasoning within this hierarchy, we enforce consistency between the models via iterative refinement. We illustrate the efficacy and adaptability of approach in three different long-horizon table-top manipulation tasks.
Introduction
Consider the task of making a cup of tea in an unfamiliar house. To successfully execute this task, an effective approach is to reason hierarchically at multiple levels: abstract level, e.g. the high level steps needed to heat up the tea, a concrete geometric level e.g., how we should physically navigate and in the kitchen, and control level e,g. how we should actuate our joints to lift cup. is further important that reasoning at each level is self-consistent with each other - an abstract plan to look in cabinets for kettles must also be physically plausible at the geometric level and executable given the actuations we are capable of. In this paper, we explore how we can create agents capable of solving novel long-horizon tasks which require hierarchical reasoning.
Large "foundation models" have become a dominant paradigm in solving tasks in natural language processing [36, 47, 7], computer vision [26]. and mathematical reasoning [27]. line with this paradigm, question of broad interest is to develop "foundation model" that can solve novel and long-horizon decision-making tasks. Some prior works [39, 6] collected paired visual, language and action data and trained monolithic neural network for solving long-horizon tasks. However, collecting paired visual, language and action data is expensive and hard to scale up. Another line of prior works [10, 28] finetune large language models (LLM) both visual and language inputs using task-specific robot demonstrations. This is problematic because, unlike the abundance of text the Internet, paired vision and language roboties demonstrations and are expensive to collect. Furthermore, finetuning high-performing language models, such as GPT3.5/4 [37, 36] and PaLM [7], is currently impossible, the model weights are not open-sourced.
readily available
denotes equal contribution. Authors also affiliated with Computer Science and Artificial Laboratory (CSAIL). Correspondence to aajaydmit , ewhandmit edu and yilundudmit. eed.u
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Internet
Internet Video
Egneebrric
Text.
Images
Mining www.rnnoooatttooo
Prepare Sandwach
text
Task Model
Visual
Action Model
Model
Compasitional Foundation Models for Hierarchical Planning. HiP uses a task model, represented using a LLM, to create an abstract plan, a visual model, represented using video model, generate an image trajectory plan, and an ego-centric action model to infer actions from the image trajectory.
Figure I:
The key characteristic of the foundation model is that solving new task or adapting to new environment is possible with much less data compared to training from scratch for that task or domain. Instead of building foundation model for long-term planning by collecting paired language- vision-action data, in this work we seek a scalable alternative can we reduce the need for a costly and tedious process of collecting paired data across three modalities and yet be relatively efficient at solving new planning tasks? We propose Compositional Foundation Models for Hierarchical Planning (HiP), foundation model that is composition of different expert models trained on language, vision, and action data individually. Because these models are trained individually. the data requirements for constructing the foundation models are substantially reduced (Figure 1). Given an abstract language instruction describing the desired task, HiP uses large language model to find a sequence of sub-tasks (i.e., planning). HiP then uses large video diffusion model to capture geometric and physical information about the world and generates a more detailed plan in form of observation-only trajectory. Finally, HiP uses large pre-trained inverse model that maps sequence of ego-centric images into actions. The compositional design choice for decision madision-making allows separate models reason different levels of the hierarchy, and jointly make expert decisions without the need for ever collecting expensive paired decision- maasionnmaking data across modalities.
Given three models trained independently, they can produce inconsistent outputs that can lead to overall planning failure. For instance, naive approach for composing models is to take the maximum- likelihood output at each stage. However, a step of plan which is high likelihood under one model, iee. looking for a tea kettle in cabinet may have zero likelihood under a seperate model, i.e. if there is no cabinet in the house. It is instead important to sample a plan that jointly maximizes likelihood across every expert model. To create consistent plans across our disparate models, we propose iterative refinement mechanism to ensure consistency using feedback from the downstream models [28]. At each step of the language model's generative process, intermediate feedback from likelihood estimator conditioned on an image of the current state is incorporated into the output distribution. Similarly, at each step of the video model generation, intermediate feedback from the action model refines video generation. This iterative refinement procedure promotes consensus among the different models and thereby enables hierarchically consistent plans that are both responsive to the goal and executable given the current state and agent. Our proposed iterative refinement approach is computationally efficient to train, as it does not require any large model finetuning. Furthermore, we do not require access to the model's weights and our approach works with any models that offer only input and output API access.
In summary, we propose compositional foundation model for hierarchical planning that leverages composition of foundation models, learned separately on different modalities of Internet and ego- centric robotics data, to construct long-horizon plans. We demonstrate promising results on three long-horizon tabletop manipulation environments.
2 Compostional Foundation Models for Hierarchical Planning
We propose HiP, foundation model that decomposes the problem of generating action trajectories for long-horizon tasks specified by a language goal into three levels hierarchy: (1) Task planning inferring language subgoal wy conditioned on observation 21, and language goal g: (2) Visual

---

Planning
Action Planning
Park conputer mousn,
22
Navis Nis sneakere train
Irefative Retibemedis
Trepative Refibeneent
Dynammic
Rovingmmes
Place dirty abjec in Mue box
Pluce computie mouse trown
Place computur monoe in heawnn bes
dt
Gemmand
Figure 2: Planning with HiP. Given language goal g and current observation LLM generates subgoal with feedback from visual plausibility model. Then, Diffusion uses observation 2; and subgoal w to generate observation trajectory 7z with feedback from an action feasibility model. Finally, action planning uses inverse dynamics to generate action a, from current and generated observation xf+1 (action planning).
planning generating a physically plausible plan sequence of image trajectories =-= (x J. one for each given language subgoal W, and observation at first timestep æ1,1; (3) Action planning inferring a sequence of action trajectories -.- = {a..1T-1) from the image trajectories Tz executing the plan. Figure 2 illustrates the model architecture and pseudocode is provided in Algorithm 1.
Let pe model this hierarchical decision- decision-making process. Given our three levels of hierarchy, pe can be factorized into the following: task distribution pe. visual distribution Pa. and action distribution Pv. The distribution over plans, conditioned on the goal and an image of the initial state, can be written under the Markov assumption as:
) (II
TNT-I
pe (W, (Tt) (1.)),
II po (10)
IIIII
Pv (an,e (I)
lusk plamning
visual planning
action planming
We seek to find action trajectories Th image trajectories T and subgoals W = {wy}} which maximize the above likelihood. Please see Appendix A for a derivation of this factorization. In the following sub-sections, we describe the form of each of these components. how they are trained, and how they are used to infer a final plan for completing the long-horizon task.
2.1 Task Planning via Large Language Models
Given task specified in language g and the current observation 24,1, use pretrained LLM the task planner, which decomposes the goal into a sequence of subgoals. The LLM aims to infer the next subgoal wy given goal and models the distribution PLLM (w; (wylg). As the language model is trained on a vast amount of data on the Internet, it captures powerful semantic priors on what steps should be taken accomplish particular task. To adapt the LLM to obtain subgoal sequence relevant to task, we prompt it with some examples of domain specific data consisting of high-level goals paired with desirable subgoal sequences.
However, directly sampling subgoals using a LLM can lead to samples that are inconsistent with the overall joint distribution in Eqn (1), as the subgoal ws not only affects the marginal likelihood of task planning but also the downstream likelihoods of the visual planning model. Consider the example in Figure where the agent is tasked with packing computer mouse, black and blue sneakers, pepsi box and toy train in brown box. Let's say the computer mouse is already in the brown box. While the subgoal placing computer mouse in brown box has high- -igh-likelihood under task model pe pe(wlgg) the resulting observation trajectory generated by visual model Pa(TT) w4, x²,1) will have low- low-likelihood under Pe given the subgoal is already completed. Next, we describe how we use iterative refinement to capture this dependency between language decoding and visual planning sample from Eqn (1).
Consistency with Visual Planning To ensure that we sample subgoal w; that maximizes the joint distribution in Eqn (1), we should sample subgoal that maximizes the following joint likelihood
i = arg max PLLM (w, (g) 1,,,, 21.1).
(2)
iee. a likelihood that maximizes both conditional subgoal generation likelihood from a LLM and the likelihood of sampled videos 7 given the language instruction and current image 4; One way to determine the optimal subgoal w is to generate multiple w; from LLM and score them using the likelihood of videos sampled from our video model P& P (T Ju,, .n,1). However, the video generation take a different approach.
process is computationally expensive,
3

---

Algorithm Decision Making with HaP
1: Models: Large language model PLlM, Subgoal classifier fo, Noise model of diffusion cø. Observation
trajectory classifier gv. Inverse dynamics py
2: Hyperparameters: Guidance scales w, w", Denoising diffusion steps K
3: Input: Current observation 2t, Language goal g
4: Task Planning
5: for i 1.... .M do
6: Generate subgoal PLLM (w:ss(ulll)
7: end for
8: Collect candidate subgoals W < ffr,hi h==1
9: Iterative Refinement from Visual Planning
10: w arg max,, [a(2,, W.g)
11: # Visual Planning
Initialize (T=) K ~ NV I)
13: for ks K...] do
14: # Ittrrative Refinement from Action Planning
15: é &+<a((TT) 3e, k)) ((ea((T))) zt, w, k) cs co((Ta),, (1))-wVe w'V(nn logg (11(7=)))
16: (T=) Denoise((
end for
18: Action Planning
19:
Extract (xz, x(21) from (Ta)o
20:
Execute 0, Pa (2, 2x41)
The likelihood of video generation (T) [1,,,,111 2(,1) primarily corresponds to the feasibility of a language subgoal wo with respect the initial image T1,1. Thus approximation of Equ (2) is to directly optimize the conditional density
= arg max p(.:lg, 24,1).
(3)
We rewrite Eqn (3)
wo arg max log PLIM(W)))g) + log p( (aellle.)) p(1;,199)
We estimate the density ratio a(aua//e)))) p( with a multi-class classifier fo( [w5] (M that g)
chooses the appropriate subgoal of from candidate subgoals {wj} generated by the LLM. The classifier implicitly estimates the relative log likelihood estimate of p(r;, [w;, g) and use these logits estimate the log density ratio with respect each of the M subgoals and find w that maximizes the estimate [45]. We dataset Delassify 1 9: {wy JM )== i} consisting of observation 1; goal L. g. candidate subgoals {w and the correct subgoal label to train fø. For further architectural details, please refer to Appendix B.I.
2.2 Visual Planning with Video Generation
Upon obtaining a language subgoal w, from task planning, our visual planner generates a plausible observation trajectory conditioned on current observation æ4,1 and subgoal w,. We use video diffusion model for visual planning given its success generating text-conditioned videos [20, 53]. To provide our video diffusion model with rich prior for physically plausible motions, we pretrain it (T (,,, on a large-scale text-to-video dataset Ego4D [13]. We then finetune it on the task-specifie video dataset Dvideo = [r] w/} consisting observation trajectories T satisfying subgoal wi. For further architectural details, please refer to Appendix B.2.
However, analogous to the consistent sampling problem in task planning, directly sampling observa- tion trajectories with video diffusion can lead to samples that inconsistent with the overall joint distribution in Eqn (1). The observation trajectory ff not only affects the marginal likelihood of visual planning, but also the downstream likelihood of the action planning model.
Consistency with Action Planning To ensure observation trajectories o that correctly maximize the joint distribution Eqn (1), we optimize an observation trajectory that maximizes the following joint likelihood
T-1
(T-) = arg max Poo (T 11
II
po (Tr,, 24,41)

---

i.e. an image sequence that maximizes both conditional observation trajectory likelihood from video diffusion and the likelihood of sampled actions Ta given the observation trajectory T.-
To sample such observation trajectory, could iteratively bias the denoising of video diffusion using the log-likelihood of the sampled actions ! log (a (1,,, Fi,++1). While this solution is principled, it is slow as it requires sampling of entire action trajectories and calculating the corre- sponding likelihoods during every step of the denoising process. Thus, we approximate the sampling and the likelihood calculation of action trajectory I[ Pv: Pa.(as,l/2s,, D1,,+1) with a binary classifier go (T(T)) that models if the observation trajectory T leads to a high-likelihood action trajectory.
We learn binary classifier PS to assign high likelihood feasible trajectories sampled from our video dataset T ~ Dvideo and low likelihood to infeasible trajectories generated by randomly shuffling the order of consecutive frames in feasible trajectories. Once trained, we can use the likelihood log Pv (1|T=) to bias the denoising of the video diffusion and maximize the likelihood of the ensuing action trajectory. For further details on binary classifier, please refer to Appendix C.2.
2.3 Action Planning with Inverse Dynamics
After generating an observation trajectory T from visual planning, our action planner generates an action trajectory T/ from the observation trajectory. We leverage egocentric internet images for providing our action planner with useful visual priors. Our action planner is parameterized as an inverse dynamics model [1, 38] that infers the action as,t given the observation pair (Zi,,, (1,11);
ai, Pe ( Zi,41)
Training To imbue the inverse dynamics Pv: with useful visual priors, we initialize it with VC- 1 [32] weights, pretrained on ego-centric images and ImageNet. We then finetune it on dataset Dun [Ty ] consisting of paired observation and action trajectories by optimizing:
max E [log P (ac, (,,, 11,+1)]
v
For further architectural details, please refer to Appendix B.3.
3 Experimental Evaluations
We evaluate the ability of HiP to solve monghooizonn planning tasks that are drawn from distributions with substantial variation, including the number and types of objects and their arrangements. We then study the effects of iterative refinement and of pretraining also compare against an alternative strategy of visually grounding the LLMM without any task-specific data. In addition, we study how granularity of subgoals affects HiP's performance, ablate over choice of visual planning model and analyze sensitivity of iterative refinement to hyperparameters (Appendix E).
overall performance of HiP. We
3.1 Evaluation Environments
We evaluate HiP on three environments, paint-block, object-arrange, and kitchen-tasske which are inspired by combinatorial planning tasks in Mao et al. [33], Shridhar et al. [43] and Xing al. [50] respectively.
paint-blocks robot has to manipulate blocks in the environment to satisfy language goal instructions, such as stack pink block on yellow block and place green block right of them. However, objects of correct colors may not be present in the environment, which case, the robot needs to first pick up white blocks and put them in the appropriate color bowls to paint them. After that, it should perform appropriate pick-and-place operations to stack a pink block on the yellow block and place the green block right of them. A new task T is generated by randomly selecting 3 final colors (out of 10 possible colors) for the blocks and then sampling a relation (out of 3 possible relations) for each pair of blocks. The precise locations of individual blocks, bowls, and boxes fully randomized across different tasks. Tasks have ~ subgoals.
boject-arrange A robot has to place appropriate objects in the brown box to satisfy language goal instructions such as place shoe, lablet, alarm clock, and scissor in brown box. However, the environment may have distractor objects. Furthermore, some objects can be dirty, indicated by a lack of texture and yellow color. For these objects, the robot must first place them in a blue cleaning box and only afterwards place those objects in the brown box. A new task T is generated
5

---

"Stack green block top of yellow block and place orange block the right of the stack"
Paint Block Task
white block yellow
place block brown
white block green
place block yellow
place white block. orange
block
right
Rearrange Object Task
"Pack hammer, frypan, toy achool bus, and butterfinger chocolate into the brown box'
place dany object the clean box
place hammer the brown box
place frypon brown box
place tay school bus brown
place butterfingee chocolate brown
"Open mierowave, kettle out of the way, light the kitchen and open upper right drawer"
Kitchen Task
open the
the lettle
the lights
sbile the right
Figure 3: Example Executions. Example Long-horizon generated plans on tasks paint object-arrange, and kitchen-taasee tasks domains.
paint-block. block.
by randomly selecting 7 objects (out of 55 possible objects), out of which 3 are distractors, and then randomly making one non-distractor object dirty. The precise locations individual objects and boxes are fully randomized across different tasks. Tasks usually have 3 ~ 5 subgoals.
kitchen-tasker A robot has to complete kitchen subtasks to satisfy language goal instructions such as open microwave, move kettle out of the way, light the kitchen area, and open upper right drawer. However, the environment may have objects irrelevant to the subtasks that the robot must ignore. Furthermore, some kitchen subtasks specified in the language goal might already be completed, and the robot should ignore those tasks. There are 7 possible kitchen subtasks: opening the microwave, moving the kettle, switching on lights, turning on the bottom knob, turning on the top knob, opening the left drawer, and opening the right drawer. A new task T is generated by randomly selecting out of possible kitchen subtasks, randomly selecting an instance of microwave out of possible instances, randomly selecting instance of kettle out of 4 possible instances, randomly and independently selecting texture counter, floor and drawer out possible textures and randomizing initial pose of kettle and microwave. With 50% probability, one selected kitchen subtask completed before the start of the task. Hence, tasks usually have 3 ~ 4 subtasks (i.c. subgoals).
Train and Test Tasks For all environments, we sample two sets of tasks Ttrains Trest P(T). We use the train of tasks Tirain create datasets Delusity. D Din and other datasets required for training baselines. We ensure the test set of tasks Trest contains novel combinations of object colors paint-block novel combinations of object categories in object- aaraanee and novel combinations of kitchen subtasks in kitchen-tasks.
Evaluation Metrics We quantitatively evaluate model by measuring its task completion rate for paint-block and object- and subtask completion rate for kitchen tasks. We use simulator to determine if the goal, corresponding on Terain (seen) to test ability to solve long-horizon tasks and on Trest (unseen) to its ability to generalize to long- horizon tasks consisting of novel combinations of object colors in paint-bloo block,
task, has been achieved. We evaluate model

---

Paint-bles
Obj Object-arrange arrange
Katchen-tasks
Model
Seen
Unseen
Seen
Unseen
Seen
Unseen
Transformer BC (oracle subgoals)
8.3 1.9
5.1 1.6
10.2
7.3
48.4 + 21.6
32.1 24.2
Gato (oracle subgoals)
31.2 + 2.4
28.6 299
37.9 = 3.3
06.5
70.2 = 10.8
66.8 = 12.2
Trajectory Transformer (oracle subgoals)
22.3
30.5
29.8 2.9
06.4 20.7
52.1
Action Difluser (oracle subgnals)
21.6 + 2.6
18.2 + 2.3
29.2 2.4
27.6 2.1
65.9 + 23.2
55.1 + 22.8
HiP (Ours, oracle subgouls)
81.2 1.8
79.6
91.8
92.3 2.3
92.8 7.1
89.8 1: 7.6
UniPi
37.2
3.8
35.3
44.1
11.2.22
74.6 14.8
73.4
SayCan
67.2 1: 3.3
62.8
1: 3.7
70.3
2.6
66.9 -: 2.8
HiP (Ours)
74.3 72.8
1.7
75
7.8
75.4
85.8 9.4
83.5
10.2
Table I: Performance Long-Horizon tasks. HiP not only outperforms the basclines in solving seen oonghhoiizon tasks but its performance remains intact when solving unseen long ong-horizon tasks containing novel combination of objects colors in paint-block, novel combination of objects categorics in Poject-rearrane and novel combination of subtasks in kitchen-tasks.
and kitchen subtasks in kitchen-tasks. We sample 1000 tasks from Ttrain and Trest respectively, and obtain average task completion rate on paint-bloch and abject-arrange arrange domains and average subtask completion rate on kitchen tasks domain. We report the mean and the standard error over seeds in Table 1.
object categories in object-
3.2 Baselines
There are several existing strategies for constructing robot manipulation policies conditioned on language goals, which we use as baselines in our experiments:
Cond-Conditioned Policy A goal-conditioned transformer as, ~ p(a,,t/ wx) that outputs action di,t given a language subgoal wy and current observation Fi,t (Transformer BC) [6]. We provide the model with oracle subgoals and encode these subgoals with a pretrained language encoder (Flan-T5-ae TS-ase).. We also compare against goal-conditioned policy with Gato [39] transformer.
p(tr: Oi, p(a., Fi,t11 that bypasses task planning, generates video plans for the entire task {r]}- and infers actions as.t using an inverse model.
Video Planner video diffusion model (UniPi) [12] {T
Action Planners Transformer models (Trajectory Transformer) [23] and diffusion models (Dif- fuser) [24, 4] {a.,aT-i}} ~ - w)) that produce an action sequence {a:,:T-1} given language subgoal wo and current visual observation Fi,t. We again provide the agents with oracle subgoals and encode these subgoals with a pretrained language encoder (Flan-T5 (--a--T--Bas))
LLM as Skill Manager A hierarchical system (SayCan) [2] with LLM high level policy that sequences skills sampled from a repetoire of skills to accomplish long-horizon task. We use CLIPort [43] policies as skills and the unnormalized logits over the pixel space it produces as affordances. These affordances grounds the LLM to current observation for producing next subgoal.
3.3
Results
We begin by comparing the performance of HiP and baselines to solve long-horizon tasks in paint-block, object-arrange kitchen-taske environments. Table shows that HiP signifi- cantly outperforms the baselines, although the baselines have an advantage and have access to oracle subgoals. HiP's s superior performance shows the importance of (i) hierarchy given it outperforms goal- conditioned policy (Transformer BC and Gato), (ii) task planning since it outperforms video planners (UniPi), and (iii) visual planning given outperforms action planners (Trajectory Transformer, Action Diffuser). It also shows the importance of representing skills with video-based planners which can be pre-trained on Internet videos and can be applied to tasks (such as titchenttssks SayCan, in contrast, requires tasks to be decomposed into primitives paired with an affordance function, which can be difficult to define for many tasks like the kitchen task. Thus, we couldn't 't run SayCan on witcheenttast environment. Finally, to quantitatively show how the errors in fa( wa, g) affect the performance of HiP, we compare to HiP with oracle subgoals. For further details on the training and evaluation of HiP, please refer to Appendix C. For implementation details on Gato and SayCan, please refer to Appendix D. For runtime analysis HiP, please refer to Appendix F.
Combinatorial Generalization to Unseen Long-horizon Tasks We also quantitatively test the ability of HiP to generalize to unseen long-horizon tasks, consisting of novel combinations of object colors in paint-block, object categories object-arrange and subtasks in ki technchtaaee taaks.. Table I shows that HiP's performance remains intact when solving unseen hong-horizon tasks, and
7

---

Subgnal trajectories
plane white bloc pallow
Initial observation
pluze yelioe
place while llock
Goal
Stack green block on top yellow block and plare orange
placs grecn tkek in pollen block
plaze whiie blok in orange
plaze blck
right
Figure Execution trajectory of HiP on an novel long-horizon task in paint- -blck environment.
still significantly outperforms the baselines. Figure 4 visualizes the execution of HiP in unseen long-horizon tasks in paint-block.
RgrDD Predtmining
Pre-triningg
Paint Block
Rearrange
Kitchen Tacks
PVD
100
108h Tola Data
75t Trile Dro
105 Trais Tata
100% Tola Dats
75h Tril Dais
50% Truss Tatn
1053 Trile Dato
520 Thele Boos
50% Tosls Dula
Figure 5: Pretraining video diffusion model with the Ego4D dataset consistently yields higher success and lower scores (lower is better), even reduced training dataset sizes. With pretraining, the model's FVD score escalates less gradually and its success rate falls less steeply as the dataset size shrinks.
Pre-training Video Diffusion Model We investigate how much our video diffusion model benefits from pre-training on the Internet-scale data. We report both the success rate HiP and Fréchet Video Distance (FVD) score that quantifies the similarity between generated videos and ground truth videos, where lower scores indicate greater similarity Figure 5. We see that pretraining video diffusion leads to higher success rate and lower FVD score. If we reduce the training dataset to 75% and 50% of the original dataset, the FVD score for video diffusion models (both, with and without Ego4D dataset pretraining) increases and their success rate falls. However, the video diffusion model with Ego4D dataset pretraining consistently gets higher success rate and lower FVD scores across different dataset sizes. As decrease the domain-specinf training data, it is evident that the gap in performance between the model with and without the Ego4D pre- pre-training widens. For details on how process the Ego4D dataset, please refer Appendix C.2.
Pre-training Inverse Dynamics Model We also analyze the benefit of pre-training our inverse dynamies model and report the mean squared error between the predicted and ground truth actions in Figure 6. The pre-training comes in the form of initializing the inverse dynamics model with

---

Paint Block
Object Arrange
Kitchen Tasks
0.10
0.10
Redde boobd)
Robbe la (onte))
Rodks la (wlures)
Moen Squard Bree
0.08
V2-- 13
Vild serrads
0.08
VT--3 (W-4, 13
Wid verrady
Mosn Squad Eave
0.08
wra (C-1) 2.53
Wial versa))
V.2-8 13k
13k)
(emia. 19M
0.04
0.04
0.04
Meen
00
00
0.00
2500
5000
4000
5000
Number denvestratios
Nomber
Namber
denoonteations
seatchen-taskee when initialized with VC-1 weights, inverse dynamics model matches the performance of a randomly initialized model trained on 10K trajectories with just 1K (3.5k) trajectories. smaller ResNet- 18 model requires 2.5K (6k) trajectories to approach the same performance. The yellow and brown lines are overlaid on top of each other.
Figure 6: Pretraining inverse dynamics model. In paint-block block and object-arrange
weights from VC-1 [32], a vision-transfomnn (ViT-B) [9] trained on ego-centric images with masked- autoencoding objective [16]. In paint-block and object-arrange. we see that the inverse dynamics, when initialized with weights from VC-1, only requires 1K labeled robotic trajectories to achieve the same performance as the inverse dynamics model trained on 10K labeled robotic trajectories but without VC-1 initialization. We also compare against an inverse dynamics model parameterized with a smaller network (ResNet-18) However, the resulting inverse dynamics model still requires 2.5K robotic trajectories to get close to the performance of the inverse dynamics model with VC-1 initialization in paint-block and object-arrange. In sritchen-taskee dynamics, when initialized with weights from VC-1, only requires 3.5k labeled robotic trajectories to achieve the same performance as the inverse dynamics model trained on 10K labeled robotic trajectories but without VC-1 initialization. When parameterized with ResNet-18, the inverse dynamics model still requires 6k robotic trajectories to get close to the performance of the inverse dynamics model with VC-1 initialization.
inverse
Iverutvve Refinement
Visually Grounding LLM
Prediction Acturnty
Saress Rase
80
80
Learnel Cheedfier
60
60
60
Fromu VLM
20
20
HIP
H.P without
HiP withort any Refinement
Paint Black
Bearnnge Ohjjct
Viual Refiomentt
Figure 7: Ablation Studies. (Left) While task plan refinement is critical to HiP's performance, visual plan refinement improves HiP's performance by smaller margin in paint block environment. (Right) While frozen pretrained VLM (MiniGPT4) matches the performance of learned classifier in saint-bloch environment, its performance deteriorates in more visually complex object- arr ange environment.
Importance of Task Plan and Visual Plan Refinements We study the importance of refinement in task and visual planning in Figure 7. We compare HiP without visual plan refinement and HiP without visual and task plan refinement in paint block environment. We see that task plan refinement for visual grounding of LLM critical the performance of HiP. Without it, the task plan is agnostic to the robot's observation and predicts subgoals that lead to erroneous visual and action planning. Furthermore, visual plan refinement improves the performance of HiP as well, albeit by small margin. For a description of the hyperparameters used, please refer to Appendix C.4.
Exploring Alternate Strategies for Visually Grounding LLM We use a learned classifier fo(21,1, w. 9) to visually ground the LLM. We explore if we can use a frozen pretrained Vision- Language Model (MiniGPT-4 [58]) as a classifier in place of the learned classifier. Although we didn't use any training data, we found the prompt engincering using the domain knowledge of the task to be essential in using the Vision-Language Model (VLM) as a classifier (see Appendix C.1 for details). We use subgoal prediction accuracy to quantify the performance of the learned classifier and the frozen VLM. Figure 7 illustrates that while both our learned multi-class classifier the paint-block environment, the classifier significantly
and frozen VLM perform comparably outperforms the VLM in the more visually complex abject-arrange environment. We detail the two common failure modes of the VLM approach in object- aarangee environment Appendix C.I. As VLMs continue to improve, it is possible that their future versions match the performance of

---

learned classifiers and thus replace them in visually complex domains as well. For further details on the VLM parameterization, please refer to Appendix C.I.
4 Related Work
The field of foundation models for decision-making [52] has seen significant progress in recent years. A large body of work explored using large language models as zero-shot planners [21, 22, 29, 30, 2], but often difficult to directly ground the language model on vision. To address this problem of visually grounding the language model, other works have proposed to directly fine-tune large language models for embodied tasks [28, 10]. However, such an approach requires large paired vision and language datasets that are difficult to acquire. Most similar to our work, SayCan [2] uses an L.M to hierarchically execute different tasks by breaking language goals into a sequence of instructions, which are then inputted to skill-based value functions. While SayCan assumes this fixed set of skill-based value functions, our skills are represented as video-based planners [12], enabling generalization to new skills.
Another set of work has explored how to construct continuous space planners with diffusion mod- els [25, 4, 49, 55, 48, 57, 12]. Existing works typically assume task-specific datasets from which the continuous-space planner is derived [24, 3, 55]. Most similar to our work, UniPi [12] proposes to use videos to plan in image space and similarily relies on internet videos train image space planners. We build on top of UniPi to construct our foundation model for hierarchical planning, and illustrate how UniPi may be combined with LLMs to construct longer horizon continuous video plans.
Moreover, some works [28, 54, 5] explored how different foundation models may be integrated with each other. In Flamingo [5], models are combined through joint finetuning with paired datasets, which difficult collect. In contrast both Zeng et al. [54] and Li et al. [28] combine different models zero-shot using either language or iterative consensus. Our work proposes to combine language, video, and ego-centric action models together by taking the product of their learned distributions [11]. We use a similar iterative consensus procedure as in Li et al. [28] to sample from the entire joint distribution and use this combined distribution construct a hierarchical planning system.
Limitations and Conclusion
Limitations Our approach has several limitations. As high-quality foundation models for visual sequence prediction and robot action generation do not exist yet, our approach relies on smaller-scale models that we directly train. Once high-quality video foundation models available, we can use them to guide our smaller-scale video models [51] which would reduce the data requirements of our smaller-scale video models. Furthermore, our method uses approximations to sample from the joint distribution between all the model. An interesting avenue for future work is to explore more efficient and accurate methods to ensure consistent samples from the joint distribution.
Conclusion In this paper, we have presented an approach to combine many different foundation models into a consistent hierarchical system for solving long-horizon robotics problems. Currently, large pretrained models are readily available in the language domain only. Ideally, one would train a foundation model for videos and ego-centric actions, which we believe will be available in the near future. However, our paper focuses on leveraging separate foundation models trained on different modalities of internet data, instead of training single big foundation model for decision making. Hence, for the purposes of this paper, given our computational resource limitations, we demonstrate our general strategy with smaller-scale video and ego-centric action models trained in simulation, which serve as proxies for larger pretrained models. We show the potential of this approach in solving three long- horizoo robot manipulation problem domains. Across environments with novel compositions of states and goals, our method significantly outperforms the state-of of-e--of approaches towards solving these tasks.
In addition to building larger, more work suggests the possibility of further using other pretrained models in other modalities, such as touch and sound, which may be jointly combined and used by our sampling approach. Overall, our work paints a direction towards decision making by leveraging many different powerful pretrained models in combination with tiny bit of training data. We believe that such a system will be substantially cheaper to train and will ultimately result more capable and general-purpose decision making systems.
visual sequence and robot control models, our
10

---

Acknowledgements
The authors would like to thank the members of Improbable AI Lab for discussions and helpful feedback. We thank MIT Supercloud and the Lincoln Laboratory Supercomputing Center for providing compute resources. This research was supported by an NSF graduate fellowship. a DARPA Machine Common Sense grant, ARO MURI Grant Number W91 1 1911NF210027, ONR MURI Grant Number N00011-22-1-2777
ARO MURI and an MIT-IBM Watson AI Lab grant. The views and conclusions contained in this document are those of the authors and should not be interpreted representing the official policies, either expressed or implied, of the United States Army Research Office, the Office of Naval Research, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes, notwithstanding any copyright notation herein.
Author Contributions
Anurag Ajay co-conceived the framework of leveraging pretrained foundation models for decision making, implemented visual planning and action planning in HiP, evaluated HiP on long-horizon tasks, performed ablation studies and helped in paper writing.
Seungwook Han co-conceived in conceiving the framework of leveraging pretrained foundation models for decision making, implemented task planning in HiP and helped in paper writing.
Yilun Du co-conceived the framework of leveraging pretrained foundation models decision making, implemented trajectory transformer and transformer BC and evaluated them on long-horizon tasks, implemented data generation scripts for object arrange and paint block environment, and lead paper writing.
Shuang Li helped in conceiving the idea iterative refinement for consistency between pretrained foundation models, participated in research discussions and helped in writing paper.
Abhi Gupta participated in research discussions and helped in making figures.
Tommi Jaakkola participated in research discussions.
Joshua Tenenbaum participated in research discussions.
Leslie Kaelbling participated in research discussions, suggested baselines and ablation studies, conceived the structure of the paper and helped in paper writing.
Akash Srivastava participated research discussions, suggested the idea using classifier for consistency between observations and the large language model and provided feedback paper writing.
Pulkit Agrawal was involved in research discussions, suggested ablation studies related to iterative refinement, provided feedback on writing, positioning of the work and overall advising.
References
[1] P. Agrawal, A. V. Nair, P. Abbeel, J. Malik, and Levine. Learning to poke by poking: Experiential learning of intuitive physics. Advances in neural information processing systems, 29, 2016.
[2] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, al. Do can, not say: Grounding language robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
[3] A. Ajay, A. Kumar, P. Agrawal, S. Levine, and O. Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010. / 2020.
[4] A. Ajay, Y. Du, A. Gupta, J. Tenenbaum, T. Jaakkola, and Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657 2022.
[5] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lene, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: visual language model for few-shot learning. NeurlPS, 2022.
URL https:/ hamps://arree //arxiv....// org/ 2b2/ 14198.

---

[6] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv 2022.
preprint arXiv:2212.068/
[7] A. Chowdhery, S. Naraagg, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.022 02311, 2022.
[8] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. De- hghani, S. Brahma, et al. Scaling -inettuee- language models. arXiv preprint arXiv:2210.11416, 2022.
[9] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1999, 2020.
[10] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprini a3Xiv:230.00337, 2023.
[1]] Y. Du, S. Li, and I. Mordatch. Compositional visual generation with energy based models. In Advances in Neural Information Processing Systems, 2020.
[12] Y. Du, M. Yang, B. Dai, H. Dai, O. Nachum, B. Tenenbaum, Schuurmans, and P. Abbeel. Learning universal policies via text-guided video generation. arXiv e-prints, pages arXiv-2302, 2023.
[13] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995-19012, 2022.
[14] M. U. Gutmann and A. Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications natural image statistics. J. Mach. Learn. Res., 13:307-361, 2012.
[15] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778. 2016.
[16] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[17] J. Hermans, V. Begy, and G. Louppe. Likelihood-free meme with amortized approximate ratio estimators. In International conference on machine learning, pages
PMLR, 2020.
[18] J. Ho and T. Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207. 12598, 2022.
[19] J. Ho, A. Jain, and P. Abbecl. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.
[20] Ho. W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02300 12.02203, 2022.
[21] W. Huang, P. Abbeel, D. Pathak, and Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pages 9118-9147. PMLR, 2022.
[22] W. Huang, F. Xia, T. Xiao, H. Chan, J. L Liang, P. Florence, A. Zeng. J. Tompson, I. Mordatch, Chebotar, et Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2007.05608 2022.
12

---

[23] M. Janner, Q. Li, and S. Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in Neural Information Processing Systems, 2021.
[24] M. Janner, Y. Du, J. Tenenbaum, and S. Levine. Planning with diffusion for flexible behavior synthesis. International Conference on Machine Learning. 2022.
[25] M. Janner, Y. Du, J. Tenenbaum, and S. Levine. Planning with diffusion for flexible behavior synthesis. International Conference on Machine Learning. 2022.
[26] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643 026..,2643222 2023.
[27] A. Lewkowyez, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Sola et al. Solving quantitative reasoning problems with language models. arXiv preprint ar arXiv:2206. 14858, 2022.
[28] S. Li, X. Puig, C. Paxton, Y. Du, C. Wang, L. Fan, T. Chen, D.-A. Huang, E. Akyürek, A. Anandkumar, al. Pre-trained language models for interactive decision-making. Advances in Neural Information Processing Systems, 35:31 199-31212, 2022.
[29] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:2209.07753, 2022.
[30] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Tex:2motion: From natural language instructions feasible plans. arXiv preprint arXiv:2303.12155 2023.
[31] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint 100 2017.
[32] A. Majumdar, K. Yadav, S. Arnad,, Y. J. Ma, C. Chen, S. Silwal, A. Jain, V.-P. Berges, Abbeel, J. Malik, et al. Where are we in the search for an artificial visual cortex for embodied intelligence? arXiv preprint arXiv:2303. 18240, 2023.
[33] J. Mao, T. Lozano-Perez, B. Tenenbaum, and L. P. Kaelbing. PDSketch: Integrated Domain Programming, Learning, and Planning. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[34] Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint 2rXiv:2033.1266 2022.
[35] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112. 2021.
[36] OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
[37] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[38] D. Pathak, P. Mahmoudieh, G. Luo, P. Agrawal, D. Chen, Y. Shentu, E. Shelhamer, J. Malik, A. A. Efros, and T. Darrell. Zero- shot visual imitation. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 2050-2053. 2018.
[39] S. Reed, K. Zolna, E. Parisotto, G. Colmenarejo. A. Novikov, G. Marth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, et al. A generalist agent. arXiv preprint arXiv:205..06175. 2022.
[40] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
13

---

[41] C. Saharia, W. Chan, S. Saxena, L. J. Whang, E. Denton, S. K. S. Ghasemipour, B. K. Ayan, S. S. Mahdavi, G. Lopes, al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205 1488 2022.
[42] T. Salimans and J. Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint 222.::2222:0000001122 2022.
[43] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipula- tion. In Conference on Robot Learning, pages 894 906. PMLR, 2022.
[44] Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, 2015.
[45] A. Srivastava, S. Han, K. Xu, B. Rhodes, and M. U. Gutmann. Estimating the density ratio between distributions with high discrepancy using multinomial logistic regression. ArXiv, abs/2305.008s 00066, 2023.
[46] M. Sugiyama, T. Suzuki, and T. Kanamori. Density ratio estimation in machine learning. Cambridge University Press, 2012.
[47] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[48] Urain, N. Funk, Chalvatzaki, and J. Peters. Se difiusssuiossfss::::: Learning cost functions for joint grasp and motion optimization through diffusion. arXiv preprint arXiv:2209.03855: 2022.
[49] Z. Wang, J. J. Hunt, and M. Zhou, Diffusion policies as an expressive policy class for offline reinforcement learning, arXiv preprint arXiv:2208.06/ 2022.
[50] E. Xing, A. Gupta, S. Powers, and V. Dean. Kitchenshift: Evaluating zero-shot general- ization of imitation-based policy learning under domain shifts. In NeurlPS 202/ Work- shop Distribution Shifts: Connecting Methods and Applications, 2021. URL https:
/ //openreview,
[51] M. Yang, Y. Du, B. Dai, D. Schuurmans, J. B. Tenenbaum, and P. Abbeel. Probabilistic adaptation of text-to-video models. arXiv preprint arXiv:2306.0183 01872 2023.
[52] S. Yang, O. Nachum, Y. Du, J. Wei, P. Abbeel, and D. Schuurmans. Foundation models for decision making: Problems, methods, and opportunities. arXiv preprint arXiv:2303.04/2) 2023.
[53] S. Yu, K. Sohn, S. Kim, and J. Shin. Video probabilistic diffusion models in projected latent space. arXiv preprint 230.:22302006885, 2023.
[54] A. Zeng, A. Wong, S. Welker, K. Choromanski, F. Tombari, A. Purohit, M. Ryoo, V. Sindhwani,
J. Lee, V. Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint rxXi:22244000998 2022.
[55] E. Zhang, Y. Lu, W. Wang, and A. Zhang. Lad: Language augmented diffusion for reinforcement learning. arXiv preprint 10Xiv:2210./555 /5629, 2022.
[56] H. Zheng, W. Nie, A. Vahdat, K. Azizzadenesheli, and A. Anandkumar. Fast sampling of diffusion models via operator learning. International Conference on Machine Learning. pages 42390-42402. PMLR, 2023.
[57] Z. Zhong, D. Rempe. D. Xu, Y. Chen, S. Veer, T. Che, B. Ray, and M. Pavone. Guided conditional diffuuson for controllable traffic simulation. arXiv preprint arXiv:2210.17366. 2022.
[58] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.1055 2023.
14

---

Appendix
In this Appendix, we discuss how we factorize the hierarchical decision decision-making process in Section A. In Section B, we then detail the background and architecture for visually grounded task planning, visual planning with video diffusion, and action planning with inverse dynamics model. Section C, discuss the training and evaluation details for the different levels of planning in HiP and the corresponding training hyperparameters. In Section D, we discuss implementation details for Gato and SayCan. In Section E, we showcase additional ablation studies comparing different approaches to enforce consistency across the levels of hierarchy, analyzing effect of granularity of subgoals on performance of HiP, ablating the choice of video planning model and analyzing sensitivity of iterative refinement to hyperparameters. Finally, in Section F, we analyze runtime of different components of HiP.
A Factorizing Hierarchical Decision-Making Process
We model the hierarchical decision-making process described in Section 2 with pe which can be factorized into the task distribution pe. visual distribution pá. and action distribution p
N
N
II 2=1
pa (w Ti, Wess
II
PolT 2., 21,,
II Pe Po (Tll 21 To T2
Here, given random variables yi, y< and y represents {YL ..... "y.....,y-.) y and (Y), ...., Y respectively. Now, we apply Markov assumption: Given current observation 2i,1, future variables (wy, Ti T.) and past variables (wy, TH, Td Vj i) are conditionally independent.
pe (W, (22)) (THH,, II (o 11 II po w 21,19)
p (rlllt wi, 21,1 II
We model task distribution pe with a large language model (LLM) which independent of observation Ti.1- Since the image trajectory T = {Ii11:T) describes physically plausible plan for achieving subgoal w; from observation 24,1, it is conditionally independent of goal g given subgoal w; and observation T;2 Furthermore, we assume that an action ai, can be recovered from observation at the same timestep Fi, and the next timestep Ti,441- Thus, we can write the factorization as
AV [Iootvils)
NV
(W,,, (WW. (1
II
(a
B
Background and Architecture
B.1 Task Planning
Background on Density Ratio Estimation Let p and g be two densities, such that g is absolutely continuous with respect to p. denoted << p ie. q(x) 0 wherever p(x) > 0. Then, their ratio defined as r(æ) = p(x))/((a) over the support of p. We can estimate this density ratio r(æ) by training binary classifier to distinguish between samples from p and [46, 14. 17]. More recent work [45] has shown one can introduce auxiliary densities {m.). and train a multi-class classifier to distinguish samples between M classes to learn belter-calibrated and more accurate density ratio estimator. Once trained, the log density ratio can be estimated by log r(x) h,,(x)) (x), where (x)))) is the unnormalized log probability of the input sample under the ith density, parameterized by the model.
Learning Classifier to Visually Ground Task Planning We estimate the density ratio afa.a/ww..p
p(2.119)
with multi-class classifier (25,1, (wy (.g)) trained to distinguish samples amongst the conditional distributions p(ri,1 (KT.,//W,,, (.....))... -.., p(r;,1 [WM,9) g) and the marginal distribution p( Upon conver- gence, the classifier learns to assign high scores to (It,1, w; g) w; the subgoal corresponding to the observation zi,1 and task g and low scores otherwise.
15

---

Architecture We parameterize fa as 4-layer multi-layer perceptron (MLP) on top of an ImageNet- pretrained vision encoder (ResNet-18 [15]) and a frozen pretrained language encoder (Flan-T5- Base [8]) The vision encoder encodes the observation 1;,1, and the text encoder encodes the subgoals Wy and the goal g. The encoded observation, the encoded subgoals, and the encoded goal are concatenated, and passed through MLP with 3 hidden layers of sizes 512, 256, and 128. The output dimension for MLP (i.e., number of classes for multi-clasiffiatioon) environment, 5 for
is 6 for paint-blocs environment and 4 for witccen-ttaske environment.
Choice of Large Language Model
We use GPT3.5-turbo [37] as our large language model.
B.2
Visual Planning
Background Diffusion Probabilistic Models [44, 19] learn the data distribution h(x)) from dataset == = {. The data- geneeranng procedure involves predefined forward noising process g (2*x) and trainable reverse process
[x)). both parameterized as conditional Gaussian distributions. Here, 2g = x is sample, 21, 22, 2x are the latents, and æx (V I) for a sufficiently large K. Starting with Gaussian noise, samples are then iteratively generated through a series of "denoising" steps. Although tractable variational lower-bound on log Ps can be optimized to train diffusion models, Ho al. [19] propose simplified surrogate loss:
Ldcnoise (0) = Ex- (... Illlc (xxx A)]
The predicted noise (x k),, parameterized with deep neural network, estimates the noise (0, added to the dataset sample zu to produce noisy Ø*.
Guiding Diffusion Models with Text Diffusion models are most notable for synthesizing high- quality images [41, 35] and videos [20, 53] from text descriptions. Modeling the conditional data distribution g(æly)) makes possible to generate samples satisfying the text description y. To enable conditional data generation with diffusion, Ho and Salimans [18] modified the original training setup to learn both a conditional es (Zk, y, k) and an unconditional es (x k) model for the noise. The unconditional noise is represented. in practice, the conditional noise cx( 0, k). where dummy value 9 takes the place of y. The perturbed noise eø 0, k) + (Eg((TR. y, k) < 0, k)) (i.e. classifier-free guidance) is used to later generate samples.
Video Diffusion in Latent Space As diffusion models generally perform denoising in the input space [19], the optimization and inference become computationally demanding when dealing with high-dimensional data, such as videos. Inspired by recent works [40, 53]. we first use an autoencoder Venc to learn a latent space for our video data. It projects
observation trajectory Ta (i.e., video) into W capture variations in the video across time, height, and width respectively. We then diffuse over this learned latent space [53].
a 2D tri tri-plane representation [53] T= [T 7 W] where 7-', T T
Latent Space Video Diffusion for Visual Planning Our video diffusion model Po (T)) x1,1) generates video The given a language subgoal Wi and the current observation T;,1- It is param- eterized through its noise model cs cs((T))), wi, 1i,1, k) = coT(
(2,,)) k) where T/ = Vene((e) (T the latent representation of video oh over which we diffuse. We condition the noise model 6g on subgoal wy using a pretrained language encoder lnne and on current observation x4,1 using video encoder Venc- To use Uenc with a single observation T4,1, we first tile the observation along the temporal dimension to create video.
Architecture
We now detail the architectures of different components:
Video Autoencoder We borrow our architecture for Venc from PVDM [53] which uses transformers
to project video Tz E Hx to latent codes т₃ = [ HH T where 7 E RCxH H' xW'
H E RC W E C xT_ Here, T= 50 represents the time horizon of a video, = 48 represents video height, W 64 represents video width, C 4 represents latent codebook dimension, H' = 12 represents latent height, and W = 8 represents latent width.
Language Encoder We use Flan-TS-Base [8] as the pretrained frozen language encoder Jene-
Noise Model We borrow PVDM-L architecture [53] which uses 2D UNet architecture, similar to the one in Latent Diffusion Model (LDM)) [40], to represent p(T=lr²). In our case, т= = Gene (T))
and T = Veen (. To further condition noise model co Tene (w:), augment the 2D UNet Model with cross-attention mechanism borrowed by LDM [40].
For implementing these architectures, we used the codebase http:///gthhh...ooo/sslyyn////// which contains the code for PVDM and LDM.
16

---

Classifier for Consistency between Visual Planning and Action Planning To ensure consisteney between visual planning and action planning, we want to sample observation trajectories that maxi- mizes both conditional observation trajectory likelihood from diffusion and the likelihood of sampled actions given the observation trajectory (see equation 4). To approximate likelihood calculation of action trajectory, we learn binary classifier Be that models if the observation trajectory leads to a high likelihood action trajectories. Since diffusion happens in latent space and we use gradients from Ie to bias the denoising of the video diffusion, 9v (T=)) takes the observation trajectory in latent space. The binary classifier 9y trained to distinguish between observation trajectories in latent space sampled from video dataset = = (ene(Ty); T ~ Dvideo (i.e. label of 1) and observa- tion trajectories in latent space sampled from video dataset whose frames where randomly shuffled (r)) Pone (a (aT)) T Viidoo (i.e. label of 0). Here, a denotes the random shuffling of frames. To randomly shuffle frames in observation trajectory (of length 50), we first randomly select 5 frames in the observation trajectory. For each of the selected frame, we randomly permute it with its neighboring frame (i.e. either with the frame before it with the frame after it). Once Ib is trained, we use it to bias the denoising of the video diffusion
غ = <6 < Venc (xt),, k) - (o(((((T))) Menc( (24),, Lene (w), k)) Es ((T2) Vone (x))
- -w' (r=)) (Tz)s log 94 (1)(
Here, è is the noise used in denoising of the video diffusion and w, w' are guidance hyperparameters. Classifier Architecture The classifier (T = [T- - 7 has a ResNet-9 encoder that converts T3' T! and 7.hh to latent vectors, then concatenate those latent vectors and passes the concatenated vector through an MLP with 2 hidden layers of sizes 256 and 128 and an output layer of size 1.
B.3 Action Planning
To do action planning, learn inverse dynamics model to Pee (si,, Fi.+1) predicts 7- dimensional robot states 8i, = Pw (T),e) and 81,+1 = Pa (T The first 6 dimensions of the robot state represent joint angles and the last dimension of the robot state represents the gripper state (i.e., whether it's open or closed). The first 6 action dimension is represented as joint angle difference ai,el: 6] 51,441 |: 6]
si,el= 6] while the last action dimension is gripper state of next timestep
ar -- = 1
Architecture We use VIT-B [9] (vC-1 [32] initialization) along with a linear layer to parameterize
Pg. VTT-B projects the observation T2 E I 48x64x3 to 768 dimensional latent vector from which the
linear layer predicts the dimensional state 81,0-
C
Training and Evaluation
C.1 Task Planning
Inability to recognize uncommon objects
Hellucination leading to spatial ressoning
conrudttio hee?
jutwee
Yos, wilour In brown hox
Ineerster drim to
the
language
Figure 8: Failure VLM. recognize uncommon objects like computer hard drives and occasional hallucina- tion of object presence, leading to incorrect visual reasoning.
Training Objective and Dataset for Learned Classifier We use a softmax cross-entropy loss to train the multi-class classifier fo(#,,1 (w, ):)1: to classify observation 24 into one of the M given subgoal. We train it using the classification dataset Detussity = {a.,1. g, {wr J ,
consisting of observation 24,1, goal g. candidate subgoals {wr}MM M and the correct subgoal label and kitchen-tasti consists of
i. The classification dataset for paint-block, 58k, 82k and 50k datapoints respectively.
Vision-Language Model (VLM) as a Classifier We use frozen pretrained Vision-Languaae Model (VLM) (MiniGPT4 [58]) classifier. We first sample list of all possible subgoals
17

---

W = {wr M, from the LLM given the language goal g. We then use the VLM to eliminate subgoals from W that have been completed. For each subgoal, we question the VLM whether that subgoal has been completed. For example, consider the subgoal "Place white block in yellow bowl". To see if the subgoal has been completed, we ask the VIM "Is there block yellow bowl?". Consider the subgoal "Place green block in brown box" as another example. To see if the subgoal has been completed, we ask the VLM "Is there a green block in brown box?". Furthermore, if the VLM says "yes" and the subgoal has been completed, we also remove other subgoals from W that should have been completed, such as "Place white block in green bowl". Once we have eliminated the completed subgoals, we use the domain knowledge to determine which subgoal to execute out of all the remaining subgoals. As an example, if the goal is to "Stack green block on top of blue block in brown box" and we have a green block in green bowl and blue block in blue bowl, we should execute the subgoal "Place blue block in brown box" before the subgoal "Place green block on blue block". While this process of asking questions from VL.M to determine the remaining subgoals and then sequencing the remaining subgoals doesn' 't require any training data, it heavily relies on the task's domain knowledge.
Failure Modes of VLM We observe two common failure modes of the VLM approach in object-arrange environment and visualize them in Figure 8. First, because the model is not trained on any in-domain data, often fails to recognize uncommon objects, such computer hard drives, in the observations. Second, it occasionally hallucinates the presence of objects at certain locations and thus leads to incorrect visual reasoning.
VLM as a Subgoal Predictor We also tried to prompt the VLM with examples of goal g and subgoal candidates [ and then directly use it to generate the next subgoal u. given the observation Ii,1 and the goal g. However, it completely failed. We hypothesize that the VI.M fails to directly generate the next subgoal due to its inability to perform in-context learning.
Evaluation We evaluate the trained classifier Se and the frozen VLM for subgoal prediction accuracy on 5k unseen datapoints, consisting of observation, goal, candidate subgoals and correct subgoal, generated from test tasks Tea.. We average over 4 seeds and show the results in Figure 7.
C.2 Visual Planning
Ego4D dataset processing We pre-train on canonical clips of the Ego4D dataset which are text- annotated short clips made from longer videos. We further divide each canonical clip into 10sec segments from which we derive 50 frames. We resize each frame to 48 x 64. We create a pretraining Ego4D dataset of (approximately) 344k short clips, each consisting of 50 frames and a text annotation. We use the loader from R3M [34] codebase (https://ithuu..com) htt.://oo//.accoooeeeercccccc pretraining Ego4D dataset.
to load our
Training Objective and Dataset We use pixel-level L1 reconstruction and negative perceptual similarity for training the autoencoder Venc- We borrow this objective from PVDM [53] paper except we don't use adversarial loss. We keep the language encoder frozen. We use denoising loss in video latent space [x (.t,i,,,,Tssse "r. (en.), Ille < k: fene (w), Vene (1), k) [|]] to train the
noise model Eg. We replace w with null token so that es learns both a text-conditional model and
an unconditional model. We pretrain the autoencoder Vene and the noise model es on the processed
Ego4D dataset. We then finetune it on our dataset Dridco == {ri, w;} consisting of approximately 100k observation trajectories of length T 50 and associated text subgoals.
Classifier Training Objective and Dataset We use a binary cross-entropy loss to train the binary classifier ge (T=) that predicts if the observation trajectory in latent space T. = Vene (en)(TT))) leads to high-likelihoe action trajectory. It trained using trajectories from video dataset Ta ~ D Diidco-
C,3 Action Planning
Training Objective and Dataset We train inverse dynamics p& on a dataset Dinv Since actions are differences between robotic joint states, we train Pe to directly predict robotic state $ = Pe (z.,e) by minimizing the mean squared error between the predicted robotic state and ground truth robotic state. Hence, Din = (T T consists of 1K paired observation and robotic state trajectories, each having a length of T = 50, in paint-block and
domains. In setchen-taske domain, it consists of 3.5k paired observation and robotic state trajectories, each having a length of
T = 50.
18

---

Evaluation We evaluate the trained Pe (i.e., VC-1 initialized model and other related models in Figure 6) on 100 unseen paired observation and robotic state trajectories generated from test tasks Teeu We use mean squared error to evaluate our inverse dynamics models. We use 4 seeds to calculate the standard error, represented by the shaded area in Figure
C.4 Hyperparameters
Task Planning We train Sa for 50 epochs using AdamW optimizer [31], a batch size of 256, a learning rate of 1e 3 and weight decay of 1e 6. We used one V100 Nvidia GPU for training the multi-class classifier.
Visual Planning We borrow our hyperparameters for training video diffusion from the PVDM paper [53]. We use Adam W optimizer [31], a batch size of 24 and learning rate of le - 4 for training the autoencoder. We use AdamW optimizer, a batch size of 64, and a learning rate of - for training the noise model. During the pretraining phase with the Ego4D dataset, autoencoder for epochs and then the noise model for 5 epochs. During the finetuning phase with Dridco. we train the autoencoder for 10 epochs and then the noise model for 40 epochs. We used two A6000 Nvidia GPUs for training these diffusion models. We train Ie for 10 epochs using A optimizer, batch size of 256 and a learning rate of 1e - 4. We used one V100 Nvidia GPU for training the binary classifier. During classifier-free guidance, we use s and w' 1.
train the
Action Planning We train VC-1 initialized inverse dynamics model for 20 epochs with Adam optimizer [31], a batch size of 256 and learning rate of 3e - 5. We trained other randomly initialized ViT-B inverse dynamics models and randomly initialized ResNet-18 18 inverse dynamics models for 20 epochs with Adam W optimizer, a batch size of 256, and a learning rate of 1e 4. We used one V100 Nvidia GPU for training these inverse dynamics models.
D Implementation Details for Gato and Say Can
In training the visual and action planning for HiP, we 100k robot videos for visual planner and the inverse dynamics, when trained from scratch, utilizes 10k state-action trajectory pairs. In order to ensure fair comparison,
use 1 110k datapoints for training Gato [39] and SayCan [2].
Gato We borrow the Gato [39] architecture from Vima Codebase and use it for training language conditioned policy with imitation learning. We use 1 110k (langauge, observation trajectory. action trajectory) datapoints in each of the three domains for training Gato. Furthermore, provide oracle subgoals to Gato.
SayCan We borrow the SayCan [2] algorithm from the SayCan codebase and adapt it to our settings. Following the recommendations of SayCan codebase,
CLIPort policies primitives. CLIPort policies take in top-down RGBD view and outputs pick and place pixel coordinates. Then, an underlying motion planner picks the object from the specified pick-coordinate and places the object at the specified place-coordinate. We train CLIPort policies on 1 10k (language, observation, action) datapoints in paint-block and object-arrange domain. The SayCan paper uses value function as an affordance function to select the correct subgoal given current observation and high level goal. However, CLIPort policies don't have value function. The SayCan codebase uses hardcoded scoring function which doesn't It apply to object perenggrranggeeeee domain. To overcome these issues, we use the LLM grounding strategy from ? It uses unnormalized logits over the pixel space given by CLIPort policies as affordance and uses it to ground LLM to current observation and thus predict the subgoal. We then compare SayCan with HiP and other baselines on paint- and object-and arrange domain in Table 1. While SayCan outpeforms other baselines, HiP still outperforms it both on seen and unseen tasks of paint-block and object- couldn't it run SayCan on kitchen-taske domain as there's 's no clear-cut primitive in that domain. This points limitation of SayCan which requires tasks to be expressed in terms of primitives with each primitive paired with an affordance function.
domain. We
E
Additional Ablation Studies
E.
Consistency between task planning and visual planning
To make task planning consistent with visual planning, we need to select subgoal wi which maximizes the joint likelihood (see equation 2) of LLM PLLM(W;) (w:lg) and video diffusion po (T ww, 2T,1). While
19

---

generating videos for different subgoal candidates w; and calculating the likelihood of the generated video is computationally expensive, we would still like to evaluate its performance in subgoal prediction given it is theoretically grounded. To this end, we first sample M subgoals W = {wy j= Md
from the LLM. Then, we calculate wi = arg wew log 22,1 and use W as our predicted subgoal. Since log pT( I tc, x1,1) is intractable, approximation. We use this approach for subgoal prediction in paint-block environment and compare its performance to that of the learned classifier. It achieves a subgoal prediction accuracy of 54.3 + 7.2% whereas the learned classifier achieves a subgoal prediction accuracy of 98.2 + 1.5% in paint-block environment. Both approaches outperform the approach of randomly selecting a subgoal from W (i.e., no task plan refinement), which yields a subgoal prediction accuracy of 16.67% given M 6. The poor performance of the described approach could result from the fact that the diffusion model only coarsely approximates the true distribution p(Tg) W;, T4,1), which results in loose variational oower-bound and thus uncalibrated likelihoods from the diffusion model. A larger diffusion model could better approximate p( Ti. n0,, Ii,1), resulting in tighter variational lower-bound and celter-calibrate likelihoods.
estimate its variational lower-bound
an
E.2 Consistency between visual planning and action planning
To make visual planning consistent with action planning, we need to select observation trajectory (r2)" which maximizes joint likelihood (see equation 4) of conditional video diffusion (T [Wa, 21 and inverse model [ Pe (as,e 2 21,+1). While sampling action trajectories and calculating their likelihoods during every step of the denoising process is computationally inefficient, we would still like to evaluate its effectiveness in visual plan refinements. However, we perform video diffusion in latent space while our inverse model is in observation space. Hence, for purpose of this experiment, we learn another inverse model (T(()) that uses a sequence model (i.e. a transformer) to produce an action trajectory T.. given an observation trajectory in latent space T.. We train P for 20 epochs on 10k paired observation and action trajectories, each having a length of T = 50. We use AdamW optimizer, a batch size of 256 and a learning rate of le 4 during training. To generate an observation trajectory that maximizes the joint likelihood, we first sample 30 observation trajectories from video diffusion Ps (T') [W, 2i,1) conditioned on subgoal w, and observation Ts,1- For each generated observation trajectory Ti we sample a corresponding action trajectory = and calculate its corresponding log- likeliield log (r. (een(TT))) We select the observation trajectory = with highest log-likelihood. Note that we only use Pg for visual plan refinement and use Pv: for action execution to ensure fair comparison. If we use this approach for visual plan refinement with HiP, we obtain a success rate of 72.5 1.9 on unseen tasks in paint paint-block environment. This is comparable to the performance of HiP with visual plan refinements from learned classifier Iv which obtains success rate of 72.8 + 1.7 on unseen tasks in paint- paint-block environment. In contrast, HiP without any visual plan refinement obtains success rate of 71.1 + 1.3 on unseen tasks in paint-block environment. These results show that gg serves as a good approximation for estimating whether an observation trajectory leads to a high-likelihood action trajectory, while still being computationally efficient.
Architecture for Pи We use a transformer model to represent = TM " ".. We first use ResNet-9 encoder to convert T_. T. and r.W to latent vectors. We then concatenate those latent vectors and project the resulting vector to a hidden space of 64 dimension using linear layer. We then pass the 64 dimensional vector to a trajectory transformer model [23] which generates an action trajectory T, of length 50. The trajectory transformer uses transformer architecture with 4 layers and 4 self-attention heads.
E.3 How granularity of subgoals affects performance of Hip ?
We conduct a study in paint-block environment to analyze how granuality of subgoals affect HiP. In our current setup, a subgoal in paint-block domain is of form "Place <block color> block in/on/to <f inal block location>" and involves pick and a place operation. We refer to our current setup as HiP (standard). We introduce two additional level of subgoal granuality:
Only one pick or place operation: The subgoal will be of form "Pick <block color> block in/on <initial block location>" or "Place <block color> block in/on/to <f <final block location>". will involve either one pick or one place operation. We refer to the model trained in this setup HiP (more granular).
20

---

Two pick and place operations: The subgoal will be of form "Place <ist block color> block in/on/to <final 1st block location> and Place <2nd block color> block in/on/to <f fnnal 2nd block location>". It will involve two pick and place operations. We refer to the model trained in this setup as HiP (less granular).
Model
HiP (morc granular)
Hip (Standard)
HiP (less granular)
UniPi
Paint-block (Seen)
74.5 1 1.8
74.3 + 1.9
61.8 + 3.1
37.2 3.8
Paint-block (Unseen)
73.1 2.1
72.8 1 1.7
58.2 3.4
35.3 3.2
Table 2: Granularity Subgoals. Performance of HaP as we vary the granularity of subgoals. Initially, doesn't get affected but then starts to deteoriate when subgoals become too coarse.
Note that UniPi has the least granuality in terms of subgoals it tries to imagine the entire trajectory from goal description. Table in the rebuttal document compares HiP (standard), HiP (more granular). HiP (less granular) and UniPi on seen and unseen tasks in paint-block environment. We observe that HiP (standard) and HiP (more granular) have similar success rates where HiP (less granular) has lower success rate. UniPi has the lowest success rate amongst these variants. We hypothesize that success rate of HiP remains intact when we decrease the subgoal granuality as long as the performance of visual planner doesn't degrade. Hence, HiP (standard) and HiP (more granular) have similar success rates. However, when the performance of visual planner degrades as we further decrease the subgoal granuality, we see a decline in success rate as well. That's why HiP (less granular) sees a decline in success rate and UniPi has the lowest success rate amongst all variants.
E.4
Ablation Visual Planning Model
Paint montt-blen block
Object- Sebect-araage
Kitchen-tad tasks
Model
Seen
Unseen
Seen
Unseen
Seen
Unseen
HiP (RSSM) Hip
69.5 1.6
3.8 75 + 2.8
59.2
3.9
50.6 + 16.2
46.8 + 19.4
74.3 + 1.9
72.8
+ 1.7
75.4 2.6
85.8 + 9.4
83.5 + 10.2
Table 3: Ablating Visual Planner. While performance gap between HiP and HiP (RSSM) is small in Paint-block, widens in more visually complex domains, such Object-arrange and Kitchen-tasks, thereby showing the importance of video diffusion model.
To show the benefits of video diffusion model, we perform an ablation where we use (text-conditioned) recurrent state space model (RSSM), taken from DreamerV3 3 [? ]. as visual model for HiP. We borrow the RSSM code from dreamerv3-torch codebase. To adapt RSSM to our setting, we condition RSSM on subgoal (i.e. subgoal encoded into latent representation by Flan- Plan-T5-Baa T5-Base) instead of actions. Hence, sequence model of RSSM becomes h4 = fhe-11 2 w) where w is latent representation of subgoal. Furthermore, we don't predict any reward since aren't in a reinforcement learning setting and don't predict continue vector since we decode for a fixed number of steps. Hence, we remove reward prediction and continue prediction from the prediction loss. To make the comparisons fair, we pretrain RSSM with Ego4D data as well. We report the results in Table 3. We see that HiP with video diffusion model outperforms HiP with RSSM all the three domains. While the performance gap between HiP(RSSM) and HiP (i.c. using video diffusion) is small in paint- seint-block block environment, widens in sejject-arrange and complex.
domains as the domains become more visually
E.5 Analyzing sensitivity of iterative refinement to hyperparameters
w 0.5
W/ 0.75
w' 1.0
== 1.25
wf 1.5
W = 1.75
// 2.0
Paint-block (Seen)
71.8 2.3
72.3 2.0
73.9 19
7044 7..1
68.2 1.9
Paint-block (Unseen)
71.1 2.5
2.5
1.
7.7
714115 1.
2.7
66.8 1.4
Table 4: Sensitivity of visual iterative refinement to guidance scale. Performance of Hip vary the guidance scale w', HiP performs best when w' E {1, 1.25) but performance degrades for higher values of w'.
The subgoal classifier doesn't introduce any test time hyperparameters and we use standard hyper- parameters (le 3 learning rate, 1e - 6 weight decay, 256 batch size, 50 epochs, Adam optimizer)
21

---

for its training which remains fixed across all domains. We observed that the performance changes minimally across different hyperparameters, given a learning rate decay over training. However, the observation trajectory classifier dæ introduces an additional test time hyperparameter w' which appropriately weights the gradient from observation trajectory classifier. Table 4 in the rebuttal document varies w' between 0.5 and 2 in intervals of 0.25 and shows success rate of HiP. We see that HiP gives the best performance when w' E {1,1.25) but it's performance degrades for higher values of w
Analyzing Runtime of HiP
Subgoal
Subgoal
Visual planning per subyual
Action planning subgoul
Action execution
Episodic
Domain
candidate generation
classifccaion
per subgual
runlime
Paint-block 1.85s
0.4ss
7.32s
6.35s
Ubject-arrange 1.9s
0.43s
7.39s
0.89s
9.57s
78.71
Kitchen-tarie tasks 1.81s
0.41s
7.35%
0.98s
1.28%
40.37
Table 5: Run time of HiP. Average episodic run-time of HiP, along with average run-time of its different com-
ponents for Paint-blocks Object-arrange and Kitchen-taakk domains. While HiP has similar planning times across different domains, has different action execution times and cpisodic runtimes across domains due to differences in simulation properties and average number of subgoals.
provide average runtime of HiP for single episode in all the three domains in Table 5 of the rebuttal document. We average across 1000 seen tasks in each domain. We break the average runtime by different components: task planning (subgoal candidate generation and subgoal classification), visual planning. action planning and action execution. We execute the action plan for subgoal in open-loop and then get observation from the environment for deciding the next subgoal. From Table see that majority of the planning time is taken by visual planning. Recent works [24, 42, 56] have proposed techniques to reduce sampling time diffusion models, which can incorporated into our framework for improving visual planning speed in the future.
22


Reviews:
[{"review_id": "uPCrCvuVO7", "reviewer": "Reviewer_7a1H", "summary": "This paper presents a way to leverage large foundation models to perform long-horizon hierarchical planning tasks. Specifically, the authors leverage an LLM for subgoal generation from the text instruction, a video generation model for generating a plan for each subgoal, and an action prediction model that outputs an action given the current and next observation generated by the video generation model. To make the predictions between different models consistent, the authors propose the iterative refinement procedure to refine the prediction of a model based on joint distributions. The paper includes results in simulated manipulation environments and shows improvement over prior goal-conditioned/action planner/video planner-based methods. The authors also include ablation studies to demonstrate effectiveness of various design choices in the method, such as the use of iterative refinement.", "questions": "Please address the concerns raised in the section above.\n\nUpdate after rebuttal:\n\nI decided to raise my score after reading author’s response.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors presents a nice way to combine three different large foundation models in the problem of hierarchical planning. The iterative refinement method seems very effective in aligning predictions of various models, which is of great significance in research that plays with using multiple foundation models.\n\n2. The authors perform rigorous empirical evaluations via comparing to various baselines and conducting a large number of ablation studies to show the good performance of the proposed method.", "weaknesses": "1. While iterative refinement method is intuitively reasonable and a natural choice for making predictions of models in the pipeline consistent, the authors use approximations of the target joint distribution in the iterative refinement step. It is unclear if the some approximations fully capture the target distribution. For example, I'm not sure if Eq. (3) is a good approximation of (2) since Eq. (3) doesn't have the reachability part and thus doesn't fully capture the feasibility of achieving the language goal.\n\n2. It is unclear if the method can outperform previous hierarhical planning methods such as [1, 2, 3, 4, 5, 6]. Since the authors only evaluate the method in simple simulated pick-and-place tasks, I'm not sure if we need large-scale video diffusion models for video generation and rather simple latent-space world model like [1] is very likely to work.\n\n3. Following the comment in 2, I think the authors should evaluate the method in more domains included in [1,2,3,4,5,6] and also more realistic domains such as real-world robot manipulation settings to fully demonstrate the necessity of using big video foundation models.\n\n[1] Hafner, Danijar, Kuang-Huei Lee, Ian Fischer, and Pieter Abbeel. \"Deep hierarchical planning from pixels.\" Advances in Neural Information Processing Systems 35 (2022): 26091-26104.\n\n[2] Hafner, Danijar, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. \"Mastering diverse domains through world models.\" arXiv preprint arXiv:2301.04104 (2023).\n\n[3] Zhao, Chao, Shuai Yuan, Chunli Jiang, Junhao Cai, Hongyu Yu, Michael Yu Wang, and Qifeng Chen. \"ERRA: An Embodied Representation and Reasoning Architecture for Long-horizon Language-conditioned Manipulation Tasks.\" IEEE Robotics and Automation Letters (2023).\n\n[4] Kujanpää, Kalle, Joni Pajarinen, and Alexander Ilin. \"Hierarchical Imitation Learning with Vector Quantized Models.\" arXiv preprint arXiv:2301.12962 (2023).\n\n[5] Mendonca, Russell, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, and Deepak Pathak. \"Discovering and achieving goals via world models.\" Advances in Neural Information Processing Systems 34 (2021): 24379-24391.\n\n[6] Tian, Stephen, Suraj Nair, Frederik Ebert, Sudeep Dasari, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. \"Model-based visual planning with self-supervised functional distances.\" arXiv preprint arXiv:2012.15373 (2020).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 7a1H for their constructive feedback. We now answer the following concerns raised in the review.\n\n>unclear if the some approximations fully capture the target distribution. For example, I'm not sure if Eq. (3) is a good approximation of (2) since Eq. (3) doesn't have the reachability part and thus doesn't fully capture the feasibility of achieving the language goal.\n\nIn Appendix D.1 of our supplemental material, we compare with optimizing the correct target distribution compared to our approximations. We find that directly optimizing Equation (2) gets an average success rate of $54.3 \\pm 7.2$ compared to optimizing  Equation (3) which gets a performance of $98.2 \\pm 1.5$ on *paint-block* domain. This suggests that our approximations reasonably capture the target distribution.\n\nFurthermore, if we consider the rewrite of Equation (3), \n\n$\\max_{w_i} \\log p_\\text{LLM}(w_i|g) + \\log\\left(\\frac{p(x_{i,1} | w_i, g)}{p(x_{i,1} | g)}\\right)$ \n\nwe see that the first part $\\log p_\\text{LLM}(w_i|g)$ ensures reachability as it makes LLM produce subgoals that make progress towards the high-level goal and the second part $\\log\\left(\\frac{p(x_{i,1} | w_i, g)}{p(x_{i,1} | g)}\\right)$ ensures feasibility as the classifier approximating the log density ratio selects subgoals feasible from $x_{i,1}$.  \n\n\n>It is unclear if the method can outperform previous hierarhical planning methods such as [1, 2, 3, 4, 5, 6]. Since the authors only evaluate the method in simple simulated pick-and-place tasks, I'm not sure if we need large-scale video diffusion models for video generation and rather simple latent-space world model like [1] is very likely to work.\n\nTo show the benefits of video diffusion model, we perform an ablation where we use (text-conditioned) recurrent state space model (RSSM), taken from Dreamer-v3 [2] (and used in other prior works [1,3]), as visual model for HiP. We borrow the RSSM code from [dreamerv3-torch codebase](https://vitalab.github.io/article/2023/01/19/DreamerV3.html). To adapt RSSM to our setting, we condition RSSM on subgoal (i.e. subgoal encoded into a latent representation by Flan-T5-Base) instead of actions. Hence, sequence model of RSSM becomes $h_t = f(h_{t-1}, z_{t-1}, w)$ where w is latent representation of subgoal. Furthermore, we don't predict any reward since we aren't in a reinforcement learning setting and don't predict continue vector since we decode for a fixed number of steps. Hence, we remove reward prediction and continue prediction from the prediction loss. To make the comparisons fair, we pretrain RSSM with Ego4D data as well. We report the results in Table 1 of the rebuttal document. We see that HiP the video diffusion model outperforms HiP with RSSM in all the three domains. While the performance gap between HiP(RSSM) and HiP (i.e. using video diffusion) is small in *paint-block* domain, it widens in *object-arrange* and *kitchen-tasks* domains as the domains become more visually complex. Note that we didn't directly compare to Dreamer-v3 as Dreamer-v3 operates in online RL setting. Since we wanted to show benefits of using video diffusion, we took RSSM from Dreamer-v3 and used it with HiP for a fair comparison. We will add this ablation study on choice of video model in camera ready version of our paper\n\n>Following the comment in 2, I think the authors should evaluate the method in more domains included in [1,2,3,4,5,6] and also more realistic domains such as real-world robot manipulation settings to fully demonstrate the necessity of using big video foundation models.\n\nWe evaluate HiP on a third domain *kitchen-tasks*, inspired by the *kitchen-shift* domain in Xing et al. [4], and is a generalization of *Robokitchen* domain used in Mendoca at al. [3]. Please see our common response for more details on the task setup and results.\n\n\n**References**\\\n[1] Hafner et al. \"Deep hierarchical planning from pixels.\" NeurIPS, 2022.\\\n[2] Hafner et al. \"Mastering diverse domains through world models.\" arXiv, 2023.\\\n[3] Mendonca et al. \"Discovering and achieving goals via world models.\" NeurIPS, 2021.\\\n[4] Xing et al. \"KitchenShift: Evaluating Zero-Shot Generalization of Imitation-Based Policy Learning Under Domain Shifts.\""}, {"review_id": "7F2skYL9r7", "reviewer": "Reviewer_3JTC", "summary": "The paper introduces a framework called Hierarchical Planning with Foundation Models (HiP), which is designed to improve decision-making in new environments with long-term goals. HiP uses hierarchical reasoning to plan subgoals, visually reason about plans, and execute actions through visual-motor control. The framework utilizes different kinds of knowledge to support various levels of decision-making. It employs a large language model for constructing symbolic plans, a large video diffusion model for grounding the plans in the environment, and an inverse dynamics model for inferring actions from generated videos. The models are kept consistent through iterative refinement. The effectiveness of HiP is demonstrated through its application in two long-horizon table-top manipulation tasks.\n\nThe paper's introduction explains that successful execution of tasks in unfamiliar environments requires reasoning at abstract, geometric, and control levels. The authors propose using multiple modalities of Internet-scale data to reason across these levels. They explain the limitations of existing models and propose HiP as a solution, which uses three different large foundation models to construct a physically executable plan for long-horizon tasks. The models are iteratively refined based on feedback from downstream models, ensuring that the final plan satisfies constraints at all levels. The authors also discuss the potential of training foundation models for videos and ego-centric actions, and the limitations of their current computational resources. They conclude by highlighting the promise of their proposed approach in long-horizon decision-making tasks and hope to inspire future work in more complex real-world tasks.", "questions": "See the weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Hierarchical Planning with Foundation Models (HiP) leverages different modalities of knowledge, which can improve decision-making in novel environments with long-horizon goals.\n- The framework uses a large language model, a large video diffusion model, and an inverse dynamics model, allowing it to construct symbolic plans, ground those plans in the environment, and infer actions from generated videos.\n- The iterative refinement process ensures consistency between the models and enables hierarchically consistent plans that are responsive to the goal and executable given the current state and agent.\n- The approach is computationally efficient to train as it doesn't require any large model finetuning.\n- The authors demonstrate promising results on two long-horizon tabletop manipulation environments, illustrating the efficacy and adaptability of the approach.", "weaknesses": "- The paper is technically similar to UniPi. The technical novelty could be potentially incremental - it adds language model to decompose the language goal to language subgoals and then apply video prediction model to it.\n- The work relies on foundation models for video prediction and ego-centric action prediction. Concern: Although it can be expected that video prediction model will be available in the future, but it may not be assumed to be suitable for robotics applications. Additionally, it seems to be overclaim that robotics should rely on these models.\n    1. The applications in robotics may need to model multiple aspects of physical environments, where “expected” video foundation models can struggle to learn\n    How could forces be modeled in video model? How to guarantee the generated video are feasible in physics and can be executed for unseen tasks? The shown pick-and-place is a useful environment, but doesn’t support all the claims. Either the claims need to be revised or more experiments are needed.\n    2. It is also likely that vision (image/video) is only one of several important modalities in real-world robotic decision-making. For example, for some insertion tasks, tactile sensing could be important.\n    3. The computational cost of these models is not known yet at all. Although GPT4 is available for use, it only needs to transfer text over internet. If the needed models were to transfer high-quality images or even videos, how much bandwidth would they need? Is that possible to allow real-time robotic decision-making? It seems unlikely that in the near future we will have pretrained action/video models for all scenarios that can give real-time results over internet. If the goal of this paper is to explore along this direction, I think these aspects should all be considered, instead of assuming all these challenges are nonexistent and just claim the contributions.\n    - Overall, I think the paper is a meaningful initial exploration towards this direction, but the claims of the paper should be clear about this as some assumptions are not necessarily realistic. The title “Hierarchical Planning with Foundation Models” does not necessarily match the contributions and novelties of this paper.\n- L68 — The step 2 “visual planning” is the key step to plan in physical level, but image-level is not necessarily the correct level of abstraction for many cases. When does it apply and when not? This seems to include lots of unnecessary details of the physical world for tasks other than just tabletop manipulation.\n- More general concern: The approach relies on the availability of large pretrained models in different domains. Currently, these models are only readily available in the language domain. Furthermore, as mentioned in the paper, the approach is demonstrated on smaller-scale video and ego-centric action models trained in simulation, which serve as proxies for larger pretrained models. This might limit the generalizability of the results to more complex real-world tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 3JTC for their constructive feedback. We now answer the following concerns raised in the review.\n\n>Novelty\n\nPrior works using foundation models for robotics typically learn a single large model on Internet/robotics data. However, LLMs cannot construct visually grounded plans and UniPi cannot construct long horizon plans. The novelty of approach is to use a hierarchy of different foundation models in combination, to hierarchically reason across different modalities, enabling us to scale to long-horizon plans. This hierarchy further allows us to use passive pre-training on different modalities of data, significantly reducing the need for domain-specific data. We further propose to use iterative refinement to connect models across different hierarchies together and ensure compatability across modalities. We demonstrate our results on three long-horizon robot manipulation domains.\n\n>Use Case of Video Models in Robotics\n\nVideo prediction models trained on Internet data can provide us with a rich source of motion information and physics. Given a text subgoal and current image observation, a synthesized video can tell us the precise hand motions necessary to open a door. Our submission shows that video models pretrained on Internet data (Ego4D) generate better videos (in terms of FVD; see Figure 5 in our submission) and lead to higher success rate (see Figure 2 in the rebuttal document). As a result, while large video prediction models do not eliminate the need for robotics data, we believe they provide a rich prior of semantic and physics information that reduce the amount of in-domain robotics data needed, by teaching us priors on how objects move and their physics. We will clarify this claim and accordingly update our introduction.\n\n> How to model forces and tactile sensing?\n\nWe agree that the current instantiation does not integrate force and tactile sensing and will discuss it in our limitations. However, we note that a video model can be used for both force and tactile sense. For instance, [2] proposed a visual model to predict future visual observations and gelsight observations given current visual and gelight observations and planned actions. Building on this work, we can extend our visual planner to additionally generate future gelsight observations given a subgoal. Then, these predicted gelsight observations can be used as input to our inverse dynamics model. While our submission doesn't focus on touch sensing, we believe future works that build foundation models for it can easily be integrated. We will update our Section 5 to discuss the limitations of our current instantiation of hierarchy and how our framework can be expanded to include additional sensory modality like touch.\n\n> Computational Cost / Bandwith of Video\n\n**Bandwidth Cost of Images/Videos:** In our submission, we generate videos of size 64x48 consisting of 50 frames, which amount to ~15Kb. Even if we generate 200 frames at a resolution of 256x256, they would still be under 1MB. Hence, we do not expect the transfer of these data to become a bottleneck. Also, by generating a video and action plan for each subgoal instead of at each timestep, we further reduce the bandwidth requirement. However, this approach may not be suitable for dynamically changing environments that generate subgoals at a high frequency and will discuss this limitation in Section 5 in camera ready version.  \n\n**Computational Cost of Running Models:** We provide average runtime of HiP for a single episode in all the three domains in Table 2 of the rebuttal document. Please see our common response for more details. We ran all our inference experiments on A6000 which is more powerful than on-board GPUs available on robotics platform. However, we believe that improvements in both hardware and software (e.g. quantization techniques for faster inference) will help realize our models on on-board GPUs.\n\n>  Claim of paper / Contributions\n\nAs discussed in the comments above, we will clarify some assumptions that our approach makes (eg: video models serve as a prior and doesn't neccassarily eliminate the need for in-domain robotics data), and further expand upon the limitations of our work in Section 5 (e.g.: the scalability when needed to generate new subgoals at a high frequency) and tone down our overall claims.\n\nOur work focuses on leveraging hierarchies for long-horizon decision making with each level of hierarchy making use of a foundational model pre-trained on non-robotics data. Hence, we titled our paper as \"Hierarchical Planning with Foundation Models\". However, we are happy to modify as the reviewer sees fit.\n\n>L68 — Images as Abstraction of Physical World\n\nIn general, we agree that image-level may not be the most efficient  abstraction for the physical planning and can capture unnecessary detail. In settings in which there is no ego-motion, which corresponds to many manipulation tasks, then image-level motion corresponds well to physical movement. In other settings with ego-motion, then image-level plans may be unnecessary details from ego-motion, but will still also include the relevant physical motion. However, we note that image-level representation is still sufficient for this setting, and the best text-conditioned video generation models can capture relatively accurate physical motion even with ego-motion[3]. We did a study where we replaced our video diffusion model with a latent RSSM, and modeled physical-level plans in the latent space. As seen in Table 1 of the rebuttal document, video generation outperforms this baseline, with larger performance gap with more visually complex domains\n\n**References**:\n[1] Majumdar et al. \"Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?\" arXiv, 2023.\n[2] Calandra et al. \"More than a feeling: Learning to grasp and regrasp using vision and touch.\", RAL, 2018.\n[3] Ho et al. \"Imagen Video: High Definition Video Generation with Diffusion Models\" arXiv,2022."}, {"review_id": "tCmel3So9E", "reviewer": "Reviewer_z2ah", "summary": "This work presents a new approach of combining foundation models from different modalities to into hierarchical system to solve long-horizon robotics problem. The performance of the system is evaluated on two long-horizon tasks and largely out-performed other baselines. ", "questions": "1. What does dataset stats look like?\n2. Can you provide more description rationalize the use of training data to evaluate HiP (Table 1)?\n3. line 213: What is benefits of pre-training video diffusion model in terms of success rate? The description seems to be only indirectly describing the benefits?\n4. It would be interesting to see some analysis on the effectiveness of subgoal generation in this approach, for example, if the granularity of the subgoal generation various, what effect does it have on the overall approach. (This is interesting to me, without this will not affect my review score.)\n5. How is the computation speed look like for this approach, how fast it is to generate for an episode?", "limitations": "Limitations are addressed in the paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The approach of this work is a effective combination of previously available models. The effectiveness of iterative refinement is particularly interesting (which seems to be one of the main performance booster in this approach). The framework details are well supported by the writing and supplementary materials. ", "weaknesses": "I don't find weakness of this work, but I do have some questions in terms of the dataset, experiment, and analysis, see questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer z2ah for evaluating our work. We now answer the following concerns raised in the review.\n\n>What does dataset stats look like?\n\nWe described our evaluation environments in Section 3.1 of our original submission and provided further details about our datasets for task planning, visual planning and action planning in Appendix C of our original submission. We now summarize these dataset statistics here:\n1. Task planning:\n\t (a) LLM prompting: For all the three domains, we prompt the LLM with 5 examples of (high level goals, set of desirable subgoals) pairs sampled from tasks in given domain.\n\t (b) Training multi-class classifier $f_\\phi$ for subgoal classification: We used classification dataset $D_{classify}$ , consisting of observation $x_{i,1}$, goal g, candidate subgoals $(w_j)\\_{j=1}^M$ and the correct subgoal label i, for training multi-class classifier $f_\\phi$. The classification dataset for *paint-block*, *object-arrange*, and *kitchen-tasks* consists of 58k, 82k and 50k datapoints respectively.\n\n2. Visual planning: After pretraining our visual model on Ego4D dataset consisting of 344k datapoints, we finetune our visual model on dataset $D_{video} \\coloneqq (\\tau_x^i, w_i)$ consisting of (observation trajectory, subgoal) pairs. The video dataset consisted of 100k datapoints for all the domains.\n\n3. Action planning: We train VC-1 [1] initialized inverse dynamics model on dataset $D_{\\text{inv}} \\coloneqq (\\tau_x^i, \\tau_s^i)$ consisting of paired observation and robot state trajectories. The inv dataset consisted of 1k datapoints on *paint-block* and *object-arrange* domains and 3.5k datapoints on *kitchen-tasks* domain.\n\n>Can you provide more description rationalize the use of training data to evaluate HiP (Table 1)?\n\nWe are not sure if we understand this question completely, but how we understood it is that you are asking about the evaluation metrics used in Table 1. We measure the task completion rate (i.e. whether the final goal has been achieved) for evaluating HiP and other baselines on *paint-block* and *object-arrange* domains. To be consistent with evaluation metrics used in [1,2], we use subtask completion rate (i.e. percentage of subtasks completed) for evaluating HiP and other baselines on *kitchen-tasks* domain. To evaluate a model on Seen and Unseen tasks, we sample 1000 tasks from $T_{train}$ and $T_{test}$ respectively, and obtain average task completion rate on *paint-block* and *object-arrange* domains and average subtask completion rate on *kitchen-tasks* domain. We repeat this procedure over 4 different seeds and report the mean and the standard error over those seeds in Table 1. We will update our draft to better describe our evaluation metrics in camera ready version of the paper. We apologize if we misinterpreted your question and we would be happy to offer further clarifications.\n\n>What is benefits of pre-training video diffusion model in terms of success rate? The description seems to be only indirectly describing the benefits?\n\nFigure 2 in the rebuttal document provides success rate of HiP when video diffusion model is trained with 100% training data, 75% training data and 50% training data, with or without Ego4D [2] pretraining. We see that pretraining with Ego4D consistently yields better success rate even with reduced training dataset sizes.\n\n>It would be interesting to see some analysis on the effectiveness of subgoal generation in this approach, for example, if the granularity of the subgoal generation various, what effect does it have on the overall approach.\n\nWe appreciate that the reviewer suggested this interesting axis to explore. So, we conduct a study in *paint-block* domain to analyze how granuality of subgoals affect HiP. In our current setup, a subgoal in *paint-block* domain is of form \"Place <block color> block in/on/to <final block location>\" and involves a pick and a place operation. We refer to our current setup as HiP (standard). We introduce two additional level of subgoal granuality:\n1. Only one pick or place operation: The subgoal will be of form \"Pick <block color> block in/on <initial block location>\" or \"Place <block color> block in/on/to <final block location>\". It will involve either one pick or one place operation. We refer to the model trained in this setup as HiP (more granular).\n2. Two pick and place operations: The subgoal will be of form \"Place <1st block color> block in/on/to <final 1st block location> and Place <2nd block color> block in/on/to <final 2nd block location>\". It will involve two pick and place operations. We refer to the model trained in this setup as HiP (less granular).\n\nNote that UniPi has the least granuality in terms of subgoals as it tries to imagine the entire trajectory from goal description. Table 3 in the rebuttal document compares HiP (standard), HiP (more granular), HiP (less granular) and UniPi on seen and unseen tasks in *paint-block* domain. We observe that HiP (standard) and HiP (more granular) have similar success rates where HiP (less granular) has a lower success rate. UniPi has the lowest success rate amongst these variants. We hypothesize that success rate of HiP remains intact when we decrease the subgoal granuality as long as the performance of visual planner doesn't degrade. Hence, HiP (standard) and HiP (more granular) have similar success rates. However, when the performance of visual planner degrades as we further decrease the subgoal granuality, we see a decline in success rate as well. That's why HiP (less granular) sees a decline in success rate and UniPi has the lowest success rate amongst all variants.\n\n**References:**\\\n[1] Majumdar et al. \"Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?\" arXiv:2303.18240, 2023.\\\n[2] Grauman et al. \"Ego4D: Around the World in 3,000 Hours of Egocentric Video.\" CVPR, 2022."}, {"review_id": "1R53BOSdZi", "reviewer": "Reviewer_zvNJ", "summary": "The paper presents Hierarchical Planning with Foundation Models (HiP), a framework that addresses long-horizon decision-making in novel environments, using multiple modalities of data and reasoning at various levels of hierarchy. The key elements of this framework involve the use of a large language model for constructing symbolic plans, a video diffusion model to generate observation trajectory plans, and a large pre-trained robot ego-centric action model to map these plans to a robot's control space. The authors propose an iterative refinement approach for feedback incorporation, promoting a consensus among different models, without the need for large model finetuning. Experimental results on two long-horizon tabletop manipulation tasks were presented, demonstrating promising results for the proposed strategy. The authors also mention the potential for including other modalities, like touch and sound, in the future.", "questions": "See above. Could the authors discuss more about the efficiency and robustness? And also compare with a few foundation robotics model to better situate the work?\n\nThis paper presents a potentially significant step towards more effective long-horizon decision-making. The presented results are promising. I'm happy to raise my scores if the authors address my concerns.", "limitations": "The authors discussed the limited. They didn't comment on the broader societal impacts but I didn't see any concern.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The technical content of the paper appears to be sound, and the proposed framework has shown promising results in the experimental results provided. The methodology to combine language models, video diffusion models, and ego-centric robot control models into one system is comprehensive.\nThe paper is well-structured and clearly explains the approach and the reasoning behind the decisions made. The paper's flow from problem statement to proposed solution, and finally to experimental evaluation, is logical and easy to follow.", "weaknesses": "1. The baselines compared are all relatively weak. It's more appropriate to compare with foundation robotics models, such as SayCan, Gato[1], palm-e[2]. The authors argue that compared to saycan, they can generalise to new skills. However, they didn't demonstrate the generalization to new skills either. (They evaluated on unseen tasks but not new skills.)\n2. While the iterative refinement approach is interesting, it could be further scrutinized to understand its limitations better, particularly concerning computational efficiency and robustness.\n3. As this is mostly an experimental, please provide code to ensure reproducibility. \n4. Formatting issue in line 9, 45.\n\n[1] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez,\nYury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian\nChen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. In Transactions on Machine Learning\nResearch (TMLR), November 10, 2022.\n[2] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson,\nQuan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke,\nKarol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal\nlanguage model. In arXiv preprint arXiv:2303.03378, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer zvNJ for their constructive feedback. We now answer the following concerns raised in the review.\n\n>The baselines compared are all relatively weak. It's more appropriate to compare with foundation robotics models, such as SayCan, Gato[1], palm-e[2]. The authors argue that compared to saycan, they can generalise to new skills. However, they didn't demonstrate the generalization to new skills either. (They evaluated on unseen tasks but not new skills.)\n\nWe add comparisons to 2 existing foundation models for decision making: Gato [1] and Say-Can [2]. In training the visual and action planning for HiP, we use 100k robot videos for visual planner and the inverse dynamics, when trained from scratch, utilizes 10k state-action trajectory pairs. In order to ensure fair comparison, we use 110k datapoints for training Gato and Say-Can.\n\n*Gato:* We borrow the Gato architecture from [Vima Codebase](https://github.com/vimalabs/VIMA/tree/main) and use it for training a language conditioned policy with imitation learning. We use 110k (langauge, observation trajectory, action trajectory) datapoints in each of the three domains for training Gato. Furthermore, we provide oracle subgoals to Gato. We compare Gato with HiP and other baselines on all three domains in Table 1 of the rebuttal document. We see that HiP outperforms Gato on both seen and unseen tasks in all of the three domains.\n\n*SayCan:* We borrow the SayCan algorithm from the [SayCan codebase](https://github.com/google-research/google-research/blob/master/saycan/SayCan-Robot-Pick-Place.ipynb) and adapt it to our settings. Following the recommendations of SayCan codebase, we use CLIPort policies as primitives. CLIPort policies take in top-down RGBD view and outputs pick and place pixel coordinates. Then, an underlying motion planner picks the object from the specified pick-coordinate and places the object at the specified place-coordinate. We train CLIPort policies on 110k (language, observation, action) datapoints in *paint-block* and *object-arrange* domain. The SayCan paper uses value function as an affordance function to select the correct subgoal given current observation and high level goal. However, CLIPort policies don't have a value function. The SayCan codebase uses a hardcoded scoring function which doesn't apply to *object-arrange* domain. To overcome these issues, we use the LLM grounding strategy from Huang et al. [3]. It uses unnormalized logits over the pixel space given by CLIPort policies as affordance and uses it to ground LLM to current observation and thus predict the subgoal. We then compare SayCan with HiP and other baselines on *paint-block* and *object-arrange* domain in Table 1 of the rebuttal document. While SayCan outpeforms other baselines, HiP still outperforms it both on seen and unseen tasks of *paint-block* and *object-arrange* domain. We couldn't run SayCan on *kitchen-tasks* domain as there's no clear-cut primitive in that domain. This points to a limitation of SayCan which requires tasks to be expressed in terms of primitives with each primitive paired with an affordance function. \n\nPaLM-E[4] is based on finetuning PaLM models and associated vision encoders on robotics data. However, PaLM models aren't available and finetuning a large LLM (540B parameters) along with a vision encoder (22B parameters) requires a large number of GPUs that aren't available in an academic lab. \n\n>As this is mostly an experimental, please provide code to ensure reproducibility.\n\nWe have uploaded anonymized preliminary version of code. We also provided architectural details in Appendix B of our submission and further details on training and evaluation procedure in Appendix C of our submission. We will work on cleaning up the code over coming weeks and provide a clean version of the code with detailed instructions with the camera ready version of the paper.\n\n>While the iterative refinement approach is interesting, it could be further scrutinized to understand its limitations better, particularly concerning computational efficiency and robustness.\n\n**Computational efficiency of iterative refinement:** We provide average runtime of HiP for a single episode in all the three domains in Table 2 of the rebuttal document. As we can see, visual planning takes majority of the runtime. The prediction time by subgoal classifier $f_\\phi$ is minimal and gradients from observation trajectory classifier $g_\\psi$ adds minimal runtime overhead. However, the subgoal classifier and observation trajectory classifier adds memory overhead.\n\n**Robustness of iterative refinement:** By robustness, we assume you are referring to sensitivity analysis with respect to hyperparameters. The subgoal classifier doesn't introduce any test time hyperparameters and we use standard hyperparameters (1e-3 learning rate, 1e-6 weight decay, 256 batch size, 50 epochs, Adam optimizer) for its training which remains fixed across all domains. We observed that the performance changes minimally across different hyperparameters, given a learning rate decay over training. However, the observation trajectory classifier $g_\\psi$ introduces an additional test time hyperparameter $\\omega'$ which appropriately weights the gradient from observation trajectory classifier. Table 4 in the rebuttal document varies $\\omega'$ between 0.5 and 2 in intervals of 0.25 and shows success rate of HiP. We see that HiP gives the best performance when $\\omega' \\in \\{1, 1.25\\}$ but it's performance degrades for higher values of $\\omega'$. We will include this analysis in camera ready version of our paper.\n\n\n**References:**\\\n[1] Reed et al. \"A Generalist Agent.\" TMLR, 2022.\\\n[2] Ahn et al. \"Do As I Can, Not As I Say: Grounding Language in Robotic Affordances.\" CoRL, 2022.\\\n[3] Huang et al. \"Grounded Decoding: Guiding Text Generation with Grounded Models for Robot Control.\" arXiv:2303.00855, 2023.\\\n[4] Driess et al. \"PaLM-E: An Embodied Multimodal Language Model.\" ICML, 2023."}]

Author rebuttal:
We thank the reviewers for their thoughtful suggestions. We want to start by addressing the common concerns brought up by the reviewers and dive into the remaining points in the individual responses.

### **Evaluation on more realistic robotic domain**: 

We evaluate HiP on an additional third domain kitchen-tasks, inspired by the kitchen-shift domain in Xing et al. [1], and is a generalization of Robokitchen domain used in Mendoca et al. [2]. Description of kitchen-tasks domain: A robot has to complete kitchen subtasks given in language goal instructions, such as open microwave and light the kitchen area. However, the environment may have objects irrelevant to the subtasks that the robot must ignore. Furthermore, some kitchen subtasks may already be completed, and the robot needs to ignore those tasks when completing the goal. There are 7 possible kitchen subtasks: opening the microwave, moving the kettle, switching on lights, turning on the bottom knob, turning on the top knob, opening the left drawer, and opening the right drawer. A new task T is generated by constructing a random sequence of 4 out of 7 possible kitchen subtasks. In generating the random sequence, we randomly select an instance of microwave out of 3 possible instances, an instance of kettle out of 4 possible instances, a texture of counter, floor, and drawer independently out of 3 possible textures and randomizing the initial poses of the kettle and microwave. With 50% probability, one of 4 selected kitchen subtask is completed before the start of the task. Hence, tasks usually have 3~4 subtasks (i.e. subgoals).

**Results**: We evaluate HiP and new baselines on kitchen-tasks in Table 1 of the rebuttal document. We use subtask completion rate as our evaluation metric for kitchen-tasks domain to be consistent with evaluation metric used in [1,2]. We see that HiP outperforms all other baselines significantly on both seen and unseen tasks in kitchen-tasks domain.

### **New Baselines**: 

We add comparisons to 2 existing foundation models for decision making: Gato and Say-Can. We compare these against these new baselines on our experiment domains in Table 1 of the rebuttal document. We see that HiP outperforms Gato and Say-Can on both seen and unseen tasks. 

Say-Can specifically has an inherent limitation in that it requires tasks to be expressed in terms of primitives, each of which is paired with an affordance function. However, CLIPort policies don't have a value function. To overcome this issue in our experiment domains, we use the LLM grounding strategy from Huang et al. [9].

### **Ablation on Video Model**: 

To show the benefits of video diffusion model, we perform an ablation where we use (text-conditioned) recurrent state space model (RSSM) ,taken from Dreamer-v3 [10], as visual model for HiP. We report the results in Table 1 of the rebuttal document. We see that HiP the video diffusion model outperforms HiP with RSSM in all the three domains. It is evident that the performance gap increases in *object-arrange* and *kitchen-tasks* domains, where the domains are more visually complex.

### **Runtime characteristics of HiP**:

 We provide average runtime of HiP for a single episode in all the three domains in Table 2 of the rebuttal document. We average across 1000 seen tasks in each domain. We break the average runtime by different components: task planning (subgoal candidate generation and subgoal classification), visual planning, action planning and action execution. We execute the action plan for a subgoal in open-loop and then get observation from the environment for deciding the next subgoal. From Table 2, we see that majority of the planning time is taken by visual planning.

### **Technical contributions**: 

Our submission makes following technical contributions: 

(i) Proposing a hierarchical task planning framework for long-horizon decision-making, divided into high-level task planning in text, visual planning with videos, and action planning with joint state trajectories 

(ii) Designing an iterative refinement strategy to ensure consistency amongst the different levels of planning

(iii) Demonstrating that pre-training each level of hierarchy on different modalities of easily accessible Internet data with small amounts of domain-specific data can be effective. 

In summary, we propose a general robot planning strategy that leverages a hierarchy of foundation models, which can learned separately on different modalities of Internet and robotics data, to construct long-horizon plans. This is different from existing works leveraging foundation models, with typically use a single model (typically a LLM or video model) to make decisions, and allows our approach to reason hierarchically across different modalities, enabling us to scale better to long-horizon plans. We illustrate the efficacy of our results on three long-horizon robot manipulation domains.

**References**:

[1] Xing et al. "KitchenShift: Evaluating Zero-Shot Generalization of Imitation-Based Policy Learning Under Domain Shifts."

[2] Mendonca et al. "Discovering and achieving goals via world models." NeurIPS, 2021.

[3] Salimans et al. "Progressive Distillation for Fast Sampling of Diffusion Models.", ICLR 2022.

[4] Zheng et al. "Fast Sampling of Diffusion Models via Operator Learning", arXiv:2211.13449, 2023.

[5] Janner et al. "Planning with Diffusion for Flexible Behavior Synthesis." ICML 2022.

[6] Ho et al. "Imagen Video: High Definition Video Generation with Diffusion Models" arXiv,2022.

[7] Yu et al."Video Probabilistic Diffusion Models in Projected Latent Space." arXiv, 2023.

[8] Singer et al. "Make-A-Video: Text-to-Video Generation without Text-Video Data." arXiv,2022.

[9] H et al. "Grounded Decoding: Guiding Text Generation with Grounded Models for Robot Control." arXiv:2303.00855, 2023.

[10] Hafner et al. "Mastering diverse domains through world models." arXiv, 2023.

Meta-review: The paper presents HiP, which uses several foundation models to produce robot policies. It uses an LLM to plan, a video model to create dynamically feasible trajectory, and an inverse dynamics model to output actions. HiP also enforces consistency via iterative refinement. The results are shown on robotic manipulation tasks to outperform baselines.

Generally, through the rebuttal process the authors have improved the paper, particularly through additional baselines and experiments, which the authors thoroughly implemented. As such, the reviewers have improved their score towards accepting the paper. The negative review is from 3JTC, who hasn’t responded. Overall, the paper presents a scalable, timely, and interesting approach and thoroughly ablates and compares HiP. The experiments are compelling, with the exception of no real-world experiments. 

Based on the improvements and overall result, I recommend accepting the paper.

I do agree with reviewer 3JTC’s remaining point of the title does not clearly show the contribution and novelty of the paper. The authors should change it for the camera ready, perhaps to indicate the use of the specific models (language model for tasks, video model for dynamics, and inverse dynamics model for actions) and to indicate the use-case presented: robotic manipulation or visual-motor control. Without specifying a specific title, the authors should consider these guidelines.
Decision: Accept (poster)