Title: Compositional Foundation Models for Hierarchical Planning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Foundation Models

Composition

Hierarchical Planning
Abstract: To make effective decisions in novel environments with long-horizon goals, it is crucial to engage in hierarchical reasoning across spatial and temporal scales. This entails planning abstract subgoal sequences, visually reasoning about the underlying plans, and executing actions in accordance with the devised plan through visual-motor control. We propose Compositional Foundation Models for Hierarchical Planning (HiP), a foundation model which leverages multiple expert foundation model trained on language, vision and action data individually jointly together to solve long-horizon tasks. We use a large language model to construct symbolic plans that are grounded in the environment through a large video diffusion model. Generated video plans are then grounded to visual-motor control, through an inverse dynamics model that infers actions from generated videos. To enable effective reasoning within this hierarchy, we enforce consistency between the models via iterative refinement. We illustrate the efficacy and adaptability of our approach in three different long-horizon table-top manipulation tasks.

Compositional Foundation Models for Hierarchical Planning

Abstract

To make effective decisions in novel environments with long-horizon goals, it is crucial to engage in hierarchical reasoning across spatial and temporal scales. This entails planning abstract subgoal sequences, visually reasoning about the underlying plans, and executing actions in accordance with the devised plan through visual-motor control. We propose Compositional Foundation Models for Hierarchical Planning ( HiP ), a foundation model which leverages multiple expert foundation model trained on language, vision and action data individually jointly together to solve long-horizon tasks. We use a large language model to construct symbolic plans that are grounded in the environment through a large video diffusion model. Generated video plans are then grounded to visual-motor control, through an inverse dynamics model that infers actions from generated videos. To enable effective reasoning within this hierarchy, we enforce consistency between the models via iterative refinement . We illustrate the efficacy and adaptability of our approach in three different long-horizon table-top manipulation tasks.

1 Introduction

Consider the task of making a cup of tea in an unfamiliar house. To successfully execute this task, an effective approach is to reason hierarchically at multiple levels: an abstract level, e.g. the high level steps needed to heat up the tea, a concrete geometric level e.g. , how we should physically navigate to and in the kitchen, and at a control level e.g. how we should actuate our joints to lift a cup. It is further important that reasoning at each level is self-consistent with each other - an abstract plan to look in cabinets for tea kettles must also be physically plausible at the geometric level and executable given the actuations we are capable of. In this paper, we explore how we can create agents capable of solving novel long-horizon tasks which require hierarchical reasoning.

Large 'foundation models" have become a dominant paradigm in solving tasks in natural language processing [36, 47, 7], computer vision [26], and mathematical reasoning [27]. In line with this paradigm, a question of broad interest is to develop a 'foundation model' that can solve novel and long-horizon decision-making tasks. Some prior works [39, 6] collected paired visual, language and action data and trained a monolithic neural network for solving long-horizon tasks. However, collecting paired visual, language and action data is expensive and hard to scale up. Another line of prior works [10, 28] finetune large language models (LLM) on both visual and language inputs using task-specific robot demonstrations. This is problematic because, unlike the abundance of text on the Internet, paired vision and language robotics demonstrations are not readily available and are expensive to collect. Furthermore, finetuning high-performing language models, such as GPT3.5/4 [37, 36] and PaLM [7], is currently impossible, as the model weights are not open-sourced.

, §

Internet

Text

Task

Model

Egocentric

Images

Action

Model

Internet

Video

Visual

Model

The key characteristic of the foundation model is that solving a new task or adapting to a new environment is possible with much less data compared to training from scratch for that task or domain. Instead of building a foundation model for long-term planning by collecting paired languagevision-action data, in this work we seek a scalable alternative - can we reduce the need for a costly and tedious process of collecting paired data across three modalities and yet be relatively efficient at solving new planning tasks? We propose Compositional Foundation Models for Hierarchical Planning ( HiP ), a foundation model that is a composition of different expert models trained on language, vision, and action data individually . Because these models are trained individually, the data requirements for constructing the foundation models are substantially reduced (Figure 1). Given an abstract language instruction describing the desired task, HiP uses a large language model to find a sequence of sub-tasks (i.e., planning). HiP then uses a large video diffusion model to capture geometric and physical information about the world and generates a more detailed plan in form of an observation-only trajectory. Finally, HiP uses a large pre-trained inverse model that maps a sequence of ego-centric images into actions. The compositional design choice for decision-making allows separate models to reason at different levels of the hierarchy, and jointly make expert decisions without the need for ever collecting expensive paired decision-making data across modalities.

Given three models trained independently, they can produce inconsistent outputs that can lead to overall planning failure. For instance, a naïve approach for composing models is to take the maximumlikelihood output at each stage. However, a step of plan which is high likelihood under one model, i.e. looking for a tea kettle in a cabinet may have zero likelihood under a seperate model, i.e. if there is no cabinet in the house. It is instead important to sample a plan that jointly maximizes likelihood across every expert model. To create consistent plans across our disparate models, we propose an iterative refinement mechanism to ensure consistency using feedback from the downstream models [28]. At each step of the language model's generative process, intermediate feedback from a likelihood estimator conditioned on an image of the current state is incorporated into the output distribution. Similarly, at each step of the video model generation, intermediate feedback from the action model refines video generation. This iterative refinement procedure promotes consensus among the different models and thereby enables hierarchically consistent plans that are both responsive to the goal and executable given the current state and agent. Our proposed iterative refinement approach is computationally efficient to train, as it does not require any large model finetuning. Furthermore, we do not require access to the model's weights and our approach works with any models that offer only input and output API access.

In summary, we propose a compositional foundation model for hierarchical planning that leverages a composition of foundation models, learned separately on different modalities of Internet and egocentric robotics data, to construct long-horizon plans. We demonstrate promising results on three long-horizon tabletop manipulation environments.

2 Compostional Foundation Models for Hierarchical Planning

We propose HiP , a foundation model that decomposes the problem of generating action trajectories for long-horizon tasks specified by a language goal g into three levels of hierarchy: (1) Task planning - inferring a language subgoal w i conditioned on observation x i, 1 and language goal g ; (2) Visual

<latexi sh

1_b

64="uSW0H7

JU

zr vRTN

f2j

k

>A

B

c

V

8

E

3

q/

9L

m

M

F

poQ

+

g

D

G

Z

Y

w

y

I

d

X

5

C

K

P

n

O

<latexi sh

1_b

64="5jC3rV+U

R

Dy

9Q8

gcZ2oO

w

>A

B

n

NS

EJ

q/

L

0

GP

K

T

m7

W

vH

u

M

f

F

d

k

X

z

I

p

Y

<latexi sh

1_b

64="Fyn

c

+

WGXBf9

5z

g

d/q

8

>A

7

V

NS

EJ3

O

I

M

0o

2

pZ

R

PCj

Y

u

r

v

w

U

H

L

T

D

K

<latexi sh

1_b

64="oyI3KrfM

u

zQJ0

wSNX

UYkW

>A

B9H

c

VDL

g

EO

CGF

v

T

Z

pj

n

+

P

/

d

R

7

8q

m

2

5

<latexi sh

1_b

64="G

NrL

+Fc md

Ayoq

HY

9

XI

>

B

VD

TgJ

EOz

j

R

U

k

C

7

M

Z

/

P

W

v

3

S

p

n

u

f

w

5

8

Q

K

2

0

<latexi sh

1_b

64="mFLyp

On5A

8c

C

+GUK

/

>

B9H

VD

SgN

E

r

f

Y

oM

3

P

I

TZ

Q

v

w

j

0

R7

q

u

J

2

W

d

k

X

z

<latexi sh

1_b

64="M

TS

+

8

Uv

DJ

APrp

2j

B

>

n

c

V

N

E

3

q/

9L

0

G

R

K

g

wm7

Q

H

y5

u

f

F

dW

o

Z

k

C

I

Y

O

z

X

<latexi sh

1_b

64="rEQHB98

ndWLJvo yNfj

wI

M0A

>

c

VD

Oz

G

Y

KU

m

k

5

CZ

/

X

2

P

FR

3

7

p

q

u

T

+

<latexi sh

1_b

64="rEQHB98

ndWLJvo yNfj

wI

M0A

>

c

VD

Oz

G

Y

KU

m

k

5

CZ

/

X

2

P

FR

3

7

p

q

u

T

+

<latexi sh

1_b

64="q

o

YNDS

d

czrV7ML

j

W

J

I

>A

B

n

8

E

3U

/

9

0mk

F

Q

y2

pZ

PC

X

H

+

G

O

T

R

g

f

5

K

u

w

v

<latexi sh

1_b

64="M

TS

+

8

Uv

DJ

APrp

2j

B

>

n

c

V

N

E

3

q/

9L

0

G

R

K

g

wm7

Q

H

y5

u

f

F

dW

o

Z

k

C

I

Y

O

z

X

Pack	computer	mouse,

black	and	blue	sneakers,

pepsi	next	box,	toy	train

in	brown	box

LLM

Place	computer

mouse	in	brown

box

Place	dirty	object

in	blue	box

…

Diffusion

Place	computer

mouse	in	brown

box

Inverse

Dynamics

w

k

g

w

k

+1

w

x

t

x

t

Command

a

t

ˆ

x

t

+1

Task	Planning

Visual	Planning

Action	Planning

Iterative

Refinement

…

ˆ

x

t

+1

ˆ

x

t

+2

ˆ

x

t

+3

Iterative

Refinement

Environment

planning - generating a physically plausible plan as a sequence of image trajectories τ i x = { x i, 1: T } , one for each given language subgoal w i and observation at first timestep x i, 1 ; (3) Action planning inferring a sequence of action trajectories τ i a = { a i, 1: T -1 } from the image trajectories τ i x executing the plan. Figure 2 illustrates the model architecture and a pseudocode is provided in Algorithm 1.

Let p Θ model this hierarchical decision-making process. Given our three levels of hierarchy, p Θ can be factorized into the following: task distribution p θ , visual distribution p ϕ , and action distribution p ψ . The distribution over plans, conditioned on the goal and an image of the initial state, can be written under the Markov assumption as:

We seek to find action trajectories τ i a , image trajectories τ i x and subgoals W = { w i } which maximize the above likelihood. Please see Appendix A for a derivation of this factorization. In the following sub-sections, we describe the form of each of these components, how they are trained, and how they are used to infer a final plan for completing the long-horizon task.

2.1 Task Planning via Large Language Models

Given a task specified in language g and the current observation x i, 1 , we use a pretrained LLM as the task planner, which decomposes the goal into a sequence of subgoals. The LLM aims to infer the next subgoal w i given a goal g and models the distribution p LLM ( w i | g ) . As the language model is trained on a vast amount of data on the Internet, it captures powerful semantic priors on what steps should be taken to accomplish a particular task. To adapt the LLM to obtain a subgoal sequence relevant to our task, we prompt it with some examples of domain specific data consisting of high-level goals paired with desirable subgoal sequences.

However, directly sampling subgoals using a LLM can lead to samples that are inconsistent with the overall joint distribution in Eqn (1), as the subgoal w i not only affects the marginal likelihood of task planning but also the downstream likelihoods of the visual planning model. Consider the example in Figure 2 where the agent is tasked with packing computer mouse, black and blue sneakers, pepsi box and toy train in brown box. Let's say the computer mouse is already in the brown box. While the subgoal of placing computer mouse in brown box has high-likelihood under task model p θ ( w i | g ) , the resulting observation trajectory generated by visual model p ϕ ( τ i x | w i , x i, 1 ) will have a low-likelihood under p ϕ given the subgoal is already completed. Next, we describe how we use iterative refinement to capture this dependency between language decoding and visual planning to sample from Eqn (1).

Consistency with Visual Planning To ensure that we sample subgoal w i that maximizes the joint distribution in Eqn (1), we should sample a subgoal that maximizes the following joint likelihood

i.e. a likelihood that maximizes both conditional subgoal generation likelihood from a LLM and the likelihood of sampled videos τ i x given the language instruction and current image x i, 1 . One way to determine the optimal subgoal w ∗ i is to generate multiple w i from LLM and score them using the likelihood of videos sampled from our video model p ϕ ( τ i x | w i , x i, 1 ) . However, the video generation process is computationally expensive, so we take a different approach.

Algorithm 1 Decision Making with HiP

1: Models: Large language model p LLM, Subgoal classifier f ϕ , Noise model of diffusion ϵ ϕ , Observation trajectory classifier g ψ , Inverse dynamics p ψ

2: Hyperparameters: Guidance scales ω, ω ′ , Denoising diffusion steps K

3: Input: Current observation x t , Language goal g

4:

# Task Planning

5: for i = 1 . . . M do

6: Generate subgoal w i ∼ p LLM ( w i | g )

7: end for

8: Collect candidate subgoals W ←{ w i } i M =1

9: # Iterative Refinement from Visual Planning

10: w ← arg max w f ϕ ( x t , W, g )

11: # Visual Planning

12: Initialize ( τ x ) K ∼ N (0 , I )

13: for k = K... 1 do

14: # Iterative Refinement from Action Planning

15: ˆ ϵ ← ϵ ϕ (( τ x ) k , x t , k ) + ω ( ϵ ϕ (( τ x ) k , x t , w, k ) -ϵ ϕ (( τ x ) k , x t , k )) -ω ′ ∇ ( τ x ) k log g ψ (1 | ( τ x ) k )

16: ( τ x ) k -1 ← Denoise (( τ x ) k , ˆ ϵ )

17: end for

18: # Action Planning

19: Extract ( x t , ˆ x t +1 ) from ( τ x ) 0

20: Execute a t ← p ψ ( a t | x t , ˆ x t +1 )

The likelihood of video generation p ϕ ( τ i x | w i , x i, 1 ) primarily corresponds to the feasibility of a language subgoal w i with respect to the initial image x i, 1 . Thus an approximation of Eqn (2) is to directly optimize the conditional density

We can rewrite Eqn (3) as

We estimate the density ratio p ( x i, 1 | w i ,g ) p ( x i, 1 | g ) with a multi-class classifier f ϕ ( x i, 1 , { w j } i M =1 , g ) that chooses the appropriate subgoal w ∗ i from candidate subgoals { w j } j M =1 generated by the LLM. The classifier implicitly estimates the relative log likelihood estimate of p ( x i, 1 | w i , g ) and use these logits to estimate the log density ratio with respect to each of the M subgoals and find w ∗ i that maximizes the estimate [45]. We use a dataset D classify := { x i, 1 , g, { w j } j M =1 , i } consisting of observation x i, 1 , goal g , candidate subgoals { w j } j M =1 and the correct subgoal label i to train f ϕ . For further architectural details, please refer to Appendix B.1.

2.2 Visual Planning with Video Generation

Upon obtaining a language subgoal w i from task planning, our visual planner generates a plausible observation trajectory τ i x conditioned on current observation x i, 1 and subgoal w i . We use a video diffusion model for visual planning given its success in generating text-conditioned videos [20, 53]. To provide our video diffusion model with a rich prior for physically plausible motions, we pretrain it p ϕ ( τ i x | w i , x i, 1 ) on a large-scale text-to-video dataset Ego4D [13]. We then finetune it on the task-specific video dataset D video := { τ i x , w i } consisting of observation trajectories τ i x satisfying subgoal w i . For further architectural details, please refer to Appendix B.2.

However, analogous to the consistent sampling problem in task planning, directly sampling observation trajectories with video diffusion can lead to samples that are inconsistent with the overall joint distribution in Eqn (1). The observation trajectory τ i x not only affects the marginal likelihood of visual planning, but also the downstream likelihood of the action planning model.

Consistency with Action Planning To ensure observation trajectories τ i x that correctly maximize the joint distribution in Eqn (1), we optimize an observation trajectory that maximizes the following joint likelihood

i.e. an image sequence that maximizes both conditional observation trajectory likelihood from video diffusion and the likelihood of sampled actions τ i a given the observation trajectory τ i x .

To sample such an observation trajectory, we could iteratively bias the denoising of video diffusion using the log-likelihood of the sampled actions ∏ T -1 t =1 log p ψ ( a i,t | x i,t , x i,t +1 ) . While this solution is principled, it is slow as it requires sampling of entire action trajectories and calculating the corresponding likelihoods during every step of the denoising process. Thus, we approximate the sampling and the likelihood calculation of action trajectory ∏ T -1 t =1 p ψ ( a i,t | x i,t , x i,t +1 ) with a binary classifier g ψ ( τ i x ) that models if the observation trajectory τ i x leads to a high-likelihood action trajectory.

Welearn a binary classifier g ψ to assign high likelihood to feasible trajectories sampled from our video dataset τ i x ∼ D video and low likelihood to infeasible trajectories generated by randomly shuffling the order of consecutive frames in feasible trajectories. Once trained, we can use the likelihood log g ψ (1 | τ i x ) to bias the denoising of the video diffusion and maximize the likelihood of the ensuing action trajectory. For further details on binary classifier, please refer to Appendix C.2.

2.3 Action Planning with Inverse Dynamics

After generating an observation trajectory τ i x from visual planning, our action planner generates an action trajectory τ i a from the observation trajectory. We leverage egocentric internet images for providing our action planner with useful visual priors. Our action planner is parameterized as an inverse dynamics model [1, 38] that infers the action a i,t given the observation pair ( x i,t , x i,t +1 ) :

Training To imbue the inverse dynamics p ψ with useful visual priors, we initialize it with VC1 [32] weights, pretrained on ego-centric images and ImageNet. We then finetune it on dataset D inv := { τ i x , τ i a } consisting of paired observation and action trajectories by optimizing:

For further architectural details, please refer to Appendix B.3.

3 Experimental Evaluations

We evaluate the ability of HiP to solve long-horizon planning tasks that are drawn from distributions with substantial variation, including the number and types of objects and their arrangements. We then study the effects of iterative refinement and of pretraining on overall performance of HiP . We also compare against an alternative strategy of visually grounding the LLM without any task-specific data. In addition, we study how granularity of subgoals affects HiP 's performance, ablate over choice of visual planning model and analyze sensitivity of iterative refinement to hyperparameters (Appendix E).

3.1 Evaluation Environments

We evaluate HiP on three environments, paint-block , object-arrange , and kitchen-tasks which are inspired by combinatorial planning tasks in Mao et al. [33], Shridhar et al. [43] and Xing et al. [50] respectively.

paint-block : A robot has to manipulate blocks in the environment to satisfy language goal instructions, such as stack pink block on yellow block and place green block right of them . However, objects of correct colors may not be present in the environment, in which case, the robot needs to first pick up white blocks and put them in the appropriate color bowls to paint them. After that, it should perform appropriate pick-and-place operations to stack a pink block on the yellow block and place the green block right of them. A new task T is generated by randomly selecting 3 final colors (out of 10 possible colors) for the blocks and then sampling a relation (out of 3 possible relations) for each pair of blocks. The precise locations of individual blocks, bowls, and boxes are fully randomized across different tasks. Tasks have 4 ∼ 6 subgoals.

object-arrange : A robot has to place appropriate objects in the brown box to satisfy language goal instructions such as place shoe, tablet, alarm clock, and scissor in brown box . However, the environment may have distractor objects. Furthermore, some objects can be dirty, indicated by a lack of texture and yellow color. For these objects, the robot must first place them in a blue cleaning box and only afterwards place those objects in the brown box. A new task T is generated

'Stack green block on top of yellow block and place orange block to the right of the stack'

place white block

in yellow bowl

place yellow block

in brown box

place white block

in green bowl

place green block

on yellow block

place white block

in orange bowl

place orange block on

right of yellow block

'Pack hammer, frypan, toy school bus, and butterfinger chocolate into the brown box'

place dirty object

in the clean box

place hammer

in the brown box

in the brown box

place toy school bus

in the brown box

place butterfinger

chocolate in brown box

'Open microwave, move kettle out of the way, light the kitchen area, and open upper right drawer'

open the

microwave

move the kettle

to the back stove

turn on the

lights

slide the upper right

drawer

Kitchen Task

by randomly selecting 7 objects (out of 55 possible objects), out of which 3 are distractors, and then randomly making one non-distractor object dirty. The precise locations of individual objects and boxes are fully randomized across different tasks. Tasks usually have 3 ∼ 5 subgoals.

kitchen-tasks : A robot has to complete kitchen subtasks to satisfy language goal instructions such as open microwave, move kettle out of the way, light the kitchen area, and open upper right drawer . However, the environment may have objects irrelevant to the subtasks that the robot must ignore. Furthermore, some kitchen subtasks specified in the language goal might already be completed, and the robot should ignore those tasks. There are 7 possible kitchen subtasks: opening the microwave, moving the kettle, switching on lights, turning on the bottom knob, turning on the top knob, opening the left drawer, and opening the right drawer. A new task T is generated by randomly selecting 4 out of 7 possible kitchen subtasks, randomly selecting an instance of microwave out of 3 possible instances, randomly selecting an instance of kettle out of 4 possible instances, randomly and independently selecting texture of counter, floor and drawer out of 3 possible textures and randomizing initial pose of kettle and microwave. With 50% probability, one of 4 selected kitchen subtask is completed before the start of the task. Hence, tasks usually have 3 ∼ 4 subtasks (i.e. subgoals).

Train and Test Tasks For all environments, we sample two sets of tasks T train , T test ∼ p ( T ) . We use the train set of tasks T train to create datasets D classify , D video , D inv and other datasets required for training baselines. We ensure the test set of tasks T test contains novel combinations of object colors in paint-block , novel combinations of object categories in object-arrange , and novel combinations of kitchen subtasks in kitchen-tasks .

Evaluation Metrics We quantitatively evaluate a model by measuring its task completion rate for paint-block and object-arrange , and subtask completion rate for kitchen tasks . We use the simulator to determine if the goal, corresponding to a task, has been achieved. We evaluate a model on T train (seen) to test its ability to solve long-horizon tasks and on T test (unseen) to test its ability to generalize to long-horizon tasks consisting of novel combinations of object colors in paint-block ,

Table 1: Performance on Long-Horizon tasks. HiP not only outperforms the baselines in solving seen long-horizon tasks but its performance remains intact when solving unseen long-horizon tasks containing novel combination of objects colors in paint-block , novel combination of objects categories in object-rearrange and novel combination of subtasks in kitchen-tasks .

object categories in object-arrange , and kitchen subtasks in kitchen-tasks . We sample 1000 tasks from T train and T test respectively, and obtain average task completion rate on paint-block and object-arrange domains and average subtask completion rate on kitchen tasks domain. We report the mean and the standard error over 4 seeds in Table 1.

3.2 Baselines

There are several existing strategies for constructing robot manipulation policies conditioned on language goals, which we use as baselines in our experiments:

Goal-Conditioned Policy Agoal-conditioned transformer a i,t ∼ p ( a i,t | x i,t , w i ) that outputs action a i,t given a language subgoal w i and current observation x i,t (Transformer BC) [6]. We provide the model with oracle subgoals and encode these subgoals with a pretrained language encoder (Flan-T5-Base). We also compare against goal-conditioned policy with Gato [39] transformer.

Video Planner A video diffusion model (UniPi) [12] { τ i x } ∼ p ( { τ i x }| g, x i, 1 ) , a i,t ∼ p ( a i,t | x i,t , x i,t +1 ) that bypasses task planning, generates video plans for the entire task { τ i x } , and infers actions a i,t using an inverse model.

Action Planners Transformer models (Trajectory Transformer) [23] and diffusion models (Diffuser) [24, 4] { a i,t : T -1 } ∼ p ( { a i,t : T -1 }| x i,t , w i ) that produce an action sequence { a i,t : T -1 } given a language subgoal w i and current visual observation x i,t . We again provide the agents with oracle subgoals and encode these subgoals with a pretrained language encoder (Flan-T5-Base).

LLMas Skill Manager A hierarchical system (SayCan) [2] with LLM as high level policy that sequences skills sampled from a repetoire of skills to accomplish a long-horizon task. We use CLIPort [43] policies as skills and the unnormalized logits over the pixel space it produces as affordances. These affordances grounds the LLM to current observation for producing next subgoal.

3.3 Results

We begin by comparing the performance of HiP and baselines to solve long-horizon tasks in paint-block , object-arrange , kitchen-tasks environments. Table 1 shows that HiP significantly outperforms the baselines, although the baselines have an advantage and have access to oracle subgoals. HiP 's superior performance shows the importance of (i) hierarchy given it outperforms goalconditioned policy (Transformer BC and Gato), (ii) task planning since it outperforms video planners (UniPi), and (iii) visual planning given it outperforms action planners (Trajectory Transformer, Action Diffuser). It also shows the importance of representing skills with video-based planners which can be pre-trained on Internet videos and can be applied to tasks (such as kitchen-tasks ). SayCan, in contrast, requires tasks to be decomposed into primitives paired with an affordance function, which can be difficult to define for many tasks like the kitchen task. Thus, we couldn't run SayCan on kitchen-tasks environment. Finally, to quantitatively show how the errors in f θ ( x i, 1 , w i , g ) affect the performance of HiP , we compare it to HiP with oracle subgoals. For further details on the training and evaluation of HiP , please refer to Appendix C. For implementation details on Gato and SayCan, please refer to Appendix D. For runtime analysis of HiP , please refer to Appendix F.

Combinatorial Generalization to Unseen Long-horizon Tasks We also quantitatively test the ability of HiP to generalize to unseen long-horizon tasks, consisting of novel combinations of object colors in paint-block , object categories in object-arrange , and subtasks in kitchen-tasks . Table 1 shows that HiP 's performance remains intact when solving unseen long-horizon tasks, and

FVD	Score

500

400

400

300

300

200

200

100

100

Success	Rate

FVD	Score

100

80

60

40

20

500

400

400

300

300

200

200

100

Subgoal trajectories

Stack green block on

top of yellow block

and place

orange block to the

right of the stack

place white

block

in yellow bowl

place yellow

block

in brown box

place white

block

in green bowl

place green

block

on yellow block

place white

block

in orange bowl

place orange

block on

right of yellow

block

Initial observation

Goal

Paint	Block

Rearrange	Object

Kitchen	Tasks

still significantly outperforms the baselines. Figure 4 visualizes the execution of HiP in unseen long-horizon tasks in paint-block .

100%	Train	Data

100%	Train	Data

Paint	Block

500

FVD	Score

400

400

300

300

200

200

100

100

100

100%	Train	Data

100%	Train	Data

100%	Train	Data

100%	Train	Data

75%	Train	Data

75%	Train	Data

75%	Train	Data

50%	Train	Data

75%	Train	Data

50%	Train	Data

100%	Train	Data

75%	Train	Data

50%	Train	Data

100%	Train	Data

75%	Train	Data

50%	Train	Data

Kitchen	Tasks

Kitchen	Tasks

50%	Train	Data

75%	Train	Data

Paint	Block

Rearrange	Object

100%	Train	Data

75%	Train	Data

50%	Train	Data

20

40

60

80

Success	Rate

100%	Train	Data

75%	Train	Data

50%	Train	Data

100

100%	Train	Data

75%	Train	Data

50%	Train	Data

Kitchen	Tasks

100%	Train	Data

75%	Train	Data

50%	Train	Data

Paint	Block

Rearrange	Object

100%	Train	Data

75%	Train	Data

50%	Train	Data

100

200

300

400

Ego4D	Pre-training

FVD	Score

100%	Train	Data

75%	Train	Data

50%	Train	Data

100

200

300

400

500

100%	Train	Data

75%	Train	Data

50%	Train	Data

Kitchen	Tasks

100%	Train	Data

75%	Train	Data

50%	Train	Data

No	Pre-training

Ego4D	Pre-training

No	Pre-training

Paint	Block

Rearrange	Object

100%	Train	Data

75%	Train	Data

50%	Train	Data

Ego4D	Pre-training

100%	Train	Data

75%	Train	Data

50%	Train	Data

100%	Train	Data

75%	Train	Data

50%	Train	Data

100%	Train	Data

No	Pre-training

Paint	Block

Rearrange	Object

75%	Train	Data

50%	Train	Data

Ego4D	Pre-training

75%	Train	Data

50%	Train	Data

100%	Train	Data

75%	Train	Data

50%	Train	Data

Kitchen	Tasks

100%	Train	Data

75%	Train	Data

No	Pre-training

Rearrange	Object

50%	Train	Data

Ego4D	Pre-training

50%	Train	Data

100%	Train	Data

75%	Train	Data

50%	Train	Data

Kitchen	Tasks

100%	Train	Data

75%	Train	Data

50%	Train	Data

No	Pre-training

Paint	Block

Rearrange	Object

100%	Train	Data

75%	Train	Data

50%	Train	Data

100

200

300

400

Ego4D	Pre-training

FVD	Score

100%	Train	Data

75%	Train	Data

50%	Train	Data

100

200

300

400

500

100%	Train	Data

75%	Train	Data

50%	Train	Data

Kitchen	Tasks

100%	Train	Data

75%	Train	Data

No	Pre-training

Paint	Block

Rearrange	Object

20

40

60

80

Success	Rate

100

100%	Train	Data

75%	Train	Data

50%	Train	Data

100%	Train	Data

75%	Train	Data

50%	Train	Data

100%	Train	Data

75%	Train	Data

50%	Train	Data

100%	Train	Data

50%	Train	Data

50%	Train	Data

75%	Train	Data

50%	Train	Data

Pre-training Video Diffusion Model We investigate how much our video diffusion model benefits from pre-training on the Internet-scale data. We report both the success rate of HiP and Fréchet Video Distance (FVD) score that quantifies the similarity between generated videos and ground truth videos, where lower scores indicate greater similarity in Figure 5. We see that pretraining video diffusion leads to a higher success rate and lower FVD score. If we reduce the training dataset to 75% and 50% of the original dataset, the FVD score for video diffusion models (both, with and without Ego4D dataset pretraining) increases and their success rate falls. However, the video diffusion model with Ego4D dataset pretraining consistently gets higher success rate and lower FVD scores across different dataset sizes. As we decrease the domain-specific training data, it is evident that the gap in performance between the model with and without the Ego4D pre-training widens. For details on how we process the Ego4D dataset, please refer to Appendix C.2.

Pre-training Inverse Dynamics Model We also analyze the benefit of pre-training our inverse dynamics model and report the mean squared error between the predicted and ground truth actions in Figure 6. The pre-training comes in the form of initializing the inverse dynamics model with











































































































40

20

40

20

60

40

Visually	Grounding	LLM

Paint	Block

Rearrange	Object

20

40

60

80

Prediction	Accuracy

HiP

HiP	without

HiP	without

20

80

Success	Rate

40

60

Paint	Block

Learned	Classifier

HiP

HiP	without

Visual	Refinement

HiP	without

any	Refinement

Paint	Block

Rearrange	Object

20

Learned	Classifier

HiP

HiP	without

Visual	Refinement

HiP	without

any	Refinement

Visual	Refinement any	Refinement

Importance of Task Plan and Visual Plan Refinements We study the importance of refinement in task and visual planning in Figure 7. We compare to HiP without visual plan refinement and HiP without visual and task plan refinement in paint block environment. We see that task plan refinement for visual grounding of LLM is critical to the performance of HiP . Without it, the task plan is agnostic to the robot's observation and predicts subgoals that lead to erroneous visual and action planning. Furthermore, visual plan refinement improves the performance of HiP as well, albeit by a small margin. For a description of the hyperparameters used, please refer to Appendix C.4.

Exploring Alternate Strategies for Visually Grounding LLM We use a learned classifier f θ ( x i, 1 , w i , g ) to visually ground the LLM. We explore if we can use a frozen pretrained VisionLanguage Model (MiniGPT-4 [58]) as a classifier in place of the learned classifier. Although we didn't use any training data, we found the prompt engineering using the domain knowledge of the task to be essential in using the Vision-Language Model (VLM) as a classifier (see Appendix C.1 for details). We use subgoal prediction accuracy to quantify the performance of the learned classifier and the frozen VLM. Figure 7 illustrates that while both our learned multi-class classifier and frozen VLM perform comparably in the paint-block environment, the classifier significantly outperforms the VLM in the more visually complex object-arrange environment. We detail the two common failure modes of the VLM approach in object-arrange environment in Appendix C.1. As VLMs continue to improve, it is possible that their future versions match the performance of weights from VC-1 [32], a vision-transformer (ViT-B) [9] trained on ego-centric images with maskedautoencoding objective [16]. In paint-block and object-arrange , we see that the inverse dynamics, when initialized with weights from VC-1, only requires 1K labeled robotic trajectories to achieve the same performance as the inverse dynamics model trained on 10K labeled robotic trajectories but without VC-1 initialization. We also compare against an inverse dynamics model parameterized with a smaller network (ResNet-18). However, the resulting inverse dynamics model still requires 2.5K robotic trajectories to get close to the performance of the inverse dynamics model with VC-1 initialization in paint-block and object-arrange . In kitchen-tasks , inverse dynamics, when initialized with weights from VC-1, only requires 3.5k labeled robotic trajectories to achieve the same performance as the inverse dynamics model trained on 10K labeled robotic trajectories but without VC-1 initialization. When parameterized with ResNet-18, the inverse dynamics model still requires 6k robotic trajectories to get close to the performance of the inverse dynamics model with VC-1 initialization. Benefits	of	Iterative	Refinement Visually	Grounding	LLM 60 80 Prediction	Accuracy Benefits	of	Iterative	Refinement 80 Success	Rate 60 Visually	Grounding	LLM 80 Prediction	Accuracy Benefits	of	Iterative	Refinement 20 80 Success	Rate 40 60

Rearrange	Object

learned classifiers and thus replace them in visually complex domains as well. For further details on the VLM parameterization, please refer to Appendix C.1.

4 Related Work

The field of foundation models for decision-making [52] has seen significant progress in recent years. A large body of work explored using large language models as zero-shot planners [21, 22, 29, 30, 2], but it is often difficult to directly ground the language model on vision. To address this problem of visually grounding the language model, other works have proposed to directly fine-tune large language models for embodied tasks [28, 10]. However, such an approach requires large paired vision and language datasets that are difficult to acquire. Most similar to our work, SayCan [2] uses an LLM to hierarchically execute different tasks by breaking language goals into a sequence of instructions, which are then inputted to skill-based value functions. While SayCan assumes this fixed set of skill-based value functions, our skills are represented as video-based planners [12], enabling generalization to new skills.

Another set of work has explored how to construct continuous space planners with diffusion models [25, 4, 49, 55, 48, 57, 12]. Existing works typically assume task-specific datasets from which the continuous-space planner is derived [24, 3, 55]. Most similar to our work, UniPi [12] proposes to use videos to plan in image space and similarily relies on internet videos to train image space planners. We build on top of UniPi to construct our foundation model for hierarchical planning, and illustrate how UniPi may be combined with LLMs to construct longer horizon continuous video plans.

Moreover, some works [28, 54, 5] explored how different foundation models may be integrated with each other. In Flamingo [5], models are combined through joint finetuning with paired datasets, which are difficult to collect. In contrast both Zeng et al. [54] and Li et al. [28] combine different models zero-shot using either language or iterative consensus. Our work proposes to combine language, video, and ego-centric action models together by taking the product of their learned distributions [11]. We use a similar iterative consensus procedure as in Li et al. [28] to sample from the entire joint distribution and use this combined distribution to construct a hierarchical planning system.

5 Limitations and Conclusion

Limitations Our approach has several limitations. As high-quality foundation models for visual sequence prediction and robot action generation do not exist yet, our approach relies on smaller-scale models that we directly train. Once high-quality video foundation models are available, we can use them to guide our smaller-scale video models [51] which would reduce the data requirements of our smaller-scale video models. Furthermore, our method uses approximations to sample from the joint distribution between all the model. An interesting avenue for future work is to explore more efficient and accurate methods to ensure consistent samples from the joint distribution.

Conclusion In this paper, we have presented an approach to combine many different foundation models into a consistent hierarchical system for solving long-horizon robotics problems. Currently, large pretrained models are readily available in the language domain only. Ideally, one would train a foundation model for videos and ego-centric actions, which we believe will be available in the near future. However, our paper focuses on leveraging separate foundation models trained on different modalities of internet data, instead of training a single big foundation model for decision making. Hence, for the purposes of this paper, given our computational resource limitations, we demonstrate our general strategy with smaller-scale video and ego-centric action models trained in simulation, which serve as proxies for larger pretrained models. We show the potential of this approach in solving three long-horizon robot manipulation problem domains. Across environments with novel compositions of states and goals, our method significantly outperforms the state-of-the-art approaches towards solving these tasks.

In addition to building larger, more general-purposed visual sequence and robot control models, our work suggests the possibility of further using other pretrained models in other modalities, such as touch and sound, which may be jointly combined and used by our sampling approach. Overall, our work paints a direction towards decision making by leveraging many different powerful pretrained models in combination with a tiny bit of training data. We believe that such a system will be substantially cheaper to train and will ultimately result in more capable and general-purpose decision making systems.

Author Contributions

Anurag Ajay co-conceived the framework of leveraging pretrained foundation models for decision making, implemented visual planning and action planning in HiP, evaluated HiP on long-horizon tasks, performed ablation studies and helped in paper writing.

Seungwook Han co-conceived in conceiving the framework of leveraging pretrained foundation models for decision making, implemented task planning in HiP and helped in paper writing.

Yilun Du co-conceived the framework of leveraging pretrained foundation models for decision making, implemented trajectory transformer and transformer BC and evaluated them on long-horizon tasks, implemented data generation scripts for object arrange and paint block environment, and lead paper writing.

Shuang Li helped in conceiving the idea of iterative refinement for consistency between pretrained foundation models, participated in research discussions and helped in writing paper.

Abhi Gupta participated in research discussions and helped in making figures.

Tommi Jaakkola participated in research discussions.

Joshua Tenenbaum participated in research discussions.

Leslie Kaelbling participated in research discussions, suggested baselines and ablation studies, conceived the structure of the paper and helped in paper writing.

Akash Srivastava participated in research discussions, suggested the idea of using classifier for consistency between observations and the large language model and provided feedback on paper writing.

Pulkit Agrawal was involved in research discussions, suggested ablation studies related to iterative refinement, provided feedback on writing, positioning of the work and overall advising.

References

P. Agrawal, A. V. Nair, P. Abbeel, J. Malik, and S. Levine. Learning to poke by poking: Experiential learning of intuitive physics. Advances in neural information processing systems , 29, 2016.

M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691 , 2022.

A. Ajay, A. Kumar, P. Agrawal, S. Levine, and O. Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010.13611 , 2020.

A. Ajay, Y. Du, A. Gupta, J. Tenenbaum, T. Jaakkola, and P. Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657 , 2022.

J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: A visual language model for few-shot learning. NeurIPS , 2022. URL https://arxiv.org/abs/2204.14198 .

A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817 , 2022.

A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022.

H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 , 2022.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 , 2023.

Y. Du, S. Li, and I. Mordatch. Compositional visual generation with energy based models. In Advances in Neural Information Processing Systems , 2020.

Y. Du, M. Yang, B. Dai, H. Dai, O. Nachum, J. B. Tenenbaum, D. Schuurmans, and P. Abbeel. Learning universal policies via text-guided video generation. arXiv e-prints , pages arXiv-2302, 2023.

K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18995-19012, 2022.

M. U. Gutmann and A. Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. J. Mach. Learn. Res. , 13:307-361, 2012.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770778, 2016.

K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16000-16009, 2022.

J. Hermans, V. Begy, and G. Louppe. Likelihood-free mcmc with amortized approximate ratio estimators. In International conference on machine learning , pages 4239-4248. PMLR, 2020.

J. Ho and T. Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems , 2020.

J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 , 2022.

W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning , pages 9118-9147. PMLR, 2022.

W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608 , 2022.

M. Janner, Q. Li, and S. Levine. Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Information Processing Systems , 2021.

M. Janner, Y. Du, J. Tenenbaum, and S. Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning , 2022.

M. Janner, Y. Du, J. Tenenbaum, and S. Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning , 2022.

A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643 , 2023.

A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206.14858 , 2022.

S. Li, X. Puig, C. Paxton, Y. Du, C. Wang, L. Fan, T. Chen, D.-A. Huang, E. Akyürek, A. Anandkumar, et al. Pre-trained language models for interactive decision-making. Advances in Neural Information Processing Systems , 35:31199-31212, 2022.

J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:2209.07753 , 2022.

K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Text2motion: From natural language instructions to feasible plans. arXiv preprint arXiv:2303.12153 , 2023.

I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

A. Majumdar, K. Yadav, S. Arnaud, Y. J. Ma, C. Chen, S. Silwal, A. Jain, V.-P. Berges, P. Abbeel, J. Malik, et al. Where are we in the search for an artificial visual cortex for embodied intelligence? arXiv preprint arXiv:2303.18240 , 2023.

J. Mao, T. Lozano-Perez, J. B. Tenenbaum, and L. P. Kaelbing. PDSketch: Integrated Domain Programming, Learning, and Planning. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601 , 2022.

A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 , 2021.

OpenAI. Gpt-4 technical report. ArXiv , abs/2303.08774, 2023.

L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022.

D. Pathak, P. Mahmoudieh, G. Luo, P. Agrawal, D. Chen, Y. Shentu, E. Shelhamer, J. Malik, A. A. Efros, and T. Darrell. Zero-shot visual imitation. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops , pages 2050-2053, 2018.

S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175 , 2022.

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. Denton, S. K. S. Ghasemipour, B. K. Ayan, S. S. Mahdavi, R. G. Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487 , 2022.

T. Salimans and J. Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512 , 2022.

M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning , pages 894-906. PMLR, 2022.

J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , 2015.

A. Srivastava, S. Han, K. Xu, B. Rhodes, and M. U. Gutmann. Estimating the density ratio between distributions with high discrepancy using multinomial logistic regression. ArXiv , abs/2305.00869, 2023.

M. Sugiyama, T. Suzuki, and T. Kanamori. Density ratio estimation in machine learning . Cambridge University Press, 2012.

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

J. Urain, N. Funk, G. Chalvatzaki, and J. Peters. Se (3)-diffusionfields: Learning cost functions for joint grasp and motion optimization through diffusion. arXiv preprint arXiv:2209.03855 , 2022.

Z. Wang, J. J. Hunt, and M. Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193 , 2022.

E. Xing, A. Gupta, S. Powers, and V. Dean. Kitchenshift: Evaluating zero-shot generalization of imitation-based policy learning under domain shifts. In NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications , 2021. URL https: //openreview.net/forum?id=DdglKo8hBq0 .

M. Yang, Y. Du, B. Dai, D. Schuurmans, J. B. Tenenbaum, and P. Abbeel. Probabilistic adaptation of text-to-video models. arXiv preprint arXiv:2306.01872 , 2023.

S. Yang, O. Nachum, Y. Du, J. Wei, P. Abbeel, and D. Schuurmans. Foundation models for decision making: Problems, methods, and opportunities. arXiv preprint arXiv:2303.04129 , 2023.

S. Yu, K. Sohn, S. Kim, and J. Shin. Video probabilistic diffusion models in projected latent space. arXiv preprint arXiv:2302.07685 , 2023.

A. Zeng, A. Wong, S. Welker, K. Choromanski, F. Tombari, A. Purohit, M. Ryoo, V. Sindhwani, J. Lee, V. Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598 , 2022.

E. Zhang, Y. Lu, W. Wang, and A. Zhang. Lad: Language augmented diffusion for reinforcement learning. arXiv preprint arXiv:2210.15629 , 2022.

H. Zheng, W. Nie, A. Vahdat, K. Azizzadenesheli, and A. Anandkumar. Fast sampling of diffusion models via operator learning. In International Conference on Machine Learning , pages 42390-42402. PMLR, 2023.

Z. Zhong, D. Rempe, D. Xu, Y. Chen, S. Veer, T. Che, B. Ray, and M. Pavone. Guided conditional diffusion for controllable traffic simulation. arXiv preprint arXiv:2210.17366 , 2022.

D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 , 2023.

Appendix

In this Appendix, we discuss how we factorize the hierarchical decision-making process in Section A. In Section B, we then detail the background and architecture for visually grounded task planning, visual planning with video diffusion, and action planning with inverse dynamics model. In Section C, we discuss the training and evaluation details for the different levels of planning in HiP and the corresponding training hyperparameters. In Section D, we discuss implementation details for Gato and SayCan. In Section E, we showcase additional ablation studies comparing different approaches to enforce consistency across the levels of hierarchy, analyzing effect of granularity of subgoals on performance of HiP , ablating on the choice of video planning model and analyzing sensitivity of iterative refinement to hyperparameters. Finally, in Section F, we analyze runtime of different components of HiP .

A Factorizing Hierarchical Decision-Making Process

We model the hierarchical decision-making process described in Section 2 with p Θ which can be factorized into the task distribution p θ , visual distribution p ϕ , and action distribution p ψ .

Here, given random variables Y i , Y <i and Y ≤ i represents { Y 1 , . . . , Y i -1 } and { Y 1 , . . . , Y i } respectively. Now, we apply Markov assumption: Given current observation x i, 1 , future variables ( w i , τ i x , τ i a ) and past variables ( w j , τ j x , τ j a ∀ j < i ) are conditionally independent.

Wemodel task distribution p θ with a large language model (LLM) which is independent of observation x i, 1 . Since the image trajectory τ i x = { x i, 1: T } describes a physically plausible plan for achieving subgoal w i from observation x i, 1 , it is conditionally independent of goal g given subgoal w i and observation x i, 1 . Furthermore, we assume that an action a i,t can be recovered from observation at the same timestep x i,t and the next timestep x i,t +1 . Thus, we can write the factorization as

B Background and Architecture

B.1 Task Planning

Background on Density Ratio Estimation Let p and q be two densities, such that q is absolutely continuous with respect to p , denoted as q << p i.e. q ( x ) > 0 wherever p ( x ) > 0 . Then, their ratio is defined as r ( x ) = p ( x ) /q ( x ) over the support of p . We can estimate this density ratio r ( x ) by training a binary classifier to distinguish between samples from p and q [46, 14, 17]. More recent work [45] has shown one can introduce auxiliary densities { m i } i M =1 and train a multi-class classifier to distinguish samples between M classes to learn a better-calibrated and more accurate density ratio estimator. Once trained, the log density ratio can be estimated by log r ( x ) = ˆ h p ( x ) -ˆ h q ( x ) , where ˆ h i ( x ) is the unnormalized log probability of the input sample under the i th density, parameterized by the model.

Learning a Classifier to Visually Ground Task Planning Weestimate the density ratio p ( x i, 1 | w i ,g ) p ( x i, 1 | g ) with a multi-class classifier f ϕ ( x i, 1 , { w j } , g ) trained to distinguish samples amongst the conditional distributions p ( x i, 1 | w i , g ) , ..., p ( x i, 1 | w M , g ) and the marginal distribution p ( x i, 1 | g ) . Upon convergence, the classifier learns to assign high scores to ( x i, 1 , w i , g ) if w i is the subgoal corresponding to the observation x i, 1 and task g and low scores otherwise.

Architecture We parameterize f ϕ as a 4-layer multi-layer perceptron (MLP) on top of an ImageNetpretrained vision encoder (ResNet-18 [15]) and a frozen pretrained language encoder (Flan-T5Base [8]). The vision encoder encodes the observation x i, 1 , and the text encoder encodes the subgoals w j and the goal g . The encoded observation, the encoded subgoals, and the encoded goal are concatenated, and passed through a MLP with 3 hidden layers of sizes 512 , 256 , and 128 . The output dimension for MLP (i.e., number of classes for multi-classification) M is 6 for paint-block environment, 5 for object-arrange environment and 4 for kitchen-tasks environment.

Choice of Large Language Model We use GPT3.5-turbo [37] as our large language model.

B.2 Visual Planning

Background Diffusion Probabilistic Models [44, 19] learn the data distribution h ( x ) from a dataset D := { x i } . The data-generating procedure involves a predefined forward noising process q ( x k +1 | x k ) and a trainable reverse process p ϕ ( x k -1 | x k ) , both parameterized as conditional Gaussian distributions. Here, x 0 := x is a sample, x 1 , x 2 , ..., x K -1 are the latents, and x K ∼ N ( 0 , I ) for a sufficiently large K . Starting with Gaussian noise, samples are then iteratively generated through a series of 'denoising' steps. Although a tractable variational lower-bound on log p ϕ can be optimized to train diffusion models, Ho et al. [19] propose a simplified surrogate loss:

The predicted noise ϵ θ ( x k , k ) , parameterized with a deep neural network, estimates the noise ϵ ∼ N (0 , I ) added to the dataset sample x 0 to produce noisy x k .

Guiding Diffusion Models with Text Diffusion models are most notable for synthesizing highquality images [41, 35] and videos [20, 53] from text descriptions. Modeling the conditional data distribution q ( x | y ) makes it possible to generate samples satisfying the text description y . To enable conditional data generation with diffusion, Ho and Salimans [18] modified the original training setup to learn both a conditional ϵ ϕ ( x k , y , k ) and an unconditional ϵ ϕ ( x k , k ) model for the noise. The unconditional noise is represented, in practice, as the conditional noise ϵ ϕ ( x k , ∅ , k ) , where a dummy value ∅ takes the place of y . The perturbed noise ϵ ϕ ( x k , ∅ , k ) + ω ( ϵ ϕ ( x k , y , k ) -ϵ ϕ ( x k , ∅ , k ) ) (i.e. classifier-free guidance ) is used to later generate samples.

Video Diffusion in Latent Space As diffusion models generally perform denoising in the input space [19], the optimization and inference become computationally demanding when dealing with high-dimensional data, such as videos. Inspired by recent works [40, 53], we first use an autoencoder v enc to learn a latent space for our video data. It projects an observation trajectory τ x (i.e., video) into a 2D tri-plane representation [53] τ z = [ τ T z , τ H z , τ z W ] where τ T z , τ H z , τ z W capture variations in the video across time, height, and width respectively. We then diffuse over this learned latent space [53].

Latent Space Video Diffusion for Visual Planning Our video diffusion model p ϕ ( τ i x | w i , x i, 1 ) generates video τ i x given a language subgoal w i and the current observation x i, 1 . It is parameterized through its noise model ϵ ϕ (( τ i z ) k , w i , x i, 1 , k ) := ϵ ϕ (( τ i z ) k , l enc ( w i ) , v enc ( x i, 1 ) , k ) where τ i z := v enc ( τ i x ) is the latent representation of video τ i x over which we diffuse. We condition the noise model ϵ ϕ on subgoal w i using a pretrained language encoder l enc and on current observation x i, 1 using video encoder v enc. To use v enc with a single observation x i, 1 , we first tile the observation along the temporal dimension to create a video.

Architecture We now detail the architectures of different components:

Video Autoencoder We borrow our architecture for v enc from PVDM [53] which uses transformers to project video τ x ∈ R T × H × W to latent codes τ z = [ τ T z , τ H z , τ z W ] where τ T z ∈ R C × H ′ × W ′ , τ H z ∈ R C × T × W ′ , τ z W ∈ R C × H ′ × T . Here, T = 50 represents the time horizon of a video, H = 48 represents video height, W = 64 represents video width, C = 4 represents latent codebook dimension, H ′ = 12 represents latent height, and W ′ = 8 represents latent width.

Language Encoder We use Flan-T5-Base [8] as the pretrained frozen language encoder l enc .

Noise Model We borrow PVDM-L architecture [53] which uses 2D UNet architecture, similar to the one in Latent Diffusion Model (LDM) [40], to represent p ( τ z | τ ′ z ) . In our case, τ z = v enc ( τ i x ) and τ ′ z = v enc ( x i, 1 ) . To further condition noise model ϵ ϕ on l enc ( w i ) , we augment the 2D UNet Model with cross-attention mechanism borrowed by LDM [40].

For implementing these architectures, we used the codebase https://github.com/sihyun-yu/PVDM which contains the code for PVDM and LDM.

Classifier for Consistency between Visual Planning and Action Planning To ensure consistency between visual planning and action planning, we want to sample observation trajectories that maximizes both conditional observation trajectory likelihood from diffusion and the likelihood of sampled actions given the observation trajectory (see equation 4). To approximate likelihood calculation of action trajectory, we learn a binary classifier g ψ that models if the observation trajectory leads to a high likelihood action trajectories. Since diffusion happens in latent space and we use gradients from g ψ to bias the denoising of the video diffusion, g ψ ( τ i z ) takes the observation trajectory in latent space. The binary classifier g ψ is trained to distinguish between observation trajectories in latent space sampled from video dataset τ i z = v enc ( τ i x ) , τ i x ∼ D video (i.e. label of 1 ) and observation trajectories in latent space sampled from video dataset whose frames where randomly shuffled ( τ i z ) ′ = v enc ( σ ( τ i x )) , τ i x ∼ D video (i.e. label of 0 ). Here, σ denotes the random shuffling of frames. To randomly shuffle frames in an observation trajectory (of length 50 ), we first randomly select 5 frames in the observation trajectory. For each of the selected frame, we randomly permute it with its neighboring frame (i.e. either with the frame before it or with the frame after it). Once g ψ is trained, we use it to bias the denoising of the video diffusion

Here, ˆ ϵ is the noise used in denoising of the video diffusion and ω, ω ′ are guidance hyperparameters. Classifier Architecture The classifier g ψ ( τ z = [ τ T z , τ H z , τ z W ]) has a ResNet-9 encoder that converts τ T z , τ H z , and τ z W to latent vectors, then concatenate those latent vectors and passes the concatenated vector through an MLP with 2 hidden layers of sizes 256 and 128 and an output layer of size 1 .

B.3 Action Planning

To do action planning, we learn an inverse dynamics model to p ψ ( a i,t | x i,t , x i,t +1 ) predicts 7 -dimensional robot states s i,t = p ψ ( x i,t ) and s i,t +1 = p ψ ( x i,t +1 ) . The first 6 dimensions of the robot state represent joint angles and the last dimension of the robot state represents the gripper state (i.e., whether it's open or closed). The first 6 action dimension is represented as joint angle difference a i,t [: 6] = s i,t +1 [: 6] -s i,t [: 6] while the last action dimension is gripper state of next timestep a i,t [ -1] = s i,t +1 [ -1] .

Architecture We use ViT-B [9] (VC-1 [32] initialization) along with a linear layer to parameterize p ψ . ViT-B projects the observation x i,t ∈ R 48 × 64 × 3 to 768 dimensional latent vector from which the linear layer predicts the 7 dimensional state s i,t .

C Training and Evaluation

C.1 Task Planning

Inability	to	recognize	uncommon	objects

Hallucination	leading	to	wrong	spatial	reasoning

Is	screwdriver	in	brown	box?

Yes,	screwdriver	is	in	brown	box

Is	computer	hard	drive	in

brown	box?

No,	the	computer	hard	drive	is

not	in	the	brown	box

Is	scissor	in	brown	box?

Yes,	scissor	is	in	brown	box

User

Vision	Language	Model

Training Objective and Dataset for Learned Classifier We use a softmax cross-entropy loss to train the multi-class classifier f ϕ ( x i, 1 , { w j } j M =1 , g ) to classify an observation x i, 1 into one of the M given subgoal. We train it using the classification dataset D classify := { x i, 1 , g, { w j } j M =1 , i } consisting of observation x i, 1 , goal g , candidate subgoals { w j } j M =1 and the correct subgoal label i . The classification dataset for paint-block , object-arrange , and kitchen-tasks consists of 58k, 82k and 50k datapoints respectively.

Vision-Language Model (VLM) as a Classifier We use a frozen pretrained Vision-Language Model (VLM) (MiniGPT4 [58]) as a classifier. We first sample a list of all possible subgoals

W = { w i } i M =1 from the LLM given the language goal g . We then use the VLM to eliminate subgoals from W that have been completed. For each subgoal, we question the VLM whether that subgoal has been completed. For example, consider the subgoal "Place white block in yellow bowl". To see if the subgoal has been completed, we ask the VLM "Is there a block in yellow bowl?". Consider the subgoal "Place green block in brown box" as another example. To see if the subgoal has been completed, we ask the VLM "Is there a green block in brown box?". Furthermore, if the VLM says "yes" and the subgoal has been completed, we also remove other subgoals from W that should have been completed, such as "Place white block in green bowl". Once we have eliminated the completed subgoals, we use the domain knowledge to determine which subgoal to execute out of all the remaining subgoals. As an example, if the goal is to "Stack green block on top of blue block in brown box" and we have a green block in green bowl and a blue block in blue bowl, we should execute the subgoal "Place blue block in brown box" before the subgoal "Place green block on blue block". While this process of asking questions from VLM to determine the remaining subgoals and then sequencing the remaining subgoals doesn't require any training data, it heavily relies on the task's domain knowledge.

Failure Modes of VLM We observe two common failure modes of the VLM approach in object-arrange environment and visualize them in Figure 8. First, because the model is not trained on any in-domain data, it often fails to recognize uncommon objects, such as computer hard drives, in the observations. Second, it occasionally hallucinates the presence of objects at certain locations and thus leads to incorrect visual reasoning.

VLM as a Subgoal Predictor We also tried to prompt the VLM with 5 examples of goal g and subgoal candidates { w i } i M =1 and then directly use it to generate the next subgoal w i given the observation x i, 1 and the goal g . However, it completely failed. We hypothesize that the VLM fails to directly generate the next subgoal due to its inability to perform in-context learning.

Evaluation We evaluate the trained classifier f ϕ and the frozen VLM for subgoal prediction accuracy on 5k unseen datapoints, consisting of observation, goal, candidate subgoals and correct subgoal, generated from test tasks T test. We average over 4 seeds and show the results in Figure 7.

C.2 Visual Planning

Ego4D dataset processing We pre-train on canonical clips of the Ego4D dataset which are textannotated short clips made from longer videos. We further divide each canonical clip into 10sec segments from which we derive 50 frames. We resize each frame to 48 × 64 . We create a pretraining Ego4D dataset of (approximately) 344k short clips, each consisting of 50 frames and a text annotation. We use the loader from R3M [34] codebase (https://github.com/facebookresearch/r3m) to load our pretraining Ego4D dataset.

Training Objective and Dataset We use pixel-level L1 reconstruction and negative perceptual similarity for training the autoencoder v enc . We borrow this objective from PVDM [53] paper except we don't use adversarial loss. We keep the language encoder frozen. We use denoising loss in video latent space E k ∼ [1 ,K ] ,τ z ,w,x ∼D ,ϵ ∼N ( O,I ) [ ∥ ϵ -ϵ ϕ (( τ z ) k , l enc ( w ) , v enc ( x ) , k ) ∥ 2 ] to train the noise model ϵ ϕ . We replace w with a null token so that ϵ ϕ learns both a text-conditional model and an unconditional model. We pretrain the autoencoder v enc and the noise model ϵ ϕ on the processed Ego4D dataset. We then finetune it on our dataset D video := { τ i x , w i } consisting of approximately 100k observation trajectories of length T = 50 and associated text subgoals.

Classifier Training Objective and Dataset We use a binary cross-entropy loss to train the binary classifier g ψ ( τ z ) that predicts if the observation trajectory in latent space τ z = v enc ( τ x ) leads to high-likelihood action trajectory. It is trained using trajectories from video dataset τ x ∼ D video .

C.3 Action Planning

Training Objective and Dataset We train inverse dynamics p ψ on a dataset D inv. Since actions are differences between robotic joint states, we train p ψ to directly predict robotic state s i,t = p ψ ( x i,t ) by minimizing the mean squared error between the predicted robotic state and ground truth robotic state. Hence, D inv := { τ i x , τ i s } consists of 1k paired observation and robotic state trajectories, each having a length of T = 50 , in paint-block and object-arrange domains. In kitchen-tasks domain, it consists of 3.5k paired observation and robotic state trajectories, each having a length of T = 50 .

Evaluation We evaluate the trained p ψ (i.e., VC-1 initialized model and other related models in Figure 6) on 100 unseen paired observation and robotic state trajectories generated from test tasks T test. We use mean squared error to evaluate our inverse dynamics models. We use 4 seeds to calculate the standard error, represented by the shaded area in Figure 6.

C.4 Hyperparameters

Task Planning We train f ϕ for 50 epochs using AdamW optimizer [31], a batch size of 256 , a learning rate of 1 e -3 and a weight decay of 1 e -6 . We used one V100 Nvidia GPU for training the multi-class classifier.

Visual Planning We borrow our hyperparameters for training video diffusion from the PVDM paper [53]. We use AdamW optimizer [31], a batch size of 24 and a learning rate of 1 e -4 for training the autoencoder. We use AdamW optimizer, a batch size of 64 , and a learning rate of 1 e -4 for training the noise model. During the pretraining phase with the Ego4D dataset, we train the autoencoder for 5 epochs and then the noise model for 5 epochs. During the finetuning phase with D video, we train the autoencoder for 10 epochs and then the noise model for 40 epochs. We used two A6000 Nvidia GPUs for training these diffusion models. We train g ψ for 10 epochs using AdamW optimizer, a batch size of 256 and a learning rate of 1 e -4 . We used one V100 Nvidia GPU for training the binary classifier. During classifier-free guidance, we use ω = 4 and ω ′ = 1 .

Action Planning We train VC-1 initialized inverse dynamics model for 20 epochs with AdamW optimizer [31], a batch size of 256 and a learning rate of 3 e -5 . We trained other randomly initialized ViT-B inverse dynamics models and randomly initialized ResNet-18 inverse dynamics models for 20 epochs with AdamW optimizer, a batch size of 256 , and a learning rate of 1 e -4 . We used one V100 Nvidia GPU for training these inverse dynamics models.

D Implementation Details for Gato and SayCan

In training the visual and action planning for HiP , we use 100k robot videos for visual planner and the inverse dynamics, when trained from scratch, utilizes 10k state-action trajectory pairs. In order to ensure fair comparison, we use 110k datapoints for training Gato [39] and SayCan [2].

Gato We borrow the Gato [39] architecture from Vima Codebase and use it for training a language conditioned policy with imitation learning. We use 110k (langauge, observation trajectory, action trajectory) datapoints in each of the three domains for training Gato. Furthermore, we provide oracle subgoals to Gato.

SayCan We borrow the SayCan [2] algorithm from the SayCan codebase and adapt it to our settings. Following the recommendations of SayCan codebase, we use CLIPort policies as primitives. CLIPort policies take in top-down RGBD view and outputs pick and place pixel coordinates. Then, an underlying motion planner picks the object from the specified pick-coordinate and places the object at the specified place-coordinate. We train CLIPort policies on 110k (language, observation, action) datapoints in paint-block and object-arrange domain. The SayCan paper uses value function as an affordance function to select the correct subgoal given current observation and high level goal. However, CLIPort policies don't have a value function. The SayCan codebase uses a hardcoded scoring function which doesn't apply to object-arrange domain. To overcome these issues, we use the LLM grounding strategy from ? ]. It uses unnormalized logits over the pixel space given by CLIPort policies as affordance and uses it to ground LLM to current observation and thus predict the subgoal. We then compare SayCan with HiP and other baselines on paint-block and object-arrange domain in Table 1. While SayCan outpeforms other baselines, HiP still outperforms it both on seen and unseen tasks of paint-block and object-arrange domain. We couldn't run SayCan on kitchen-tasks domain as there's no clear-cut primitive in that domain. This points to a limitation of SayCan which requires tasks to be expressed in terms of primitives with each primitive paired with an affordance function.

E Additional Ablation Studies

E.1 Consistency between task planning and visual planning

To make task planning consistent with visual planning, we need to select subgoal w ∗ i which maximizes the joint likelihood (see equation 2) of LLM p LLM ( w i | g ) and video diffusion p ϕ ( τ i x | w i , x i, 1 ) . While

generating videos for different subgoal candidates w i and calculating the likelihood of the generated video is computationally expensive, we would still like to evaluate its performance in subgoal prediction given it is theoretically grounded. To this end, we first sample M subgoals W = { w j } j M =1 from the LLM. Then, we calculate w ∗ i = arg max w ∈ W log p ϕ ( τ i x | w,x i, 1 ) and use w ∗ i as our predicted subgoal. Since log p ϕ ( τ i x | w,x i, 1 ) is intractable, we estimate its variational lower-bound as an approximation. We use this approach for subgoal prediction in paint-block environment and compare its performance to that of the learned classifier. It achieves a subgoal prediction accuracy of 54 . 3 ± 7 . 2% whereas the learned classifier achieves a subgoal prediction accuracy of 98 . 2 ± 1 . 5% in paint-block environment. Both approaches outperform the approach of randomly selecting a subgoal from W (i.e., no task plan refinement), which yields a subgoal prediction accuracy of 16 . 67% given M = 6 . The poor performance of the described approach could result from the fact that the diffusion model only coarsely approximates the true distribution p ( τ i x | w i , x i, 1 ) , which results in loose variational lower-bound and thus uncalibrated likelihoods from the diffusion model. A larger diffusion model could better approximate p ( τ i x | w i , x i, 1 ) , resulting in tighter variational lower-bound and better-calibrated likelihoods.

E.2 Consistency between visual planning and action planning

To make visual planning consistent with action planning, we need to select observation trajectory ( τ i x ) ∗ which maximizes joint likelihood (see equation 4) of conditional video diffusion p ϕ ( τ i x | w i , x i, 1 ) and inverse model ∏ T -1 t =1 p ψ ( a i,t | x i,t , x i,t +1 ) . While sampling action trajectories and calculating their likelihoods during every step of the denoising process is computationally inefficient, we would still like to evaluate its effectiveness in visual plan refinements. However, we perform video diffusion in latent space while our inverse model is in observation space. Hence, for purpose of this experiment, we learn another inverse model p ψ ( τ i a | τ i z ) that uses a sequence model (i.e. a transformer) to produce an action trajectory τ i a given an observation trajectory in latent space τ i z . We train p ψ for 20 epochs on 10k paired observation and action trajectories, each having a length of T = 50 . We use AdamW optimizer, a batch size of 256 and a learning rate of 1 e -4 during training. To generate an observation trajectory that maximizes the joint likelihood, we first sample 30 observation trajectories from video diffusion p ϕ ( τ i x | w i , x i, 1 ) conditioned on subgoal w i and observation x i, 1 . For each generated observation trajectory τ i x , we sample a corresponding action trajectory τ i a and calculate its corresponding log-likelihood log p ψ ( τ i a | v enc ( τ i x )) . We select the observation trajectory τ i x with highest log-likelihood. Note that we only use p ψ for visual plan refinement and use p ψ for action execution to ensure fair comparison. If we use this approach for visual plan refinement with HiP , we obtain a success rate of 72 . 5 ± 1 . 9 on unseen tasks in paint-block environment. This is comparable to the performance of HiP with visual plan refinements from learned classifier g ψ which obtains a success rate of 72 . 8 ± 1 . 7 on unseen tasks in paint-block environment. In contrast, HiP without any visual plan refinement obtains a success rate of 71 . 1 ± 1 . 3 on unseen tasks in paint-block environment. These results show that g ψ serves as a good approximation for estimating whether an observation trajectory leads to a high-likelihood action trajectory, while still being computationally efficient.

Architecture for p ψ We use a transformer model to represent p ψ ( τ a | τ z = [ τ T z , τ H z , τ z W ]) . We first use a ResNet-9 encoder to convert τ T z , τ H z , and τ z W to latent vectors. We then concatenate those latent vectors and project the resulting vector to a hidden space of 64 dimension using a linear layer. We then pass the 64 dimensional vector to a trajectory transformer model [23] which generates an action trajectory τ a of length 50 . The trajectory transformer uses a transformer architecture with 4 layers and 4 self-attention heads.

E.3 How granularity of subgoals affects performance of HiP ?

We conduct a study in paint-block environment to analyze how granuality of subgoals affect HiP . In our current setup, a subgoal in paint-block domain is of form "Place <block color> block in/on/to <final block location> " and involves a pick and a place operation. We refer to our current setup as HiP (standard). We introduce two additional level of subgoal granuality:

Only one pick or place operation : The subgoal will be of form "Pick <block color> block in/on <initial block location> " or "Place <block color> block in/on/to <final block location> ". It will involve either one pick or one place operation. We refer to the model trained in this setup as HiP (more granular).

Two pick and place operations : The subgoal will be of form "Place <1st block color> block in/on/to <final 1st block location> and Place <2nd block color> block in/on/to <final 2nd block location> ". It will involve two pick and place operations. We refer to the model trained in this setup as HiP (less granular).

Table 2: Granularity of Subgoals. Performance of HiP as we vary the granularity of subgoals. Initially, it doesn't get affected but then starts to deteoriate when subgoals become too coarse.

Note that UniPi has the least granuality in terms of subgoals as it tries to imagine the entire trajectory from goal description. Table 2 in the rebuttal document compares HiP (standard), HiP (more granular), HiP (less granular) and UniPi on seen and unseen tasks in paint-block environment. We observe that HiP (standard) and HiP (more granular) have similar success rates where HiP (less granular) has a lower success rate. UniPi has the lowest success rate amongst these variants. We hypothesize that success rate of HiP remains intact when we decrease the subgoal granuality as long as the performance of visual planner doesn't degrade. Hence, HiP (standard) and HiP (more granular) have similar success rates. However, when the performance of visual planner degrades as we further decrease the subgoal granuality, we see a decline in success rate as well. That's why HiP (less granular) sees a decline in success rate and UniPi has the lowest success rate amongst all variants.

E.4 Ablation on Visual Planning Model

Table 3: Ablating Visual Planner. While performance gap between HiP and HiP (RSSM) is small in Paint-block , it widens in more visually complex domains, such as Object-arrange and Kitchen-tasks , thereby showing the importance of video diffusion model.

To show the benefits of video diffusion model, we perform an ablation where we use (text-conditioned) recurrent state space model (RSSM), taken from DreamerV3 [ ? ], as visual model for HiP. We borrow the RSSM code from dreamerv3-torch codebase. To adapt RSSM to our setting, we condition RSSM on subgoal (i.e. subgoal encoded into a latent representation by Flan-T5-Base) instead of actions. Hence, sequence model of RSSM becomes h t = f ( h t -1 , z t -1 , w ) where w is latent representation of subgoal. Furthermore, we don't predict any reward since we aren't in a reinforcement learning setting and don't predict continue vector since we decode for a fixed number of steps. Hence, we remove reward prediction and continue prediction from the prediction loss. To make the comparisons fair, we pretrain RSSM with Ego4D data as well. We report the results in Table 3. We see that HiP with video diffusion model outperforms HiP with RSSM in all the three domains. While the performance gap between HiP (RSSM) and HiP (i.e. using video diffusion) is small in paint-block environment, it widens in object-arrange and kitchen-tasks domains as the domains become more visually complex.

E.5 Analyzing sensitivity of iterative refinement to hyperparameters

Table 4: Sensitivity of visual iterative refinement to guidance scale. Performance of HiP as we vary the guidance scale ω ′ . HiP performs best when ω ′ ∈ { 1 , 1 . 25 } but performance degrades for higher values of ω ′ .

The subgoal classifier doesn't introduce any test time hyperparameters and we use standard hyperparameters ( 1 e -3 learning rate, 1 e -6 weight decay, 256 batch size, 50 epochs, Adam optimizer)

for its training which remains fixed across all domains. We observed that the performance changes minimally across different hyperparameters, given a learning rate decay over training. However, the observation trajectory classifier g ψ introduces an additional test time hyperparameter ω ′ which appropriately weights the gradient from observation trajectory classifier. Table 4 in the rebuttal document varies ω ′ between 0.5 and 2 in intervals of 0.25 and shows success rate of HiP. We see that HiP gives the best performance when ω ′ ∈ { 1 , 1 . 25 } but it's performance degrades for higher values of ω ′ .

F Analyzing Runtime of HiP

Table 5: Run time of HiP . Average episodic run-time of HiP , along with average run-time of its different components for Paint-block , Object-arrange and Kitchen-tasks domains. While HiP has similar planning times across different domains, it has different action execution times and episodic runtimes across domains due to differences in simulation properties and average number of subgoals.

We provide average runtime of HiP for a single episode in all the three domains in Table 5 of the rebuttal document. We average across 1000 seen tasks in each domain. We break the average runtime by different components: task planning (subgoal candidate generation and subgoal classification), visual planning, action planning and action execution. We execute the action plan for a subgoal in open-loop and then get observation from the environment for deciding the next subgoal. From Table 5, we see that majority of the planning time is taken by visual planning. Recent works [24, 42, 56] have proposed techniques to reduce sampling time in diffusion models, which can be incorporated into our framework for improving visual planning speed in the future.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents Hierarchical Planning with Foundation Models (HiP), a framework that addresses long-horizon decision-making in novel environments, using multiple modalities of data and reasoning at various levels of hierarchy. The key elements of this framework involve the use of a large language model for constructing symbolic plans, a video diffusion model to generate observation trajectory plans, and a large pre-trained robot ego-centric action model to map these plans to a robot's control space. The authors propose an iterative refinement approach for feedback incorporation, promoting a consensus among different models, without the need for large model finetuning. Experimental results on two long-horizon tabletop manipulation tasks were presented, demonstrating promising results for the proposed strategy. The authors also mention the potential for including other modalities, like touch and sound, in the future.\n\nSTRENGTHS:\nThe technical content of the paper appears to be sound, and the proposed framework has shown promising results in the experimental results provided. The methodology to combine language models, video diffusion models, and ego-centric robot control models into one system is comprehensive.\nThe paper is well-structured and clearly explains the approach and the reasoning behind the decisions made. The paper's flow from problem statement to proposed solution, and finally to experimental evaluation, is logical and easy to follow.\n\nWEAKNESSES:\n1. The baselines compared are all relatively weak. It's more appropriate to compare with foundation robotics models, such as SayCan, Gato[1], palm-e[2]. The authors argue that compared to saycan, they can generalise to new skills. However, they didn't demonstrate the generalization to new skills either. (They evaluated on unseen tasks but not new skills.)\n2. While the iterative refinement approach is interesting, it could be further scrutinized to understand its limitations better, particularly concerning computational efficiency and robustness.\n3. As this is mostly an experimental, please provide code to ensure reproducibility. \n4. Formatting issue in line 9, 45.\n\n[1] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez,\nYury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian\nChen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. In Transactions on Machine Learning\nResearch (TMLR), November 10, 2022.\n[2] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson,\nQuan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke,\nKarol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal\nlanguage model. In arXiv preprint arXiv:2303.03378, 2023.\n\nQUESTIONS:\nSee above. Could the authors discuss more about the efficiency and robustness? And also compare with a few foundation robotics model to better situate the work?\n\nThis paper presents a potentially significant step towards more effective long-horizon decision-making. The presented results are promising. I'm happy to raise my scores if the authors address my concerns.\n\nLIMITATIONS:\nThe authors discussed the limited. They didn't comment on the broader societal impacts but I didn't see any concern.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This work presents a new approach of combining foundation models from different modalities to into hierarchical system to solve long-horizon robotics problem. The performance of the system is evaluated on two long-horizon tasks and largely out-performed other baselines.

STRENGTHS:
The approach of this work is a effective combination of previously available models. The effectiveness of iterative refinement is particularly interesting (which seems to be one of the main performance booster in this approach). The framework details are well supported by the writing and supplementary materials.

WEAKNESSES:
I don't find weakness of this work, but I do have some questions in terms of the dataset, experiment, and analysis, see questions below.

QUESTIONS:
1. What does dataset stats look like?
2. Can you provide more description rationalize the use of training data to evaluate HiP (Table 1)?
3. line 213: What is benefits of pre-training video diffusion model in terms of success rate? The description seems to be only indirectly describing the benefits?
4. It would be interesting to see some analysis on the effectiveness of subgoal generation in this approach, for example, if the granularity of the subgoal generation various, what effect does it have on the overall approach. (This is interesting to me, without this will not affect my review score.)
5. How is the computation speed look like for this approach, how fast it is to generate for an episode?

LIMITATIONS:
Limitations are addressed in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a framework called Hierarchical Planning with Foundation Models (HiP), which is designed to improve decision-making in new environments with long-term goals. HiP uses hierarchical reasoning to plan subgoals, visually reason about plans, and execute actions through visual-motor control. The framework utilizes different kinds of knowledge to support various levels of decision-making. It employs a large language model for constructing symbolic plans, a large video diffusion model for grounding the plans in the environment, and an inverse dynamics model for inferring actions from generated videos. The models are kept consistent through iterative refinement. The effectiveness of HiP is demonstrated through its application in two long-horizon table-top manipulation tasks.\n\nThe paper's introduction explains that successful execution of tasks in unfamiliar environments requires reasoning at abstract, geometric, and control levels. The authors propose using multiple modalities of Internet-scale data to reason across these levels. They explain the limitations of existing models and propose HiP as a solution, which uses three different large foundation models to construct a physically executable plan for long-horizon tasks. The models are iteratively refined based on feedback from downstream models, ensuring that the final plan satisfies constraints at all levels. The authors also discuss the potential of training foundation models for videos and ego-centric actions, and the limitations of their current computational resources. They conclude by highlighting the promise of their proposed approach in long-horizon decision-making tasks and hope to inspire future work in more complex real-world tasks.\n\nSTRENGTHS:\n- Hierarchical Planning with Foundation Models (HiP) leverages different modalities of knowledge, which can improve decision-making in novel environments with long-horizon goals.\n- The framework uses a large language model, a large video diffusion model, and an inverse dynamics model, allowing it to construct symbolic plans, ground those plans in the environment, and infer actions from generated videos.\n- The iterative refinement process ensures consistency between the models and enables hierarchically consistent plans that are responsive to the goal and executable given the current state and agent.\n- The approach is computationally efficient to train as it doesn't require any large model finetuning.\n- The authors demonstrate promising results on two long-horizon tabletop manipulation environments, illustrating the efficacy and adaptability of the approach.\n\nWEAKNESSES:\n- The paper is technically similar to UniPi. The technical novelty could be potentially incremental - it adds language model to decompose the language goal to language subgoals and then apply video prediction model to it.\n- The work relies on foundation models for video prediction and ego-centric action prediction. Concern: Although it can be expected that video prediction model will be available in the future, but it may not be assumed to be suitable for robotics applications. Additionally, it seems to be overclaim that robotics should rely on these models.\n    1. The applications in robotics may need to model multiple aspects of physical environments, where “expected” video foundation models can struggle to learn\n    How could forces be modeled in video model? How to guarantee the generated video are feasible in physics and can be executed for unseen tasks? The shown pick-and-place is a useful environment, but doesn’t support all the claims. Either the claims need to be revised or more experiments are needed.\n    2. It is also likely that vision (image/video) is only one of several important modalities in real-world robotic decision-making. For example, for some insertion tasks, tactile sensing could be important.\n    3. The computational cost of these models is not known yet at all. Although GPT4 is available for use, it only needs to transfer text over internet. If the needed models were to transfer high-quality images or even videos, how much bandwidth would they need? Is that possible to allow real-time robotic decision-making? It seems unlikely that in the near future we will have pretrained action/video models for all scenarios that can give real-time results over internet. If the goal of this paper is to explore along this direction, I think these aspects should all be considered, instead of assuming all these challenges are nonexistent and just claim the contributions.\n    - Overall, I think the paper is a meaningful initial exploration towards this direction, but the claims of the paper should be clear about this as some assumptions are not necessarily realistic. The title “Hierarchical Planning with Foundation Models” does not necessarily match the contributions and novelties of this paper.\n- L68 — The step 2 “visual planning” is the key step to plan in physical level, but image-level is not necessarily the correct level of abstraction for many cases. When does it apply and when not? This seems to include lots of unnecessary details of the physical world for tasks other than just tabletop manipulation.\n- More general concern: The approach relies on the availability of large pretrained models in different domains. Currently, these models are only readily available in the language domain. Furthermore, as mentioned in the paper, the approach is demonstrated on smaller-scale video and ego-centric action models trained in simulation, which serve as proxies for larger pretrained models. This might limit the generalizability of the results to more complex real-world tasks.\n\nQUESTIONS:\nSee the weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a way to leverage large foundation models to perform long-horizon hierarchical planning tasks. Specifically, the authors leverage an LLM for subgoal generation from the text instruction, a video generation model for generating a plan for each subgoal, and an action prediction model that outputs an action given the current and next observation generated by the video generation model. To make the predictions between different models consistent, the authors propose the iterative refinement procedure to refine the prediction of a model based on joint distributions. The paper includes results in simulated manipulation environments and shows improvement over prior goal-conditioned/action planner/video planner-based methods. The authors also include ablation studies to demonstrate effectiveness of various design choices in the method, such as the use of iterative refinement.\n\nSTRENGTHS:\n1. The authors presents a nice way to combine three different large foundation models in the problem of hierarchical planning. The iterative refinement method seems very effective in aligning predictions of various models, which is of great significance in research that plays with using multiple foundation models.\n\n2. The authors perform rigorous empirical evaluations via comparing to various baselines and conducting a large number of ablation studies to show the good performance of the proposed method.\n\nWEAKNESSES:\n1. While iterative refinement method is intuitively reasonable and a natural choice for making predictions of models in the pipeline consistent, the authors use approximations of the target joint distribution in the iterative refinement step. It is unclear if the some approximations fully capture the target distribution. For example, I'm not sure if Eq. (3) is a good approximation of (2) since Eq. (3) doesn't have the reachability part and thus doesn't fully capture the feasibility of achieving the language goal.\n\n2. It is unclear if the method can outperform previous hierarhical planning methods such as [1, 2, 3, 4, 5, 6]. Since the authors only evaluate the method in simple simulated pick-and-place tasks, I'm not sure if we need large-scale video diffusion models for video generation and rather simple latent-space world model like [1] is very likely to work.\n\n3. Following the comment in 2, I think the authors should evaluate the method in more domains included in [1,2,3,4,5,6] and also more realistic domains such as real-world robot manipulation settings to fully demonstrate the necessity of using big video foundation models.\n\n[1] Hafner, Danijar, Kuang-Huei Lee, Ian Fischer, and Pieter Abbeel. \"Deep hierarchical planning from pixels.\" Advances in Neural Information Processing Systems 35 (2022): 26091-26104.\n\n[2] Hafner, Danijar, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. \"Mastering diverse domains through world models.\" arXiv preprint arXiv:2301.04104 (2023).\n\n[3] Zhao, Chao, Shuai Yuan, Chunli Jiang, Junhao Cai, Hongyu Yu, Michael Yu Wang, and Qifeng Chen. \"ERRA: An Embodied Representation and Reasoning Architecture for Long-horizon Language-conditioned Manipulation Tasks.\" IEEE Robotics and Automation Letters (2023).\n\n[4] Kujanpää, Kalle, Joni Pajarinen, and Alexander Ilin. \"Hierarchical Imitation Learning with Vector Quantized Models.\" arXiv preprint arXiv:2301.12962 (2023).\n\n[5] Mendonca, Russell, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, and Deepak Pathak. \"Discovering and achieving goals via world models.\" Advances in Neural Information Processing Systems 34 (2021): 24379-24391.\n\n[6] Tian, Stephen, Suraj Nair, Frederik Ebert, Sudeep Dasari, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. \"Model-based visual planning with self-supervised functional distances.\" arXiv preprint arXiv:2012.15373 (2020).\n\nQUESTIONS:\nPlease address the concerns raised in the section above.\n\nUpdate after rebuttal:\n\nI decided to raise my score after reading author’s response.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}