SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Train and Test Tasks For all environments, we sample two sets of tasks  $\mathcal{T}_{train}$ ,  $\mathcal{T}_{test} \sim p(\mathcal{T})$ . We use the train set of tasks  $\mathcal{T}_{train}$  to create datasets  $\mathcal{D}_{\text{classify}}$ ,  $\mathcal{D}_{\text{video}}$ ,  $\mathcal{D}_{\text{inv}}$  and other datasets required for training baselines. We ensure the test set of tasks  $\mathcal{T}_{test}$  contains novel combinations of object colors in paint-block, novel combinations of object categories in object-arrange, and novel combinations of kitchen subtasks in kitchen-tasks.
Evaluation Metrics We quantitatively evaluate a model by measuring its task completion rate for paint-block and object-arrange, and subtask completion rate for kitchen tasks. We use the simulator to determine if the goal, corresponding to a task, has been achieved. We evaluate a model on  $\mathcal{T}_{train}$  (seen) to test its ability to solve long-horizon tasks and on  $\mathcal{T}_{test}$  (unseen) to test its ability to generalize to long-horizon tasks consisting of novel combinations of object colors in paint-block,
Knnnne nnnnne
Model	Paint-block	Object-arrange	Kitchen-tasks
Seen	Unseen	Seen	Unseen	Seen	Unseen
Transformer BC (oracle subgoals)	8.3 ± 1.9	5.1 ± 1.6	10.2 ± 2.9	7.3 ± 1.7	48.4 ± 21.6	32.1 ± 24.2
Gato (oracle subgoals)	31.2 ± 2.4	28.6 ± 2.9	37.9 ± 3.3	36.5 ± 3.2	70.2 ± 10.8	66.8 ± 12.2
Trajectory Transformer (oracle subgoals)	22.1 ± 2.1	22.3 ± 2.5	30.5 ± 2.3	29.8 ± 2.9	66.4 ± 20.7	52.1 ± 22.3
Action Diffuser (oracle subgoals)	21.6 ± 2.6	18.2 ± 2.3	29.2 ± 2.4	27.6 ± 2.1	65.9 ± 23.2	55.1 ± 22.8
HiP (Ours, oracle subgoals)	81.2 ± 1.8	79.6 ± 1.9	91.8 ± 2.9	92.3 ± 2.3	92.8 ± 7.1	89.8 ± 7.6
UniPi	37.2 ± 3.8	35.3 ± 3.2	44.1 ± 3.1	44.2 ± 2.9	74.6 ± 14.8	73.4 ± 11.2
SayCan	67.2 ± 3.3	62.8 ± 3.7	70.3 ± 2.6	66.9 ± 2.8	-	-
HiP (Ours)	74.3 ± 1.9	72.8 ± 1.7	75 ± 2.8	75.4 ± 2.6	85.8 ± 9.4	83.5 ± 10.2
Table 1: Performance on Long-Horizon tasks. HiP not only outperforms the baselines in solving seen long-horizon tasks but its performance remains intact when solving unseen long-horizon tasks containing novel combination of objects colors in paint-block, novel combination of objects categories in object-rearrange and novel combination of subtasks in kitchen-tasks.
object categories in object-arrange, and kitchen subtasks in kitchen-tasks. We sample 1000 tasks from  $\mathcal{T}_{train}$  and  $\mathcal{T}_{test}$  respectively, and obtain average task completion rate on paint-block and object-arrange domains and average subtask completion rate on kitchen tasks domain. We report the mean and the standard error over 4 seeds in Table 1.
3.2 Baselines
There are several existing strategies for constructing robot manipulation policies conditioned on language goals, which we use as baselines in our experiments:
- Goal-Conditioned Policy A goal-conditioned transformer  $a_{i,t} \sim p(a_{i,t}|x_{i,t},w_i)$  that outputs action  $a_{i,t}$  given a language subgoal  $w_i$  and current observation  $x_{i,t}$  (Transformer BC) [6]. We provide the model with oracle subgoals and encode these subgoals with a pretrained language encoder (Flan-T5-Base). We also compare against goal-conditioned policy with Gato [39] transformer.
- Video Planner A video diffusion model (UniPi) [12]  $\{\tau_x^i\} \sim p(\{\tau_x^i\} |g,x_{i,1}),a_{i,t}\sim p(a_{i,t}|x_{i,t},x_{i,t + 1})$  that bypasses task planning, generates video plans for the entire task  $\{\tau_x^i\}$ , and infers actions  $a_{i,t}$  using an inverse model.
- Action Planners Transformer models (Trajectory Transformer) [23] and diffusion models (Diffuser) [24, 4]  $\{a_{i,t:T - 1}\} \sim p(\{a_{i,t:T - 1}\} |x_{i,t},w_i)$  that produce an action sequence  $\{a_{i,t:T - 1}\}$  given a language subgoal  $w_{i}$  and current visual observation  $x_{i,t}$ . We again provide the agents with oracle subgoals and encode these subgoals with a pretrained language encoder (Flan-T5-Base).
- LLM as Skill Manager A hierarchical system (SayCan) [2] with LLM as high level policy that sequences skills sampled from a repertoire of skills to accomplish a long-horizon task. We use CLIPort [43] policies as skills and the unnormalized logits over the pixel space it produces as affordances. These affordances grounds the LLM to current observation for producing next subgoal.
We begin by comparing the performance of HiP and baselines to solve long-horizon tasks in paint-block, object-arrange, kitchen-tasks environments. Table 1 shows that HiP significantly outperforms the baselines, although the baselines have an advantage and have access to oracle subgoals. HiP's superior performance shows the importance of (i) hierarchy given it outperforms goal-conditioned policy (Transformer BC and Gato), (ii) task planning since it outperforms video planners (UniPi), and (iii) visual planning given it outperforms action planners (Trajectory Transformer, Action Diffuser). It also shows the importance of representing skills with video-based planners which can be pre-trained on Internet videos and can be applied to tasks (such as kitchen-tasks). SayCan, in contrast, requires tasks to be decomposed into primitives paired with an affordance function, which can be difficult to define for many tasks like the kitchen task. Thus, we couldn't run SayCan on kitchen-tasks environment. Finally, to quantitatively show how the errors in  $f_{\theta}(x_{i,1},w_i,g)$  affect the performance of HiP, we compare it to HiP with oracle subgoals. For further details on the training and evaluation of HiP, please refer to Appendix C. For implementation details on Gato and SayCan, please refer to Appendix D. For runtime analysis of HiP, please refer to Appendix F.
Combinatorial Generalization to Unseen Long-horizon Tasks We also quantitatively test the ability of HiP to generalize to unseen long-horizon tasks, consisting of novel combinations of object colors in paint-block, object categories in object-arrange, and subtasks in kitchen-tasks. Table 1 shows that HiP's performance remains intact when solving unseen long-horizon tasks, and
Subgoal trajectories
Figure 4: Execution trajectory of HiP on an novel long-horizon task in paint-block environment.
still significantly outperforms the baselines. Figure 4 visualizes the execution of HiP in unseen long-horizon tasks in paint-block.
Figure 5: Pretraining video diffusion model with the Ego4D dataset consistently yields higher success rate and lower FVD scores (lower is better), even with reduced training dataset sizes. With pretraining, the model's FVD score escalates less gradually and its success rate falls less steeply as the dataset size shrinks.
Pre-training Video Diffusion Model We investigate how much our video diffusion model benefits from pre-training on the Internet-scale data. We report both the success rate of HiP and Fréchet Video Distance (FVD) score that quantifies the similarity between generated videos and ground truth videos, where lower scores indicate greater similarity in Figure 5. We see that pretraining video diffusion leads to a higher success rate and lower FVD score. If we reduce the training dataset to  $75\%$  and  $50\%$  of the original dataset, the FVD score for video diffusion models (both, with and without Ego4D dataset pretraining) increases and their success rate falls. However, the video diffusion model with Ego4D dataset pretraining consistently gets higher success rate and lower FVD scores across different dataset sizes. As we decrease the domain-specific training data, it is evident that the gap in performance between the model with and without the Ego4D pre-training widens. For details on how we process the Ego4D dataset, please refer to Appendix C.2.
Pre-training Inverse Dynamics Model We also analyze the benefit of pre-training our inverse dynamics model and report the mean squared error between the predicted and ground truth actions in Figure 6. The pre-training comes in the form of initializing the inverse dynamics model with
Figure 6: Pretraining inverse dynamics model. In paint-block and object-arrange (kitchen-tasks), when initialized with VC-1 weights, inverse dynamics model matches the performance of a randomly initialized model trained on 10K trajectories with just 1K (3.5k) trajectories. A smaller ResNet-18 model requires 2.5K (6k) trajectories to approach the same performance. The yellow and brown lines are overlaid on top of each other.
weights from VC-1 [32], a vision-transformer (ViT-B) [9] trained on ego-centric images with masked-autoencoding objective [16]. In paint-block and object-arrange, we see that the inverse dynamics, when initialized with weights from VC-1, only requires 1K labeled robotic trajectories to achieve the same performance as the inverse dynamics model trained on 10K labeled robotic trajectories but without VC-1 initialization. We also compare against an inverse dynamics model parameterized with a smaller network (ResNet-18). However, the resulting inverse dynamics model still requires 2.5K robotic trajectories to get close to the performance of the inverse dynamics model with VC-1 initialization in paint-block and object-arrange. In kitchen-tasks, inverse dynamics, when initialized with weights from VC-1, only requires 3.5k labeled robotic trajectories to achieve the same performance as the inverse dynamics model trained on 10K labeled robotic trajectories but without VC-1 initialization. When parameterized with ResNet-18, the inverse dynamics model still requires 6k robotic trajectories to get close to the performance of the inverse dynamics model with VC-1 initialization.
Figure 7: Ablation Studies. (Left) While task plan refinement is critical to HiP's performance, visual plan refinement improves HiP's performance by a smaller margin in paint block environment. (Right) While frozen pretrained VLM (MiniGPT4) matches the performance of a learned classifier in paint-block environment, its performance deteriorates in a more visually complex object-arrange environment.
Importance of Task Plan and Visual Plan Refinements We study the importance of refinement in task and visual planning in Figure 7. We compare to HiP without visual plan refinement and HiP without visual and task plan refinement in paint block environment. We see that task plan refinement for visual grounding of LLM is critical to the performance of HiP. Without it, the task plan is agnostic to the robot's observation and predicts subgoals that lead to erroneous visual and action planning. Furthermore, visual plan refinement improves the performance of HiP as well, albeit by a small margin. For a description of the hyperparameters used, please refer to Appendix C.4.
Exploring Alternate Strategies for Visually Grounding LLM We use a learned classifier  $f_{\theta}(x_{i,1},w_i,g)$  to visually ground the LLM. We explore if we can use a frozen pretrained Vision-Language Model (MiniGPT-4 [58]) as a classifier in place of the learned classifier. Although we didn't use any training data, we found the prompt engineering using the domain knowledge of the task to be essential in using the Vision-Language Model (VLM) as a classifier (see Appendix C.1 for details). We use subgoal prediction accuracy to quantify the performance of the learned classifier and the frozen VLM. Figure 7 illustrates that while both our learned multi-class classifier and frozen VLM perform comparably in the paint-block environment, the classifier significantly outperforms the VLM in the more visually complex object-arrange environment. We detail the two common failure modes of the VLM approach in object-arrange environment in Appendix C.1. As VLMs continue to improve, it is possible that their future versions match the performance of
learned classifiers and thus replace them in visually complex domains as well. For further details on the VLM parameterization, please refer to Appendix C.1.
The field of foundation models for decision-making [52] has seen significant progress in recent years. A large body of work explored using large language models as zero-shot planners [21, 22, 29, 30, 2], but it is often difficult to directly ground the language model on vision. To address this problem of visually grounding the language model, other works have proposed to directly fine-tune large language models for embodied tasks [28, 10]. However, such an approach requires large paired vision and language datasets that are difficult to acquire. Most similar to our work, SayCan [2] uses an LLM to hierarchically execute different tasks by breaking language goals into a sequence of instructions, which are then inputted to skill-based value functions. While SayCan assumes this fixed set of skill-based value functions, our skills are represented as video-based planners [12], enabling generalization to new skills.
Another set of work has explored how to construct continuous space planners with diffusion models [25, 4, 49, 55, 48, 57, 12]. Existing works typically assume task-specific datasets from which the continuous-space planner is derived [24, 3, 55]. Most similar to our work, UniPi [12] proposes to use videos to plan in image space and similarly relies on internet videos to train image space planners. We build on top of UniPi to construct our foundation model for hierarchical planning, and illustrate how UniPi may be combined with LLMs to construct longer horizon continuous video plans.
Moreover, some works [28, 54, 5] explored how different foundation models may be integrated with each other. In Flamingo [5], models are combined through joint finetuning with paired datasets, which are difficult to collect. In contrast both Zeng et al. [54] and Li et al. [28] combine different models zero-shot using either language or iterative consensus. Our work proposes to combine language, video, and ego-centric action models together by taking the product of their learned distributions [11]. We use a similar iterative consensus procedure as in Li et al. [28] to sample from the entire joint distribution and use this combined distribution to construct a hierarchical planning system.
5 Limitations and Conclusion
**Limitations** Our approach has several limitations. As high-quality foundation models for visual sequence prediction and robot action generation do not exist yet, our approach relies on smaller-scale models that we directly train. Once high-quality video foundation models are available, we can use them to guide our smaller-scale video models [51] which would reduce the data requirements of our smaller-scale video models. Furthermore, our method uses approximations to sample from the joint distribution between all the model. An interesting avenue for future work is to explore more efficient and accurate methods to ensure consistent samples from the joint distribution.
Conclusion In this paper, we have presented an approach to combine many different foundation models into a consistent hierarchical system for solving long-horizon robotics problems. Currently, large pretrained models are readily available in the language domain only. Ideally, one would train a foundation model for videos and ego-centric actions, which we believe will be available in the near future. However, our paper focuses on leveraging separate foundation models trained on different modalities of internet data, instead of training a single big foundation model for decision making. Hence, for the purposes of this paper, given our computational resource limitations, we demonstrate our general strategy with smaller-scale video and ego-centric action models trained in simulation, which serve as proxies for larger pretrained models. We show the potential of this approach in solving three long-horizon robot manipulation problem domains. Across environments with novel compositions of states and goals, our method significantly outperforms the state-of-the-art approaches towards solving these tasks.
In addition to building larger, more general-purposed visual sequence and robot control models, our work suggests the possibility of further using other pretrained models in other modalities, such as touch and sound, which may be jointly combined and used by our sampling approach. Overall, our work paints a direction towards decision making by leveraging many different powerful pretrained models in combination with a tiny bit of training data. We believe that such a system will be substantially cheaper to train and will ultimately result in more capable and general-purpose decision making systems.
The authors would like to thank the members of Improbable AI Lab for discussions and helpful feedback. We thank MIT Supercloud and the Lincoln Laboratory Supercomputing Center for providing compute resources. This research was supported by an NSF graduate fellowship, a DARPA Machine Common Sense grant, ARO MURI Grant Number W911NF-21-1-0328, ARO MURI W911NF2310277, ONR MURI Grant Number N00014-22-1-2740, and an MIT-IBM Watson AI Lab grant. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the United States Army Research Office, the Office of Naval Research, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes, notwithstanding any copyright notation herein.
Anurag Ajay co-conceived the framework of leveraging pretrained foundation models for decision making, implemented visual planning and action planning in HiP, evaluated HiP on long-horizon tasks, performed ablation studies and helped in paper writing.
Seungwook Han co-conceived in conceiving the framework of leveraging pretrained foundation models for decision making, implemented task planning in HiP and helped in paper writing.
Yilun Du co-conceived the framework of leveraging pretrained foundation models for decision making, implemented trajectory transformer and transformer BC and evaluated them on long-horizon tasks, implemented data generation scripts for object arrange and paint block environment, and lead paper writing.
Shuang Li helped in conceiving the idea of iterative refinement for consistency between pretrained foundation models, participated in research discussions and helped in writing paper.
Abhi Gupta participated in research discussions and helped in making figures.
Tommi Jaakkola participated in research discussions.
Joshua Tenenbaum participated in research discussions.
Leslie Kaelbling participated in research discussions, suggested baselines and ablation studies, conceived the structure of the paper and helped in paper writing.
Akash Srivastava participated in research discussions, suggested the idea of using classifier for consistency between observations and the large language model and provided feedback on paper writing.
Pulkit Agrawal was involved in research discussions, suggested ablation studies related to iterative refinement, provided feedback on writing, positioning of the work and overall advising.
[1] P. Agrawal, A. V. Nair, P. Abbeel, J. Malik, and S. Levine. Learning to poke by poking: Experiential learning of intuitive physics. Advances in neural information processing systems, 29, 2016.
[2] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
[3] A. Ajay, A. Kumar, P. Agrawal, S. Levine, and O. Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010.13611, 2020.
[4] A. Ajay, Y. Du, A. Gupta, J. Tenenbaum, T. Jaakkola, and P. Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657, 2022.
[5] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: A visual language model for few-shot learning. NeurIPS, 2022. URL https://arxiv.org/abs/2204.14198.
In this Appendix, we discuss how we factorize the hierarchical decision-making process in Section A. In Section B, we then detail the background and architecture for visually grounded task planning, visual planning with video diffusion, and action planning with inverse dynamics model. In Section C, we discuss the training and evaluation details for the different levels of planning in HiP and the corresponding training hyperparameters. In Section D, we discuss implementation details for Gato and SayCan. In Section E, we showcase additional ablation studies comparing different approaches to enforce consistency across the levels of hierarchy, analyzing effect of granularity of subgoals on performance of HiP, ablating on the choice of video planning model and analyzing sensitivity of iterative refinement to hyperparameters. Finally, in Section F, we analyze runtime of different components of HiP.
A Factorizing Hierarchical Decision-Making Process
We model the hierarchical decision-making process described in Section 2 with  $p_{\Theta}$  which can be factorized into the task distribution  $p_{\theta}$ , visual distribution  $p_{\phi}$ , and action distribution  $p_{\psi}$ .
$$
\begin{array}{l} p _ {\Theta} \left(W, \left\{\tau_ {x} ^ {i} \right\}, \left\{\tau_ {a} ^ {i} \right\} | g, x _ {1, 1}\right) = \prod_ {i = 1} ^ {N} p _ {\theta} \left(w _ {i} | g, x _ {i, 1}, w _ {<   i}, \tau_ {x} ^ {<   i}, \tau_ {a} ^ {<   i}\right) \prod_ {i = 1} ^ {N} p _ {\phi} \left(\tau_ {x} ^ {i} | w _ {\leq i}, x _ {i, 1}, g, \tau_ {x} ^ {<   i}, \tau_ {a} ^ {<   i}\right) \\ \prod_ {i = 1} ^ {N} p _ {\psi} (\tau_ {a} ^ {i} | \tau_ {x} ^ {\le i}, w _ {\le i}, x _ {i, 1}, g, \tau_ {a} ^ {<   i}) \\ \end{array}
$$
Here, given random variables  $Y^{i}$ ,  $Y^{<i}$  and  $Y^{\leq i}$  represents  $\{Y^{1},\ldots ,Y^{i - 1}\}$  and  $\{Y^1,\dots ,Y^i\}$  respectively. Now, we apply Markov assumption: Given current observation  $x_{i,1}$ , future variables  $(w_{i},\tau_{x}^{i},\tau_{a}^{i})$  and past variables  $(w_{j},\tau_{x}^{j},\tau_{a}^{j}\forall j < i)$  are conditionally independent.
$$
p _ {\Theta} \left(W, \left\{\tau_ {x} ^ {i} \right\}, \left\{\tau_ {a} ^ {i} \right\} | g, x _ {1, 1}\right) = \prod_ {i = 1} ^ {N} p _ {\theta} \left(w _ {i} | g, x _ {i, 1}\right) \prod_ {i = 1} ^ {N} p _ {\phi} \left(\tau_ {x} ^ {i} | w _ {i}, x _ {i, 1}, g\right) \prod_ {i = 1} ^ {N} p _ {\psi} \left(\tau_ {a} ^ {i} | \tau_ {x} ^ {i}, w _ {i}, x _ {i, 1}, g\right)
$$
We model task distribution  $p_{\theta}$  with a large language model (LLM) which is independent of observation  $x_{i,1}$ . Since the image trajectory  $\tau_{x}^{i} = \{x_{i,1:T}\}$  describes a physically plausible plan for achieving subgoal  $w_{i}$  from observation  $x_{i,1}$ , it is conditionally independent of goal  $g$  given subgoal  $w_{i}$  and observation  $x_{i,1}$ . Furthermore, we assume that an action  $a_{i,t}$  can be recovered from observation at the same timestep  $x_{i,t}$  and the next timestep  $x_{i,t+1}$ . Thus, we can write the factorization as
$$
p _ {\Theta} (W, \{\tau_ {x} ^ {i} \}, \{\tau_ {a} ^ {i} \} | g, x _ {1, 1}) = \left(\prod_ {i = 1} ^ {N} p _ {\theta} (w _ {i} | g)\right) \left(\prod_ {i = 1} ^ {N} p _ {\phi} (\tau_ {x} ^ {i} | w _ {i}, x _ {i, 1})\right) \left(\prod_ {i = 1} ^ {N} \prod_ {t = 1} ^ {T - 1} p _ {\psi} (a _ {i, t} | x _ {i, t}, x _ {i, t + 1})\right)
$$
B Background and Architecture
B.1 Task Planning