SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To make visual planning consistent with action planning, we need to select observation trajectory  $(\tau_{x}^{i})^{*}$  which maximizes joint likelihood (see equation 4) of conditional video diffusion  $p_{\phi}(\tau_x^i |w_i,x_{i,1})$  and inverse model  $\prod_{t = 1}^{T - 1}p_{\psi}(a_{i,t}|x_{i,t},x_{i,t + 1})$ . While sampling action trajectories and calculating their likelihoods during every step of the denoising process is computationally inefficient, we would still like to evaluate its effectiveness in visual plan refinements. However, we perform video diffusion in latent space while our inverse model is in observation space. Hence, for purpose of this experiment, we learn another inverse model  $\overline{p}_{\overline{\psi}}(\tau_a^i |\tau_z^i)$  that uses a sequence model (i.e. a transformer) to produce an action trajectory  $\tau_{a}^{i}$  given an observation trajectory in latent space  $\tau_z^i$ . We train  $\overline{p}_{\overline{\psi}}$  for 20 epochs on 10k paired observation and action trajectories, each having a length of  $T = 50$ . We use AdamW optimizer, a batch size of 256 and a learning rate of  $1e - 4$  during training. To generate an observation trajectory that maximizes the joint likelihood, we first sample 30 observation trajectories from video diffusion  $p_{\phi}(\tau_x^i |w_i,x_{i,1})$  conditioned on subgoal  $w_{i}$  and observation  $x_{i,1}$ . For each generated observation trajectory  $\tau_{x}^{i}$ , we sample a corresponding action trajectory  $\tau_{a}^{i}$  and calculate its corresponding log-likelihood  $\log \overline{p}_{\overline{\psi}}(\tau_a^i |v_{\mathrm{enc}}(\tau_x^i))$ . We select the observation trajectory  $\tau_x^i$  with highest log-likelihood. Note that we only use  $\overline{p}_{\overline{\psi}}$  for visual plan refinement and use  $p_{\psi}$  for action execution to ensure fair comparison. If we use this approach for visual plan refinement with HiP, we obtain a success rate of  $72.5\pm 1.9$  on unseen tasks in paint-block environment. This is comparable to the performance of HiP with visual plan refinements from learned classifier  $g_{\psi}$  which obtains a success rate of  $72.8\pm 1.7$  on unseen tasks in paint-block environment. In contrast, HiP without any visual plan refinement obtains a success rate of  $71.1\pm 1.3$  on unseen tasks in paint-block environment. These results show that  $g_{\psi}$  serves as a good approximation for estimating whether an observation trajectory leads to a high-likelihood action trajectory, while still being computationally efficient.
Architecture for  $\overline{p}_{\overline{\psi}}$  We use a transformer model to represent  $\overline{p}_{\overline{\psi}}(\tau_a|\tau_z = [\tau_z^T,\tau_z^H,\tau_z^W])$ . We first use a ResNet-9 encoder to convert  $\tau_z^T$ ,  $\tau_z^H$ , and  $\tau_z^W$  to latent vectors. We then concatenate those latent vectors and project the resulting vector to a hidden space of 64 dimension using a linear layer. We then pass the 64 dimensional vector to a trajectory transformer model [23] which generates an action trajectory  $\tau_{a}$  of length 50. The trajectory transformer uses a transformer architecture with 4 layers and 4 self-attention heads.
E.3 How granularity of subgoals affects performance of HiP?
We conduct a study in paint-block environment to analyze how granularity of subgoals affect HiP. In our current setup, a subgoal in paint-block domain is of form "Place <block color> block in/on/to <final block location>" and involves a pick and a place operation. We refer to our current setup as HiP (standard). We introduce two additional level of subgoal granularity:
- Only one pick or place operation: The subgoal will be of form "Pick <block color> block in/on <initial block location>" or "Place <block color> block in/on/to <final block location)". It will involve either one pick or one place operation. We refer to the model trained in this setup as HiP (more granular).
- Two pick and place operations: The subgoal will be of form "Place <1st block color> block in/on/to <final 1st block location> and Place <2nd block color> block in/on/to <final 2nd block location)". It will involve two pick and place operations. We refer to the model trained in this setup as HiP (less granular).
Model	HiP (more granular)	HiP (Standard)	HiP (less granular)	UniPi
Paint-block (Seen)	74.5 ± 1.8	74.3 ± 1.9	61.8 ± 3.1	37.2 ± 3.8
Paint-block (Unseen)	73.1 ± 2.1	72.8 ± 1.7	58.2 ± 3.4	35.3 ± 3.2
Note that UniPi has the least granularity in terms of subgoals as it tries to imagine the entire trajectory from goal description. Table 2 in the rebuttal document compares HiP (standard), HiP (more granular), HiP (less granular) and UniPi on seen and unseen tasks in paint-block environment. We observe that HiP (standard) and HiP (more granular) have similar success rates where HiP (less granular) has a lower success rate. UniPi has the lowest success rate amongst these variants. We hypothesize that success rate of HiP remains intact when we decrease the subgoal granularity as long as the performance of visual planner doesn't degrade. Hence, HiP (standard) and HiP (more granular) have similar success rates. However, when the performance of visual planner degrades as we further decrease the subgoal granularity, we see a decline in success rate as well. That's why HiP (less granular) sees a decline in success rate and UniPi has the lowest success rate amongst all variants.
E.4 Ablation on Visual Planning Model
Table 2: Granularity of Subgoals. Performance of HiP as we vary the granularity of subgoals. Initially, it doesn't get affected but then starts to deteriorate when subgoals become too coarse.
Model	Paint-block	Object-arrange	Kitchen-tasks
Seen	Unseen	Seen	Unseen	Seen	Unseen
HiP (RSSM)	70.2 ± 2.4	69.5 ± 1.6	59.6 ± 3.8	59.2 ± 3.9	50.6 ± 16.2	46.8 ± 19.4
HiP	74.3 ± 1.9	72.8 ± 1.7	75 ± 2.8	75.4 ± 2.6	85.8 ± 9.4	83.5 ± 10.2
To show the benefits of video diffusion model, we perform an ablation where we use (text-conditioned) recurrent state space model (RSSM), taken from DreamerV3 [? ], as visual model for HiP. We borrow the RSSM code from dreamerv3-torch codebase. To adapt RSSM to our setting, we condition RSSM on subgoal (i.e. subgoal encoded into a latent representation by Flan-T5-Base) instead of actions. Hence, sequence model of RSSM becomes  $h_t = f(h_{t - 1},z_{t - 1},w)$  where w is latent representation of subgoal. Furthermore, we don't predict any reward since we aren't in a reinforcement learning setting and don't predict continue vector since we decode for a fixed number of steps. Hence, we remove reward prediction and continue prediction from the prediction loss. To make the comparisons fair, we pretrain RSSM with Ego4D data as well. We report the results in Table 3. We see that HiP with video diffusion model outperforms HiP with RSSM in all the three domains. While the performance gap between HiP(RSSM) and HiP (i.e. using video diffusion) is small in paint-block environment, it widens in object-arrange and kitchen-tasks domains as the domains become more visually complex.
E.5 Analyzing sensitivity of iterative refinement to hyperparameters
Table 3: Ablating Visual Planner. While performance gap between HiP and HiP (RSSM) is small in Paint-block, it widens in more visually complex domains, such as Object-arrange and Kitchen-tasks, thereby showing the importance of video diffusion model.
HiP	ω' = 0.5	ω' = 0.75	ω' = 1.0	ω' = 1.25	ω' = 1.5	ω' = 1.75	ω' = 2.0
Paint-block (Seen)	71.8 ± 2.3	72.3 ± 2.0	74.3 ± 1.9	73.9 ± 2.2	72.1 ± 1.7	70.4 ± 2.4	68.2 ± 1.9
Paint-block (Unseen)	71.1 ± 2.5	71.4 ± 1.8	72.8 ± 1.7	73.1 ± 1.5	71.4 ± 1.5	69.3 ± 2.7	66.8 ± 1.4
Table 4: Sensitivity of visual iterative refinement to guidance scale. Performance of HiP as we vary the guidance scale  $\omega^{\prime}$ . HiP performs best when  $\omega^{\prime} \in \{1, 1.25\}$  but performance degrades for higher values of  $\omega^{\prime}$ .
The subgoal classifier doesn't introduce any test time hyperparameters and we use standard hyperparameters (1e-3 learning rate, 1e-6 weight decay, 256 batch size, 50 epochs, Adam optimizer)
for its training which remains fixed across all domains. We observed that the performance changes minimally across different hyperparameters, given a learning rate decay over training. However, the observation trajectory classifier  $g_{\psi}$  introduces an additional test time hyperparameter  $\omega'$  which appropriately weights the gradient from observation trajectory classifier. Table 4 in the rebuttal document varies  $\omega'$  between 0.5 and 2 in intervals of 0.25 and shows success rate of HiP. We see that HiP gives the best performance when  $\omega' \in \{1, 1.25\}$  but it's performance degrades for higher values of  $\omega'$ .
F Analyzing Runtime of HiP
Domain	Subgoal candidate generation	Subgoal classification	Visual planning per subgoal	Action planning per subgoal	Action execution per subgoal	Episodic runtime
Paint-block	1.85s	0.41s	7.32s	0.91s	6.35s	80.61
Object-arrange	1.9s	0.43s	7.39s	0.89s	9.57s	78.71
Kitchen-tasks	1.81s	0.41s	7.35s	0.98s	1.28s	40.37
Table 5: Run time of HiP. Average episodic run-time of HiP, along with average run-time of its different components for Paint-block, Object-arrange and Kitchen-tasks domains. While HiP has similar planning times across different domains, it has different action execution times and episodic runtimes across domains due to differences in simulation properties and average number of subgoals.
We provide average runtime of HiP for a single episode in all the three domains in Table 5 of the rebuttal document. We average across 1000 seen tasks in each domain. We break the average runtime by different components: task planning (subgoal candidate generation and subgoal classification), visual planning, action planning and action execution. We execute the action plan for a subgoal in open-loop and then get observation from the environment for deciding the next subgoal. From Table 5, we see that majority of the planning time is taken by visual planning. Recent works [24, 42, 56] have proposed techniques to reduce sampling time in diffusion models, which can be incorporated into our framework for improving visual planning speed in the future.