SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Visual perception For CALVIN experiments, we utilize a visual perception network consists of three convolutional layers and a spatial softmax layer. It is the same as the perception network used in Lynch et al. [28] and Rosete-Beas et al. [35].
Skill encoder and decoder The skill encoder and decoder are trained using the code provided by Rosete-Beas et al. [35], and have the same architecture as LMP and TACO-RL.
E Training details
D4RL experiments We train the encoder, goal-conditioned actor, critic and diffusion model simultaneously using a unified loss function:  $J_{\text{unified}} \coloneqq J_{\text{actor}} + J_{\text{critic}} + \alpha J_{\text{diffusion}}$ , where the coefficient  $\alpha$  is fixed to 0.001 for all the experiments. We train the models for 2.5M training steps for Antmaze environments and 1.0M steps for Kitchen environments. We utilize NVIDIA Geforce RTX 3060 Ti graphics card for training our models, taking approximately 14 hours per 1.0M training steps. We use Adam optimizer [23], with a learning rate of 0.0001.
CALVIN experiments We train our model for 1.0M training steps using the same graphics card used for D4RL experiments, taking approximately 40 hours per 1.0M training steps. We use the same optimizer and learning rate used in D4RL experiments.
F Hyperparameter setting
In this section, we describe hyperparameter details.
- We set dimension of goal space  $\mathcal{G}$  to 16 for Antmaze environments, and 32 for Kitchen and CALVIN environments.
- We use the number of milestones  $K = 30$  with maximum interval  $\triangle_{\mathrm{max}} = 32$  for Antmaze environments, and  $K = 14$  with maximum interval  $\triangle_{\mathrm{max}} = 16$  for Kitchen and CALVIN environments.
- We set  $\lambda$  in Equation (9) in the main paper to be adaptive to scale of estimated log-likelihood  $\log p(g|s,a)$ , by setting it as  $\lambda = \frac{\tilde{\lambda}}{|\mathcal{B}|}\sum_{(s,a,g)\in\mathcal{B}}|\log p(g|s,a)|$  as done in Fujimoto and Gu [9], where  $\mathcal{B}$  denotes a mini-batch. We use  $\tilde{\lambda}$  of 2.5 for Antmaze environments, and 0.05 for Kitchen and CALVIN environments.
- We use diffusion timestep  $N$  of 300, diffusion guidance coefficient  $\beta$  of 0.5, and target temporal distance  $\triangle_{\mathrm{target}}$  of  $0.5\triangle_{\mathrm{max}}$  for the all experiments.
- We use threshold  $\delta$  of 0.1 to determine whether the agent has reached the targeted milestone.
G Visualization of the learned latent goal space
In this section, we discuss about the ability of DTAMP to learn a latent goal space that captures the dynamics of an environment. We train the encoder along with the actor and critic, through goal-conditioned imitation learning. By doing so, we make the encoder capture useful information from states, to let the actor and critic distinguish the states and actions that can lead an agent to the goal. There are also similar methods that have been studied through prior work, such as learning forward or inverse dynamics model in a latent space [40, 12] or using contrastive learning based on state occupancy measures [7]. Figure 6 visualizes an example of latent goal space learned by DTAMP
Figure 6: The figure shows t-SNE [43] embedding of the learned latent space. Each data point represents a state in kitchen-mixed-v0 data. The black x's indicate milestones planned by DTAMP, and the black triangles indicate three states sampled from an arbitrary trajectory. Each image was created by visualizing the data point closest to the corresponding milestone. The colors represent state-values estimated using IQL.
using kitchen-mixed-v0 data. The figure shows that the trained encoder can capture the dynamics of environment, and also let the milestone planner successfully predict milestones in the learned latent space.
H.1 D4RL locomotion tasks
In this experiment, we aim to verify that DTAMP can also deal with the general reinforcement learning problems in which the objectives of the tasks are specified by the reward functions. To this end, we evaluate DTAMP on D4RL locomotion tasks (Halfcheetah, Hopper, and Walker2d). In order to let the milestone planner to predict a trajectory that maximizes the sum of rewards, we made the classifier-free guidance method to be conditioned on sum of rewards as done in Ajay et al. [1]. The comparative result shown in Table 6 shows that DTAMP achieves a marginally higher average score compared to the baselines, which indicates that the proposed method also can be used for general reinforcement learning problems. We would like to note that the D4RL locomotion tasks provide dense rewards and can be performed by planning relatively short trajectories compared to antmaze tasks (to predict more than 100 timesteps forward does not affect much on the performance on the locomotion tasks). This explains why DTAMP does not show a more significant performance improvement over Diffuser and DD in these tasks.
H.2 Robustness against environment stochasticity
In order to demonstrate the robustness against environment stochasticity, we further evaluate DTAMP on maze2d-umaze-v1 environment of the D4RL benchmark, while adding stochasticity to the
Environment	CQL	IQL	DT	TT	Diffuser	DD	DTAMP
halfcheetah-medium	42.6	47.4	42.6	46.9	44.2	49.1	47.3
halfcheetah-medium-replay	36.6	44.2	36.6	41.9	42.2	39.3	42.6
halfcheetah-medium-expert	91.6	86.7	86.8	95.0	79.8	90.6	88.2
hopper-medium	52.9	66.3	67.6	61.1	58.5	79.3	80.7
hopper-medium-replay	18.1	94.7	82.7	91.5	96.8	100.0	100.0
hopper-medium-expert	52.5	91.5	107.6	110.0	107.2	111.8	109.4
walker2d-medium	75.3	78.3	74.0	79.0	79.7	82.5	82.7
walker2d-medium-replay	26.0	73.9	66.6	82.6	61.2	75.0	79.5
walker2d-medium-expert	107.5	109.6	108.1	101.9	108.4	108.8	108.2
Average	77.6	77.0	74.7	78.9	75.3	81.8	82.1
Table 6: The table compares the performance of DTAMP evaluated on D4RL locomotion tasks with baselines. The result shows that DTAMP achieves the highest average score, which indicates that the proposed method is also able to handle general reinforcement learning problems as well.
p	Diffuser	DD	DTAMP	Ref. max score
0.0	61.9	92.6	97.6	80.5
0.3	29.6	97.0	99.0	72.3
0.5	18.5	70.5	99.4	64.3
Table 7: Normalized average score with different levels of stochasticity.
transition model. We also compare DTAMP against Diffuser [21], which predicts future states and actions using a diffusion model, and Decision Diffuser (DD) [1], which utilizes an inverse dynamics model to predict actions from the states planned by a diffusion model. In this experiment we let Diffuser and DD predict the whole trajectory only once at the beginning of each rollout as done in DTAMP.
Setting We create three different levels of stochasticity by choice of  $p \in \{0, 0.3, 0.5\}$ . The environment either executes a random action  $a \sim \mathrm{Unif}(\mathcal{A})$  with probability  $p$  or executes the action given by the agent with probability  $(1 - p)$ . Three sets of offline data were collected separately according to the choice of  $p$ , and each set of data consists of one million transitions collected by a waypoint controller. We also note that the control frequency is reduced threefold by repeating the same action three times in this experiment. By doing so, we reduce the time horizon of the environment (300 timesteps to 100 timesteps) to let Diffuser and DD predict the whole trajectory only once at the beginning of each rollout.
Result The normalized score of each model is shown in Table 7. The column named Ref. max score (reference max score) shows the average score of the waypoint controller, which represents the performance of an optimal agent. We note that even if an agent predicts optimal actions, stochasticity of environment causes inherent performance degradation. In order to neglect this effect and only compare the robustness of each model, the scores shown in Table 7 are normalized respect to the reference max scores (i.e., normalized score  $= 100 \times \frac{\text{average score}}{\text{ref. max score}}$ ).
The result shows that DTAMP is robust against the environment stochasticity and does not show performance degradation in the terms of normalized score. On the other hand, Diffuser shows greater performance degradation as stochasticity increased. DD shows no performance degradation when the stochasticity is small ( $p = 0.3$ ) while showing performance degradation when the stochasticity is large ( $p = 0.5$ ). The result indicates that our approach using a goal-conditioned actor makes DTAMP robust against environment stochasticity and allows to plan milestones only once for each rollout, while the other diffusion based planners do not.