SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Therefore, the majority of training time is from reinforcement learning with an update-to-data ratio (UTD) of 20. We expect the diffusion training may be heavily sped-up with early stopping, and leave this to future work. The default SAC algorithm with UTD=1 takes  $\approx 2$  hours.
E Further Offline Results
In this section, we include additional supplementary offline experiments to those presented in Section 4.1.
E.1 AntMaze Data Generation
We further verify that SYNTHER can generate synthetic data for more complex environments such as AntMaze [21]. This environment replaces the 2D ball from Maze2D with the more complex 8-DoF "Ant" quadruped robot, and features: non-Markovian policies, sparse rewards, and multitask data. In Table 11, we see that SYNTHER improves the TD3+BC algorithm where it trains (on the 'umaze' dataset) and achieves parity otherwise.
Table 11: We show synthetic data from SYNTHER achieves at least parity for more complex offline environments like AntMaze-v2, evaluated with the TD3+BC algorithm. We show the mean and standard deviation of the final performance averaged over 6 seeds.
Environment	TD3+BC [22]
Original	SynthER
AntMaze	umaze	70.8±39.2	88.9±4.4
medium-play	0.3±0.4	0.5±0.7
large-play	0.0±0.0	0.0±0.0
E.2 Offline Data Mixtures
In Section 4.1, we considered exclusively training on synthetic data. We present results in Table 12 with a 50-50 mix of real and synthetic data to confirm that the two are compatible with each other, similar to Section 4.2. We do so by including as many synthetic samples as there are real data. As we stated before, we do not expect an increase in performance here due to the fact that most D4RL datasets are at least 1M in size and are already sufficiently large.
Table 12: We verify that the synthetic data from SYNTHER can be mixed with the real data for offline evaluation. The 50-50 mix achieves parity with the original data, same as the synthetic data. We show the mean and standard deviation of the final performance averaged over 8 seeds.
Environment	TD3+BC [22]	IQL [41]
Original	SYNTHER	50-50	Original	SYNTHER	50-50
locomotion average	59.0±4.9	60.0±5.1	59.2±3.7	62.1±3.5	63.7±3.5	62.7±5.1
F Latent Data Generation with V-D4RL
We provide full details for the experiments in Section 4.3 that scale SYNTHER to pixel-based environments by generating data in latent space for the  $\mathrm{DrQ + BC}$  [50] and BC algorithms. Concretely, for the  $\mathrm{DrQ + BC}$  algorithm, we consider parametric networks for the shared CNN encoder, policy, and  $Q$ -functions,  $f_{\xi}$ ,  $\pi_{\phi}$ , and  $Q_{\theta}$  respectively. We also use a random shifts image augmentation, aug. Therefore, the  $Q$ -value for a state  $s$  and action  $a$  is given by  $Q_{\theta}(f_{\xi}(\mathrm{aug}(s)), a)$ . The policy is similarly conditioned on an encoding of an augmented image observation.
The policy and  $Q$ -functions both consist of an initial 'trunk' which further reduces the dimensionality of the CNN encoding to  $d_{\mathrm{feature}} = 50$ , followed by fully connected layers. We represent this as  $\pi_{\phi} = \pi_{\phi}^{\mathrm{fc}} \circ \pi_{\phi}^{\mathrm{trunk}}$  and  $Q_{\theta} = Q_{\theta}^{\mathrm{fc}} \circ Q_{\theta}^{\mathrm{trunk}}$ . This allows us to reduce a pixel-based transition to a low-dimensional latent version. Consider a pixel-based transition  $(s, a, r, s')$  where  $s, s' \in \mathbb{R}^{84 \times 84 \times 3}$ . Let  $h = f_{\xi}(\mathrm{aug}(s))$  and  $h' = f_{\xi}(\mathrm{aug}(s'))$ . The latent transition we generate is:
$$
\left(\pi_ {\phi} ^ {\text {t r u n k}} (h), Q _ {\theta} ^ {\text {t r u n k}} (h), a, r, \pi_ {\phi} ^ {\text {t r u n k}} \left(h ^ {\prime}\right), Q _ {\theta} ^ {\text {t r u n k}} \left(h ^ {\prime}\right)\right)
$$
This has dimension  $4 \cdot d_{\text{feature}} + |a| + 1$  and includes specific supervised features for both the actor and the critic; we analyze this choice in Appendix F.1. For example, on the 'cheetah-run' environment considered in V-D4RL, since  $|a| = 6$ , the overall dimension is 207 which is suitable for our residual MLP denoising networks using the same hyperparameters in Table 8. This allows us to retain the fast training and sampling speed from the proprioceptive setting but now in pixel space.
To obtain a frozen encoder  $f_{\xi}$  and trunks  $\pi_{\phi}^{\mathrm{trunk}}$ ,  $Q_{\theta}^{\mathrm{trunk}}$ , we simply train in two stages. The first stage trains the original algorithm on the original data. The second stage then retransies only the fully-connected portions of the actor and critic,  $\pi_{\phi}^{\mathrm{fc}}$  and  $Q_{\theta}^{\mathrm{fc}}$ , with synthetic data. Thus, our approach could also be viewed as fine-tuning the heads of the networks. The procedure for the BC algorithm works the same but without the critic.
We use the official V-D4RL [50] codebase for the data and algorithms in this evaluation. Their code can be found at https://github.com/conglu1997/v-d4rl and is released under an MIT license.
F.1 Ablations On Representation
We analyze the choice of low-dimensional latent representation we use in the previous section, in particular, using specific supervised features for both the actor and critic. We compare this against using actor-only or critic-only features for both the actor and critic, which corresponds to a choice of  $\pi_{\phi}^{\mathrm{trunk}} = Q_{\theta}^{\mathrm{trunk}}$ , in Table 13. We note that both perform worse with an especially large drop-off for the critic-only features. This may suggest that non-specific options for compressing the image into low-dimensional latents, for example, using auto-encoders [40], could be even less suitable for this task.
Table 13: Ablations on the latent representation used for SYNTHER on the V-D4RL cheetah expert dataset. We observe that separate specific supervised features are essential for downstream performance with a particularly large decrease if we only used critic features for the actor and critic. We show the mean and standard deviation of the final performance averaged over 4 seeds.
Latent Representation	Eval. Return
Actor and Critic (Ours)	52.3±7.0
Actor Only	43.5±7.3
Critic Only	16.0±2.8