SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Neither the discriminative reward from STG nor the progression reward from ELE is bounded. Therefore, it is reasonable to adjust certain hyperparameters to bring out the best performance of each algorithm in each task. In Table 5, the coefficient of intrinsic reward  $\eta (\eta >0)$  for different baselines is tuned to balance the value scale and GAE  $\lambda (0 < \lambda < 1)$  is tuned to adjust the impact of intrinsic rewards in different tasks.
Table 5: Specific Hyperparameters for Different Tasks
Task	ηSTG	ηELE	ηGAlfO	λGAE
Breakout	0.6	1.0	2.0	0.1
Freeway	2.0	0.1	1.0	0.15
Qbert	5.0	0.05	2.0	0.95
Space Invaders	6.0	0.1	2.0	0.95
Milk a Cow	1.0	0.5	-	0.8
Gather Wool	10.0	0.1	-	0.8
Harvest Tallgrass	1.0	0.1	-	0.95
Pick a Flower	1.0	0.1	-	0.95
The coefficients of STG* (noted as  $\eta r^i + \nu r^*$ ) in four Atari tasks are reported in Table 6.
Table 6: Coefficients for STG* in Atari Tasks
Task	η	ν
Breakout	0.6	0.01
Freeway	2.0	0.1
Qbert	5.0	0.03
Space Invaders	6.0	0.01
Training Details. For Minecraft tasks, we adopt a hybrid approach utilizing both PPO [40] and Self-Imitation Learning (SIL) [48]. Specifically, we store trajectories with high intrinsic rewards in a buffer and alternate between PPO and SIL gradient steps during the training process. This approach allows us to leverage the unique strengths of both methods and achieve superior performance compared to utilizing either method alone [46].
Intrinsic Reward Design. In Equation (9), we define our intrinsic reward  $r^i$  as the difference between  $r^{guide}$  and  $r^{base}$ :
$$
r _ {t} ^ {i} = D _ {\omega} \left(E _ {\xi} (s _ {t}), E _ {\xi} (s _ {t + 1})\right) - D _ {\omega} \left(E _ {\xi} (s _ {t}), T _ {\sigma} \left(E _ {\xi} (s _ {t})\right)\right) = r _ {t} ^ {\text {q u i d e}} - r _ {t} ^ {\text {b a s e}}. \tag {10}
$$
On the one hand, pretrained  $D_{\omega}$  clearly provides informative judgment  $r^{guide}$  of transition quality during online interaction. On the other hand, the baseline reward  $r^{base}$ , solely relying on the current state  $s_t$ , serves as a baseline to normalize  $r^i$  to a relatively lower level. In this section, we aim to investigate the necessity of incorporating  $r^{base}$ .
To assess the significance of  $r^{base}$ , we conducted experiments on the four Atari tasks utilizing only  $r^{guide}$  as the intrinsic reward, which is similar to previous adversarial methods like GAIfo [3]. In order to bring the scale of  $r^{guide}$  in line with  $r^i$ , we employ running normalization and bound the values within the range of  $[-1, 1]$  to mitigate the negative influence of outliers. All other settings remain unchanged. We denote this ablation baseline as STG'.
As illustrated in Figure 4,  $r^{guide}$  yields comparable final performance in Breakout and Space Invaders while failing to achieve satisfactory performance in Freeway and Qbert. In contrast, by leveraging  $r^{base}$ , which provides a unique reference from expert datasets for each individual  $s_t$ , we observe reduced variance and improved numerical stability compared to the running normalization trick that calculates batch means for normalization.
(b) Freeway
Figure 4: Atari experiments comparing using  $r^{guide}$  (STG') and  $r^i$  (STG) as intrinsic reward.
F Additional Ablations
Multi-Task STG. Given the four times larger datasets, we enlarge the size of the STG Transformer by increasing the number of heads (24) and layers (16) within the multi-head causal self-attention modules, augmenting the model capacity by about four times. Here we further assess the efficacy of multi-task adaptation of the STG Transformer in Minecraft. We tune the intrinsic coefficient  $\eta$  to be 5 for Breakout, Qbert, SpaceInvaders, "milk a cow", "harvest tallgras", "pick a flower", and 10 for Freeway and "gather wool". STG-Multi results in Minecraft are illustrated in Figure 5. As all four tasks share a similar biome in MineDojo, STG-Multi may provide less clear guidance for the agent than STG in downstream tasks, which may cause their discrepancy in performance.
(a) Pick a flower
Figure 5: Multi-task STG (STG-Multi) is pretrained on the whole Minecraft datasets to guide RL training in downstream tasks..
G Additional Visualizations
Intrinsic Return. As no environmental rewards participate in updating policy, the ultimate objective of LfVO is to maximize the expectation of cumulative dense intrinsic rewards, namely intrinsic return. Figure 6 shows the learning curves of smoothed intrinsic return in Atari and Minecraft. The rising trend proves that online collected observation distribution is getting closer to expert observation distribution during training, indicating the effectiveness of the offline-pretrained Modules.
(b) Freeway
Figure 6: The ultimate objective of LfVO is to maximize the expectation of cumulative dense intrinsic rewards, namely intrinsic return. The rising trend proves that online collected observation distribution is getting closer to expert observation distribution.
Multi-Trajectories Visualization. As an extension to increase diversity, we additionally visualize more trajectories in SpaceInvaders in Figure 7. We randomly sample five trajectories in the expert dataset and use t-SNE to visualize the embedding sequences encoded by STG and STG- respectively. Expert trajectories of STG exhibit more continuity in adjacent states compared with STG-. This is consistent with the visualization results in Figure 6. Furthermore, it can be observed that after 10M-step of training, the observation distribution is getting closer to expert, and the patterns of STG are closer to exert in comparison with STG-. This reflects that our WGAN-style training indeed generates meaningful reward signal to imitate the behavior of experts and TDR module accelerate the process.
Figure 7: T-SNE visualization of embeddings of a random trajectory, a rollout trajectory and expert trajectories in SpaceInvaders.