SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Table 12 we present the main set of results at 5M time steps. To measure early learning performance, in Table 13 we present the performance of using immediate checkpoints at  $300\mathrm{k}$  time steps. To observe stability benefits, in Figure 30 we display the performance of five individual seeds, with and without checkpoints.
Table 12: Average performance on the MuJoCo benchmark at 5M time steps.  $\pm$  captures a  $95\%$  confidence interval around the average performance. Results are over 10 seeds.
Algorithm	HalfCheetah	Hopper	Walker2d	Ant	Humanoid
TD7	18165 ± 255	4075 ± 225	7397 ± 454	10133 ± 966	10281 ± 588
TD7 (no checkpoints)	18328 ± 331	3851 ± 372	6519 ± 209	10388 ± 1024	9521 ± 820
Max 10 episodes	18257 ± 338	4208 ± 52	6856 ± 273	9890 ± 661	10689 ± 576
Max 50 episodes	17875 ± 192	4110 ± 83	7104 ± 559	8226 ± 743	10239 ± 273
Mean, fixed	18137 ± 417	4061 ± 160	7347 ± 459	9866 ± 804	9850 ± 497
Mean, variable	17943 ± 291	4217 ± 9	6845 ± 476	9614 ± 849	10784 ± 188
Immediate, 10 episodes	17767 ± 374	4218 ± 17	7416 ± 451	10090 ± 961	9573 ± 255
Immediate, 20 episodes	17900 ± 127	4078 ± 53	7696 ± 616	9576 ± 935	10219 ± 551
Immediate, 50 episodes	17619 ± 275	4033 ± 154	6712 ± 313	9716 ± 1091	9687 ± 584
Fixed, 10 episodes	18209 ± 332	4130 ± 56	6890 ± 410	10357 ± 893	9682 ± 1408
Fixed, 20 episodes	17925 ± 334	3777 ± 337	6882 ± 452	9716 ± 772	10471 ± 560
Fixed, 50 episodes	17807 ± 183	3759 ± 142	6800 ± 422	8881 ± 408	8830 ± 655
Table 13: Average performance on the MuJoCo benchmark at 300k time steps.  $\pm$  captures a  $95\%$  confidence interval around the average performance. Results are over 10 seeds.
Algorithm	HalfCheetah	Hopper	Walker2d	Ant	Humanoid
TD7	15031 ± 401	2948 ± 464	5379 ± 328	6171 ± 831	5332 ± 714
Immediate, 10 episodes	13778 ± 856	3306 ± 34	4595 ± 343	5977 ± 764	4660 ± 735
Immediate, 20 episodes	12289 ± 507	3052 ± 195	4518 ± 321	5504 ± 889	4531 ± 738
Immediate, 50 episodes	6658 ± 1309	2668 ± 247	2897 ± 816	3709 ± 1848	2060 ± 620
Figure 26: Maximum number of assessment episodes. Learning curves on the MuJoCo benchmark, varying the maximum number episodes that the policy is fixed for. Results are averaged over 10 seeds. The shaded area captures a  $95\%$  confidence interval around the average performance.
Figure 27: Checkpoint condition. Learning curves on the MuJoCo benchmark, using the mean performance of assessment episodes rather than the minimum. The number of assessment episodes is kept at 20. Results are averaged over 10 seeds. The shaded area captures a  $95\%$  confidence interval around the average performance.
Figure 28: Fixed assessment length. Learning curves on the MuJoCo benchmark, where there is no early termination of the assessment phase (i.e the number of assessment episodes is fixed), and the number of assessment episodes is varied. Results are averaged over 10 seeds. The shaded area captures a  $95\%$  confidence interval around the average performance.
Figure 29: Immediate. Learning curves on the MuJoCo benchmark, where the maximum number of assessment episodes does not start at 1 (i.e. there is no early learning phase) and is varied. Results are averaged over 10 seeds. The shaded area captures a  $95\%$  confidence interval around the average performance.
Figure 30: Performance of individual seeds with and without checkpoints. Learning curves of five individual seeds, with and without checkpoints, on the MuJoCo benchmark. The shaded area captures a  $95\%$  confidence interval around the average performance.
G Ablation Study
In this section we perform an ablation study on the components of TD7.
G.1 Main Components
(Figure 31) Ablation 1: No SALE. SALE is entirely removed from TD7. This means the input to the value function and policy return to the original  $Q(s, a)$  and  $\pi(s)$ , respectively. TD7 keeps LAP, policy checkpoints, and any implementation differences over TD3 (discussed in Section A.1).
(Figure 31) Ablation 2: No checkpoints. Similar to TD3 and other off-policy deep RL algorithms, TD7 is trained at every time step, rather than after a batch of episodes. The current policy is used at test time.
(Figure 31) Ablation 3: No LAP. TD7 uses the standard replay buffer where transitions are sampled with uniform probability. The value function loss (Equation 15) uses the mean-squared error (MSE) rather than the Huber loss.
Table 14: Average performance on the MuJoCo benchmark at 5M time steps.  $\pm$  captures a  $95\%$  confidence interval around the average performance. Results are over 10 seeds.
Algorithm	HalfCheetah	Hopper	Walker2d	Ant	Humanoid
TD7	18165 ± 255	4075 ± 225	7397 ± 454	10133 ± 966	10281 ± 588
TD3	14337 ± 1491	3682 ± 83	5078 ± 343	5589 ± 758	5433 ± 245
No SALE	17099 ± 335	4018 ± 170	6418 ± 261	7861 ± 253	7275 ± 608
No checkpoints	18328 ± 331	3851 ± 372	6519 ± 209	10388 ± 1024	9521 ± 820
No LAP	18104 ± 315	4188 ± 22	7233 ± 251	6940 ± 1044	10155 ± 522
Figure 31: Main ablation. Learning curves on the MuJoCo benchmark, removing the main components of TD7. Results are averaged over 10 seeds. The shaded area captures a  $95\%$  confidence interval around the average performance.
G.2 Subcomponents
(Figure 31) Ablation 4: No SALE, with encoder. The encoder is kept but trained end-to-end with the value function, rather than via the SALE loss function. This means the encoder can be simply treated as an extension of the architecture of the value function. Since the embeddings trained with the value function may be conflicting with the policy, the policy does not use the state embedding, and only takes its original input  $\pi(s)$ .
(Figure 31) Ablation 5: TD3 with encoder. We add the encoder from SALE to TD3 but train it end-to-end with the value function. The encoder is not applied to the policy.
(Figure 33) Ablation 6: Current policy. TD7 is trained in an identical fashion, but uses the current policy at test time, rather than the checkpoint policy.
(Figure 33) Ablation 7: TD3 with checkpoints. We add policy checkpoints to TD3, using the checkpoint policy at test time.
(Figure 34) Ablation 8: TD3 with LAP. We add LAP to TD3 (identical to the TD3 + LAP [Fujimoto et al., 2020]).
(Figure 35) Ablation 9: No clipping. We remove our proposed value clipping for mitigating extrapolation error. This makes the value function loss (originally Equation 15):
$$
\mathcal {L} \left(Q _ {t + 1}\right) := \operatorname {H u b e r} \left(\operatorname {t a r g e t} - Q _ {t + 1} \left(z _ {t} ^ {s a}, z _ {t} ^ {s}, s, a\right)\right), \tag {74}
$$
$$
\text {t a r g e t} := r + \gamma \min  \left(Q _ {t, 1} (x), Q _ {t, 2} (x)\right), \tag {75}
$$
$$
x := \left[ z _ {t - 1} ^ {s ^ {\prime} a ^ {\prime}}, z _ {t - 1} ^ {s ^ {\prime}}, s ^ {\prime}, a ^ {\prime} \right], \tag {76}
$$
$$
a ^ {\prime} := \pi_ {t} \left(z _ {t - 1} ^ {s ^ {\prime}}, s ^ {\prime}\right) + \epsilon , \tag {77}
$$
$$
\epsilon \sim \operatorname {c l i p} \left(\mathcal {N} \left(0, \sigma^ {2}\right), - c, c\right). \tag {78}
$$
(Figure 35) Ablation 10: TD3 with clipping. We add value clipping to the loss function of TD3.
(Figure 36) Ablation 11: No normalization. We remove the use of AvgL1Norm from TD7. This is identical to no normalization in the design study in Section D.3, except this version includes policy checkpoints (the design study does not use checkpoints).
(Figure 36) Ablation 12: No fixed encoder. We remove the fixed embeddings from the value function and the policy, this means the networks use embeddings from the current encoder, rather than the fixed encoder from the previous iteration. This makes the input:  $Q_{t + 1}(z_{t + 1}^{sa}, z_{t + 1}^s, s, a)$  and  $\pi_{t + 1}(z_{t + 1}^s, s)$ . This is identical to no normalization in the design study in Section D.3, except this version includes policy checkpoints (the design study does not use checkpoints).
(Figure 37) Ablation 13: No implementation. We remove the implementation details differences discussed in Section A.1, namely using both value functions when updating the policy, ELU activation functions in the value function network. The target network update is not changed as this may influence the use of fixed encoders.
(Figure 37) Ablation 14: Our TD3. TD3 with the implementation detail differences discussed in Section A.1, namely using both value functions when updating the policy, ELU activation functions in the value function network, and updating the target network every 250 time steps, rather than using an exponential moving average.
Table 15: Average performance on the MuJoCo benchmark at 1M time steps.  $\pm$  captures a  $95\%$  confidence interval around the average performance. Results are over 10 seeds.
Algorithm	HalfCheetah	Hopper	Walker2d	Ant	Humanoid
TD7	17434 ± 155	3512 ± 315	6097 ± 570	8509 ± 422	7429 ± 153
TD3	10574 ± 897	3226 ± 315	3946 ± 292	3942 ± 1030	5165 ± 145
No SALE	12981 ± 261	3536 ± 65	5237 ± 376	5296 ± 1336	6263 ± 289
No SALE, with encoder	15639 ± 548	3544 ± 371	5350 ± 427	6003 ± 920	5480 ± 83
TD3 with encoder	12495 ± 813	1750 ± 302	4226 ± 491	5255 ± 579	5082 ± 317
No checkpoints	17123 ± 296	3361 ± 429	5718 ± 308	8605 ± 1008	7381 ± 172
Current policy	17420 ± 273	2940 ± 636	5765 ± 800	8748 ± 397	7162 ± 274
TD3 with checkpoints	10255 ± 656	3414 ± 77	3266 ± 474	3843 ± 749	5349 ± 72
No LAP	17347 ± 207	3697 ± 144	6382 ± 339	6571 ± 1504	8082 ± 260
TD3 with LAP	10324 ± 1159	3117 ± 554	4127 ± 330	4310 ± 1150	5090 ± 190
No clipping	17378 ± 100	3762 ± 118	6198 ± 289	7695 ± 497	7251 ± 274
TD3 with clipping	10283 ± 422	2969 ± 682	3990 ± 258	3711 ± 799	5254 ± 203
No normalization	17391 ± 275	3640 ± 95	6256 ± 317	7807 ± 266	4829 ± 1809
No fixed encoder	17145 ± 138	3710 ± 120	5869 ± 531	8287 ± 379	7412 ± 337
No implementation	17334 ± 99	3750 ± 136	5819 ± 71	7756 ± 704	7042 ± 259
Our TD3	11068 ± 1399	2791 ± 632	4179 ± 297	5489 ± 448	5186 ± 108
Figure 32: SALE. Learning curves on the MuJoCo benchmark, varying the usage of SALE. Results are averaged over 10 seeds. The shaded area captures a  $95\%$  confidence interval around the average performance.
Figure 33: Checkpoints. Learning curves on the MuJoCo benchmark, varying the usage of policy checkpoints. Results are averaged over 10 seeds. The shaded area captures a  $95\%$  confidence interval around the average performance.
Figure 34: LAP. Learning curves on the MuJoCo benchmark, varying the usage of LAP. Results are averaged over 10 seeds. The shaded area captures a  $95\%$  confidence interval around the average performance.
Figure 35: Clipping. Learning curves on the MuJoCo benchmark, varying the usage of our proposed value clipping for mitigating extrapolation error. Results are averaged over 10 seeds. The shaded area captures a  $95\%$  confidence interval around the average performance.
Figure 36: SALE components. Learning curves on the MuJoCo benchmark, removing components of SALE. Results are averaged over 10 seeds. The shaded area captures a  $95\%$  confidence interval around the average performance.
Figure 37: Implementation differences. Learning curves on the MuJoCo benchmark, varying the minor implementation details between TD3 and TD7. Results are averaged over 10 seeds. The shaded area captures a  $95\%$  confidence interval around the average performance.
H Offline RL Learning Curves
Figure 38: Learning curves on the offline D4RL benchmark. Results are averaged over 10 seeds. The shaded area captures a  $95\%$  confidence interval around the average performance.
I Run time
We benchmark the run time of TD7 and the baseline algorithms. Evaluation episodes were not included in this analysis. Each algorithm is trained using the same deep learning framework, PyTorch [Paszke et al., 2019]. All experiments are run on a single Nvidia Titan X GPU and Intel Core i7-7700k CPU. Software is detailed in Appendix B. We display run time adjusted learning curves in Figure 39.
Figure 39: Run time adjusted curves. Learning curves on the MuJoCo benchmark over 5M time steps, where the x-axis is run time. Results are averaged over 10 seeds. The shaded area captures a  $95\%$  confidence interval around the average performance.
Depth vs. Breadth. Although our empirical study is extensive, the scope of our experimentation emphasizes depth over breadth. There remains many other interesting benchmarks for continuous control, such as examining a setting with image-based observations.
Baselines. We benchmark TD7 against the highest performing agents in both the online and offline setting. However, due to the fact that most representation learning methods do not cover low level states, we only benchmark against a single other method focused on representation learning. While it would be surprising if a method, originally designed for a different setting, outperformed the strongest existing baselines, exploring additional methods could provide new insight into representation learning for low level states.
Computational cost. While TD7 has a lower computational cost than other competing methods, the run time over TD3 (the base algorithm) is more than double.
Theoretical results. We perform extensive empirical analysis to uncover which factors of dynamics-based representation learning are important for performance. However, we do not address the theoretical side of representation learning. There are many important questions regarding why dynamics-based representation learning is effective, or what makes a good representation.