SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our main theoretical results in Section 5 depend on the model error defined in (4.4), which, however, cannot directly serve as the model training objective. For this reason, we evaluate different model learners: single- and multi-step ( $h$ -step) state prediction models, as well as multi-step predictive models integrated with the directional derivative error [33]. The results are reported in Figure 10. We observe that enlarging the prediction steps benefits training. The algorithm also converges faster in walker2d when considering derivative error, which approximately minimizes (4.4) and supports our analysis. However, calculating the directional derivative error by searching  $k$  nearest points in the buffer significantly increases the computational cost, for which reason we use  $h$ -step state predictive models as default in experiments.
Figure 10: Ablation on different model learners: single-step and multi-step state prediction models, and multi-step state prediction models trained with an additional directional derivative error.
C.5 Figures in the Main Text in Larger Sizes
Here, we provide identical figures that are larger in size. Figure 11, 12, 13, 14 correspond to Figure 1, 4, 6, 7 in the main text, respectively.
Figure 11: Comparisons between RP PGMs (the green labels) and MF/MB baselines (the black labels) in the MuJoCo [57] tasks.
Figure 12: Performance of vanilla and SN-based MB RP PGMs with varying  $h$ . The vanilla method only works with a small  $h$  and fails when  $h$  increases, while the SN-based method enables a larger  $h$ .
Figure 13: Gradient variance of RP PGMs. The variance is significantly lower with SN when  $h$  is large.
Figure 14: Performance and gradient bias in differentiable simulation. The last column is the full training curves of APG, which need 20 times more steps than RP-DP-SN to reach a comparable return in the hopper task and fail in the half-cheetah task, respectively.