SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 10: Hyperparameters used for training the gap predictor. Values that are within brackets are separately tuned through a grid search.
Hyperparameter	Maze2D	Locomotion	Block Stacking
# synthetic data	500000	500000	500000
Observation normalization	Yes	Yes	Yes
Gap predictor learning rate	0.0002	0.0002	{0.00002, 0.0002, 0.001}
Gap predictor batch size	32	32	{32, 64, 128, 256}
Gap predictor train steps	2000000	2000000	2000000
tfor perturbation magnitude	0.9	0.9	0.9
D.2 Implementation of Restoration Gap Guidance
In this section, we document hyperparameters employed for restoration gap guidance. We adopt the hyperparameters from Diffuser (Janner et al., 2022) for determining the planning horizon and diffusion steps. Furthermore, the values of  $\alpha$ ,  $\beta$ , and  $\lambda$  are determined through a grid search. The hyperparameters utilized for the Maze2D experiments are presented in Table 11, while those for the Locomotion experiments are summarized in Table 12, 13, and 14. The hyperparameters for the Block Stacking experiments are provided in Table 15.
The planning performance exhibits a clear trend based on the choices of  $\alpha$ ,  $\beta$ , and  $\lambda$  parameters. This allows us to perform a grid search using the relatively minimal number of evaluation episodes. As depicted in Figure 3, the planning performance initially increases with rising values of  $\lambda$  but begins to decline when the values become excessively large. Specifically, we conduct 10, 15, and 10 evaluation episodes for the Maze2D, Locomotion, and Block Stacking experiments, respectively.
Table 11: Specific hyperparameters for Maze2D experiments. Values that are within brackets are separately tuned through a grid search.
Hyperparameter	Large	Medium	UMaze
Planning horizon	384	256	128
Diffusion steps	256	256	64
# samples for MC estimate of restoration gap	10	10	10
α for scaling the overall guidance	{0.05, 0.1}	{0.05, 0.1}	{0.05, 0.1}
β for scaling restoration gap guidance	1.0	1.0	1.0
λ for scaling attribution map regularization	{0.1, 1.0, 3.0}	{0.1, 1.0, 3.0}	{0.1, 1.0, 3.0}
Table 12: Specific hyperparameters for HalfCheetah experiments. Values that are within brackets are separately tuned through a grid search.
Hyperparameter	Med-Expert	Medium	Med-Replay
Planning horizon	4	32	32
Diffusion steps	20	20	20
# samples for MC estimate of restoration gap	64	64	64
α for scaling the overall guidance	{0.01, 0.1}	{0.01, 0.1}	{0.01, 0.1}
β for scaling restoration gap guidance	{0.1, 1.0, 10.0}	{0.1, 1.0, 10.0}	{0.1, 1.0, 10.0}
λ for scaling attribution map regularization	{0.001, 0.01, 0.1, 1.0}	{0.001, 0.01, 0.1, 1.0}	{0.001, 0.01, 0.1, 1.0}
Table 13: Specific hyperparameters for Hopper experiments. Values that are within brackets are separately tuned through a grid search.
Hyperparameter	Med-Expert	Medium	Med-Replay
Planning horizon	32	32	32
Diffusion steps	20	20	20
# samples for MC estimate of restoration gap	64	64	64
α for scaling the overall guidance	{0.01, 0.1}	{0.01, 0.1}	{0.01, 0.1}
β for scaling restoration gap guidance	{0.1, 1.0, 10.0}	{0.1, 1.0, 10.0}	{0.1, 1.0, 10.0}
λ for scaling attribution map regularization	{0.001, 0.01, 0.1, 1.0}	{0.001, 0.01, 0.1, 1.0}	{0.001, 0.01, 0.1, 1.0}
Table 14: Specific hyperparameters for Walker2d experiments. Values that are within brackets are separately tuned through a grid search.
Hyperparameter	Med-Expert	Medium	Med-Replay
Planning horizon	32	32	32
Diffusion steps	20	20	20
# samples for MC estimate of restoration gap	64	64	64
α for scaling the overall guidance	{0.01, 0.1}	{0.01, 0.1}	{0.01, 0.1}
β for scaling restoration gap guidance	{0.1, 1.0, 10.0}	{0.1, 1.0, 10.0}	{0.1, 1.0, 10.0}
λ for scaling attribution map regularization	{0.001, 0.01, 0.1, 1.0}	{0.001, 0.01, 0.1, 1.0}	{0.001, 0.01, 0.1, 1.0}
Table 15: Specific hyperparameters for Block Stacking experiments. Values that are within brackets are separately tuned through a grid search.
Hyperparameter	Unconditional Stacking	Conditional Stacking
Planning horizon	128	128
Diffusion steps	1000	1000
# samples for MC estimate of restoration gap	5	5
α for guide scale	{0.01, 0.02, 0.05, 0.1, 0.2, 0.5}	{0.1, 0.3, 0.5, 0.7}
β for scaling restoration gap guidance	1.0	{0.01, 0.02, 0.04, 0.07}
λ for scaling attribution map regularization	{0.1, 0.3, 0.5, 1.0, 3.0, 5.0}	{0.001, 0.003, 0.005, 0.008}
D.3 Implementation of Attribution Map Regularizer
For the Maze2D experiments and the Block Stacking experiments, we adopt Grad-CAM (Selvaraju et al., 2017), while we employ DeepLIFT (Shrikumar et al., 2017) for the Locomotion experiments, as the attribution method. The rationale for the choice of each method is that both Grad-CAM and DeepLIFT offer simplicity and efficiency in computation. Specifically, we apply the DeepLIFT for
the Locomotion experiments due to their relatively smaller planning horizon compared to the other tasks. This is because Grad-CAM compresses saliency information into a single value. However, it is worth noting that our proposed method has shown robustness across different input attribution methods, as demonstrated in Table 4.
D.4 The Amount of Computation
Our proposed method, refining diffusion planner, requires training the gap predictor which estimates the restoration gap. For this purpose, we generate synthetic data of 500,000 plans generated by Diffuser. The generation process takes approximately 30 to 50 hours, depending on the situation, on a single NVIDIA Quadro 8000 GPU. The training time of the gap predictor on the same GPU can range from 5 to 8 hours.
Appendix E Baseline Performance Sources
E.1 Maze2D Tasks
The scores for CQL are taken from Table 2 in Fu et al. (2020). The scores for MPPI and IQL are taken from Table 1 in Janner et al. (2022).
E.2 Locomotion Tasks
The scores for BC, CQL, and IQL are found in Table 1 of Kostrikov et al. (2022), while DT scores are taken from Table 2 in Chen et al. (2021), TT from Table 1 in Janner et al. (2021), MOPO from Table 1 in Yu et al. (2020), MOReL from Table 2 in Kidambi et al. (2020), MBOP from Table 1 in Argenson & Dulac-Arnold (2021), and Diffuser from Table 2 in Janner et al. (2022).
E.3 Block Stacking Tasks
The scores corresponding to BCQ and CQL are obtained from Table 3 of Janner et al. (2022). To obtain the scores for Diffuser, the official implementation and model provided by the authors are used, which can be found at https://github.com/jannerm/diffuser.