SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$\mathrm{DrQv2 + SE}$  on walker walk with varying  $\beta$  We provide additional experimental results with varying  $\beta \in \{0.1, 0.01, 0.001\}$  on Walker Walk task where  $\mathrm{DrQv2 + SE}$  significantly struggles to improve the sample-efficiency of  $\mathrm{DrQv2}$ . In Figure 15, we find that the performance of  $\mathrm{DrQv2 + SE}$  is consistently worse than the vanilla  $\mathrm{DrQv2}$  with different  $\beta$  values. This implies that adding SE intrinsic reward can be sometimes harmful for performance by making it difficult for the agent to exploit the task reward.
Figure 15: Learning curves as measured on the episode return. The solid line and shaded regions represent the interquartile mean and standard deviation, respectively, across eight runs.
A2C+VCSE with varying  $\beta$  We also report the experimental results of A2C+VCSE with varying  $\beta$  in Figure 16. We find that A2C+VCSE consistently improves A2C, in contrast to A2C+SE which fails to significantly outperform A2C even with different  $\beta$  values (see Figure 4).
Figure 16: Learning curves as measured on the success rate. The solid line and shaded regions represent the interquartile mean and standard deviation, respectively, across eight runs.
DrQv2+SE with decaying  $\beta$  We conduct additional experiments that compare DrQv2+VCSE with the state entropy baseline that uses decaying schedule for  $\beta$ , similarly to Seo et al. (2021) that uses  $\beta$  schedule for the intrinsic reward. In Figure 17, we find that such a schedule cannot significantly
improve the performance of SE, except Hopper Stand where the performance is stabilized. Moreover, we find that the decaying schedule sometimes could degrade the performance, i.e., Walker Walk Sparse. We also note that designing such a decaying schedule is a tedious process that requires researchers to tune the performance, making it less desirable even if it works reasonably well. We indeed find that the performance becomes very sensitive to the magnitude of decaying schedule. On the other hand,  $\mathrm{DrQv2 + VCSE}$  exhibits consistent performance without the decaying schedule, which highlights the benefit of our approach that maximizes the value-conditional state entropy.
Figure 17: Learning curves on six visual locomotion control tasks from DeepMind Control Suite (Tassa et al., 2020) as measured on the episode return. The solid line and shaded regions represent the interquartile mean and standard deviation, respectively, across eight runs.
C Experiments with Ground-Truth States
C.1 MiniGrid Experiments
To demonstrate that our method also works in fully-observable MiniGrid where we do not use state encoder for computing the intrinsic bonus, we provide additional experiments that use fully observable states instead of partially observable grid encoding (see Section 5.1). Specifically, we use a set of one-hot vectors that represents a current map as inputs to the agent, and use the location of agent as inputs for computing the intrinsic bonus. Figure 18 shows that VCSE consistently accelerates the training, which highlights the applicability of VCSE to diverse domains with different input types.
Figure 18: Learning curves on four navigation tasks from fully-observable MiniGrid (Chevalier-Boisvert et al., 2018) as measured on the success rate. The solid line and shaded regions represent the interquartile mean and standard deviation, respectively, across 16 runs.
Due to the unstable performance from introducing the decay schedule, we run experiments with multiple decaying schedules and report the best performance for each task.
C.2 DeepMind Control Suite Experiments
We further report experimental results in state-based DeepMind Control Suite experiments where we do not use state encoder for computing the intrinsic bonus. To make the scale of state norms be consistent across diverse tasks with different state dimensions, we divide the state input with its state dimension before computing the intrinsic bonus. As our underlying RL algorithm, we used Soft Actor-Critic (SAC; Haarnoja et al. 2018). For SE and VCSE, we disabled automatic tuning hyperparameter  $\alpha$  and used lower value of  $\alpha = 0.001$  in SAC, because we find that such an automatic tuning conflicts with introducing the intrinsic reward, similar to noise scheduling in DrQv2. Figure 19 shows that VCSE consistently accelerates the training, which highlights the applicability of VCSE to diverse domains with different input types.
Figure 19: Learning curves on four locomotion tasks from state-based DeepMind Control Suite (Tassa et al., 2020) as measured on the success rate. The solid line and shaded regions represent the interquartile mean and standard deviation, respectively, across 16 runs.
D Additional Illustrations
We provide the additional illustration that helps understanding the procedure of estimating the conditional entropy with KSG estimator, which is explained in Section 3.2.
Figure 20: Illustration of a procedure for computing  $\epsilon_{x}(i)$  and  $n_x(i)$  when using the KSG estimator with  $k = 2$ . Given a centered point  $z_{i}$ , we first find  $k$ -nearest neighbor state  $z_{i}^{k\mathrm{NN}}$ . Then  $\epsilon_{x}(i)$  is twice the distance from  $x_{i}$  to  $x_{i}^{k\mathrm{NN}}$  and  $n_x$  can be computed as 5 by counting all the points located within  $(x_{i} - \epsilon_{x}(i) / 2, x_{i} + \epsilon_{x}(i) / 2)$ . We note that  $\epsilon_{y}(i)$  and  $n_y(i)$  can be also similarly computed.