SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dongyoung Kim KAIST
Jinwoo Shin KAIST
Pieter Abbeel  
UC Berkeley
Younggyo Seo* KAIST
A promising technique for exploration is to maximize the entropy of visited state distribution, i.e., state entropy, by encouraging uniform coverage of visited state space. While it has been effective for an unsupervised setup, it tends to struggle in a supervised setup with a task reward, where an agent prefers to visit high-value states to exploit the task reward. Such a preference can cause an imbalance between the distributions of high-value states and low-value states, which biases exploration towards low-value state regions as a result of the state entropy increasing when the distribution becomes more uniform. This issue is exacerbated when high-value states are narrowly distributed within the state space, making it difficult for the agent to complete the tasks. In this paper, we present a novel exploration technique that maximizes the value-conditional state entropy, which separately estimates the state entropies that are conditioned on the value estimates of each state, then maximizes their average. By only considering the visited states with similar value estimates for computing the intrinsic bonus, our method prevents the distribution of low-value states from affecting exploration around high-value states, and vice versa. We demonstrate that the proposed alternative to the state entropy baseline significantly accelerates various reinforcement learning algorithms across a variety of tasks within MiniGrid, DeepMind Control Suite, and Meta-World benchmarks. Source code is available at https://sites.google.com/view/rl-vcse.
Recent advances in exploration techniques have enabled us to train strong reinforcement learning (RL) agents with fewer environment interactions. Notable techniques include injecting noise into action or parameter spaces (Sehnke et al., 2010; Ruckstiess et al., 2010; Wawrzynski, 2015; Lillicrap et al., 2016; Fortunato et al., 2018) and using visitation counts (Thrun, 1992b; Bellemare et al., 2016; Sutton & Barto, 2018; Burda et al., 2019) or errors from predictive models (Stadie et al., 2015; Pathak et al., 2017, 2019; Sekar et al., 2020) as an intrinsic reward. In particular, a recently developed approach that maximizes the entropy of visited state distribution has emerged as a promising exploration technique (Hazan et al., 2019; Lee et al., 2019; Mutti et al., 2021) for unsupervised RL, where the agent aims to learn useful behaviors without any task reward (Liu & Abbeel, 2021a,b; Yarats et al., 2021b).
The idea to maximize the state entropy has also been utilized in a supervised setup where the task reward is available from environments to improve the sample-efficiency of RL algorithms (Tao et al., 2020; Seo et al., 2021; Nedergaard & Cook, 2022; Yuan et al., 2022). Notably, Seo et al. (2021) have shown that maximizing the sum of task reward and intrinsic reward based on a state entropy estimate can accelerate RL training. However, in this supervised setup, we point out that this approach often suffers from an imbalance between the distributions of high-value and low-value states, which occurs as an agent prefers to visit high-value states for exploiting the task reward. Because state entropy increases when the distribution becomes more uniform, low-value states get to receive a higher
*Now at Dyson Robot Learning Lab. Correspondence to younggyo.seo@dyson.com.
Figure 1: Illustration of our method. We randomly sample states from a replay buffer and compute the Euclidean norm in state and value spaces using pairs of samples within a minibatch. We then sort the samples based on their maximum norms. We find the  $k$ -th nearest neighbor among samples (e.g.,  $k = 3$  in the figure) and use the distance to it as an intrinsic reward. Namely, our method excludes the samples whose values significantly differ for computing the intrinsic reward. Then we train our RL agent to maximize the sum of the intrinsic reward and the extrinsic reward.
intrinsic bonus than high-value states, which biases exploration towards low-value states. This makes it difficult for the agent to explore the region around high-value states crucial for solving target tasks, which exacerbates when high-value states are narrowly distributed within the state space.
In this paper, we present a novel exploration technique that maximizes value-conditional state entropy which separately estimates the state entropies that are conditioned on the value estimates of each state, then maximizes their average. This intuitively can be seen as partitioning the state space with value estimates and maximizing the average of state entropies of partitioned spaces. Namely, our method avoids the problem of state entropy maximization by preventing the distribution of low-value states from affecting exploration around high-value states, and vice versa, by only considering the states with similar value estimates for computing the intrinsic bonus. For value-conditional state entropy estimation, we utilize the Kraskov-Stögbauer-Grassberger estimator (Kriskov et al., 2004) along with a value normalization scheme that makes value distribution consistent throughout training. We define our intrinsic reward as proportional to the value-conditional state entropy estimate and train RL agents to maximize the sum of task reward and intrinsic reward.
We summarize the main contributions of this paper:
- We present a novel exploration technique that maximizes value-conditional state entropy which addresses the issue of state entropy exploration in a supervised setup by taking into account the value estimates of visited states for computing the intrinsic bonus.
- We show that maximum value-conditional state entropy (VCSE) exploration successfully accelerates the training of RL algorithms (Mnih et al., 2016; Yarats et al., 2021a) on a variety of domains, such as MiniGrid (Chevalier-Boisvert et al., 2018), DeepMind Control Suite (Tassa et al., 2020), and Meta-World (Yu et al., 2020) benchmarks.
Exploration in RL Exploration has been actively studied to solve sparse reward tasks or avoid being stuck at local optima. One of the classical approaches to encourage exploration is  $\epsilon$ -greedy algorithm (Sutton & Barto, 2018). This idea has been extended to approaches that inject noise into the action space (Wawrzynski, 2015; Lillicrap et al., 2016) or parameter space (Sehnke et al., 2010; Ruckstiess et al., 2010; Fortunato et al., 2018; Plappert et al., 2018) and approaches that maximize the action entropy (Ziebart, 2010; Haarnoja et al., 2018). Similarly to our work that introduces an intrinsic reward, there have been approaches that use the errors from predictive models as an intrinsic reward (Schmidhuber, 1991; Oudeyer et al., 2007; Stadie et al., 2015; Pathak et al., 2017, 2019; Sekar et al., 2020; Badia et al., 2020). Instead of using the knowledge captured in the model, we use a metric that can be quantified from data. The idea of using the state visitation count as an intrinsic
reward (Thrun, 1992b; Bellemare et al., 2016; Tang et al., 2017; Ostrovski et al., 2017; Burda et al., 2019) is also related. However, our method differs in that we directly maximize the diversity of data. Our work also differs from a line of work that balances exploration and exploitation (Thrun, 1992a; Brafman & Tennenholtz, 2002; Tokic, 2010; Whitney et al., 2021; Schäfer et al., 2022) as we instead consider a situation where exploitation biases exploration towards specific state space regions.
Maximum state entropy exploration The approach most related to our work is to utilize state entropy as an intrinsic reward. A popular application of maximizing state entropy is unsupervised RL (Lee et al., 2019; Hazan et al., 2019; Mutti & Restelli, 2020; Mutti et al., 2021; Liu & Abbeel, 2021b; Yarats et al., 2021b; Zhang et al., 2021; Guo et al., 2021; Mutti et al., 2022a,b; Yang & Spaan, 2023), where the agent learns useful behaviors without task reward. We instead consider a setup where task reward is available. A potentially related approach is of Yang & Spaan (2023) which introduces safety constraint for exploration in that one can also consider the approach of introducing a value constraint for exploration. However, our work differs in that our motivation is not to discourage exploration on specific state regions. The work closest to ours is approaches that make agents maximize state entropy along with task rewards (Tao et al., 2020; Seo et al., 2021; Nedergaard & Cook, 2022; Yuan et al., 2022). We point out they often suffer from an imbalance between distributions of high-value and low-value states and propose to take into account the value estimates to address this issue.
Definition Let  $Z = (X, Y)$  be a random variable whose joint entropy is defined as  $\mathcal{H}(X, Y) = -E_{(x, y) \sim p(x, y)}[\log p(x, y)]$  and marginal entropy is defined as  $\mathcal{H}(X) = -E_{x \sim p_X(x)}[\log p_X(x)]$ . The conditional entropy is defined as  $\mathcal{H}(Y \mid X) = E_{(x, y) \sim p(x, y)}[\log p(y \mid x)]$ , which quantifies the amount of information required for describing the outcome of  $Y$  given that the value of  $X$  is known.
Reinforcement learning We formulate a problem as a Markov decision process (MDP; Sutton & Barto 2018), which is defined as a tuple  $(\mathcal{S}, \mathcal{A}, p, r^{\mathrm{e}}, \gamma)$ . Here,  $\mathcal{S}$  denotes the state space,  $\mathcal{A}$  denotes the action space,  $p(s_{t+1} | s_t, a_t)$  is the transition dynamics,  $r^{\mathrm{e}}$  is the extrinsic reward function  $r_t^{\mathrm{e}} = r^{\mathrm{e}}(s_t, a_t)$ , and  $\gamma \in [0,1)$ . Then we train an agent to maximize the expected return.
3.1 Maximum State Entropy Exploration
$k$ -nearest neighbor entropy estimator When we aim to estimate the entropy of  $X$  but the density  $p_X$  is not available, non-parametric entropy estimators based on  $k$ -nearest neighbor distance can be used. Specifically, given  $N$  i.i.d. samples  $\{x_i\}_{i=1}^N$ , the Kozachenko-Leonenko (KL) entropy estimator (Kozachenko & Leonenko, 1987; Singh et al., 2003; Kraskov et al., 2004) is defined as:
$$
\widehat {H} _ {\mathrm {K L}} (X) = - \psi (k) + \psi (N) + \log c _ {d _ {X}} + \frac {d _ {X}}{N} \sum_ {i = 1} ^ {N} \log D _ {x} (i), \tag {1}
$$
where  $\psi$  is the digamma function,  $D_{x}(i)$  is twice the distance from  $x_{i}$  to its  $k$ -th nearest neighbor,  $d_{X}$  is the dimensionality of  $X$ , and  $c_{d_X}$  is the volume of the  $d_{X}$ -dimensional unit ball.
State entropy as an intrinsic reward Let  $\mathcal{B}$  be a replay buffer and  $S$  be a random variable of dimension  $d_{s}$  with a probability density  $p_S(s) = \sum_{s\in \mathcal{B}}\mathbf{1}_{S = s} / |\mathcal{B}|$ . The main idea of maximum state entropy exploration is to encourage an agent to maximize the state entropy  $\mathcal{H}(S)$  (Liu & Abbeel, 2021b; Seo et al., 2021). The KL entropy estimator in Equation 1 is used to estimate the state entropy  $\widehat{H}_{\mathrm{KL}}(S)$ , and the intrinsic reward  $r_i^{\mathrm{SE}}$  is defined by ignoring constant terms and putting logarithm along with an additive constant 1 for numerical stability as below:
$$
\widehat {H} _ {\mathrm {K L}} (S) = - \psi (k) + \psi (N) + \log c _ {d _ {S}} + \frac {d _ {S}}{N} \sum_ {i = 1} ^ {N} \log D _ {s} (i) \quad r _ {i} ^ {\mathrm {S E}} = \log \left(D _ {s} (i) + 1\right) \tag {2}
$$
where  $D_{s}(i)$  is twice the distance from  $s_i$  to its  $k$ -nearest neighbor.
2For instance,  $c_{d_X} = \widehat{\pi}^{d_X / 2} / \Gamma (1 + d_X / 2)$  for Euclidean norm where  $\widehat{\pi}\approx 3.14159$  is a constant.
3.2 Conditional Entropy Estimation
Naive conditional entropy estimator Given  $N$  i.i.d. samples  $\{z_i\}_{i=1}^N$  where  $z_i = (x_i, y_i)$ , the conditional entropy can be estimated by using the chain rule of conditional entropy  $H(Y \mid X) = H(X, Y) - H(X)$ . Specifically, one can compute  $\widehat{H}_{\mathrm{KL}}(X, Y)$  and  $\widehat{H}_{\mathrm{KL}}(X)$  and use these estimates for computing  $\widehat{H}(Y \mid X)$ . But a major drawback of this naive approach is that it does not consider length scales in spaces spanned by  $X$  and  $Y$  can be very different, i.e., the scale of distances between samples and their  $k$ -nearest neighbors in each space could be very different.
KSG conditional entropy estimator To address the issue of estimating entropy in different spaces spanned by  $X$  and  $Y$ , we employ Kraskov-Stögbauer-Grassberger (KSG) estimator (Kraskov et al., 2004), which is originally designed for estimating mutual information. The main idea of KSG estimator is to use different  $k$ -values in the joint and marginal spaces for adjusting the different length scales across spaces. Given a sample  $z_{i}$  in the joint space and the maximum norm  $\| z - z^{\prime}\|_{\max} = \max \{||x - x^{\prime}||,||y - y^{\prime}||\}$ , we find a  $k$ -th nearest neighbor  $z_{i}^{k\mathrm{NN}} = (x_{i}^{k\mathrm{NN}},y_{i}^{k\mathrm{NN}})$ . To find a value that makes  $x_{i}^{k\mathrm{NN}}$  be  $n_x(i)$ -th nearest neighbor of  $x_{i}$  in the space spanned by  $X$ , we count the number  $n_x(i)$  of points  $x_{j}$  whose distances to  $x_{i}$  are less than  $\epsilon_{x}(i)/2$ , where  $\epsilon_{x}(i)$  is twice the distance from  $x_{i}$  to  $x_{i}^{k\mathrm{NN}}$  (see Appendix D for an illustrative example). We note that  $\epsilon_{y}(i)$  and  $n_y(i)$  can be similarly defined by replacing  $x$  by  $y$ . Now let  $\epsilon(i)$  be twice the distance between  $z_{i}$  and  $z_{i}^{k\mathrm{NN}}$ , i.e.,  $\epsilon(i) = 2 \cdot \|z_{i} - z_{i}^{k\mathrm{NN}}\|_{\max}$ . Then KSG estimators for the joint and marginal entropies are given as:
$$
\widehat {H} _ {\mathrm {K S G}} (X, Y) = - \psi (k) + \psi (N) + \log \left(c _ {d _ {X}} c _ {d _ {Y}}\right) + \frac {d _ {X} + d _ {Y}}{N} \sum_ {i = 1} ^ {N} \log \epsilon (i) \tag {3}
$$
$$
\widehat {H} _ {\mathrm {K S G}} (X) = - \frac {1}{N} \sum_ {i = 1} ^ {N} \psi \left(n _ {x} (i) + 1\right) + \psi (N) + \log c _ {d _ {X}} + \frac {d _ {X}}{N} \sum_ {i = 1} ^ {N} \log \epsilon_ {x} (i) \tag {4}
$$
Then we use the chain rule of conditional entropy  $\widehat{H}_{\mathrm{KSG}}(X,Y) - \widehat{H}_{\mathrm{KSG}}(X)$  and estimators from Equation 3 and Equation 4 to obtain a conditional entropy estimator and its lower bound as below:
$$
\begin{array}{l} \widehat {H} _ {\mathrm {K S G}} (Y \mid X) = \frac {1}{N} \sum_ {i = 1} ^ {N} \left[ \psi \left(n _ {x} (i) + 1\right) + d _ {X} \left(\log \epsilon (i) - \log \epsilon_ {x} (i)\right) + d _ {Y} \log \epsilon (i) \right] + C \\ \geq \frac {1}{N} \sum_ {i = 1} ^ {N} \left[ \psi \left(n _ {x} (i) + 1\right) + d _ {Y} \log \epsilon (i) \right] - \psi (k) + \log c _ {d _ {Y}} \tag {5} \\ \end{array}
$$
where  $C$  denotes  $-\psi (k) + \log c_{d_Y}$  and lower bounds hold because  $\epsilon (i)\geq \epsilon_{x}(i)$  always holds.
We present a new exploration technique that maximizes the value-conditional state entropy (VCSE), which addresses the issue of state entropy exploration in a supervised setup by taking into account the value estimates of visited states for computing the intrinsic bonus. Our main idea is to prevent the distribution of high-value states from affecting exploration around low-value states, and vice versa, by filtering out states whose value estimates significantly differ from each other for computing the intrinsic bonus. In this section, we first describe how to define the value-conditional state entropy and use it as an intrinsic reward (see Section 4.1). We then describe how we train RL agents with the intrinsic reward (see Section 4.2). We provide the pseudocode of our method in Algorithm 1.
4.1 Maximum Value-Conditional State Entropy
Value-conditional state entropy Let  $\pi$  be a stochastic policy,  $f_v^{\pi}$  be an extrinsic critic function, and  $V_{\pi}^{\mathrm{vs}}$  be a random variable with a probability density  $p_{V_{\pi}^{\mathrm{vs}}}(v) = \sum_{s \in \mathcal{B}} \mathbf{1}_{f_v^{\pi}(s) = v} / |\mathcal{B}|$ . Our key idea is to maximize the value-conditional state entropy  $\mathcal{H}(S \mid V_{\pi}^{\mathrm{vs}}) = E_{v \sim p_{V_{\pi}^{\mathrm{vs}}}(v)}[\mathcal{H}(S \mid V_{\pi}^{\mathrm{vs}} = v)]$ , which
${}^{3}z$  and  ${z}^{\prime }$  are arbitrary samples from a random variable  $Z$  . We also note that this work uses Euclidean norm for  $\left| {\left| x - {x}^{\prime }\right| \rVert }\right.$  and  $\left| {\left| y - {y}^{\prime }\right| \rVert }\right.$  ,while any norm can be used for them.
Algorithm 1 Maximum Value-Conditional State Entropy Exploration
1: Initialize policy  $\pi$ , extrinsic critic  $f_{v}^{\pi}$ , critic  $f_{v,\mathrm{T}}^{\pi}$ , replay buffer  $\mathcal{B}$   
2: for each environment step  $t$  do  
3: Collect transition  $\tau_{t} = (s_{t}, a_{t}, s_{t+1}, r_{t}^{\mathbf{e}})$  and update  $\mathcal{B} \gets \mathcal{B} \cup \{\tau_{t}\}$   
4: Sample mini-batch  $\{\tau_{t_j}\}_{j=1}^{B} \sim \mathcal{B}$   
5: for  $j = 1$  to  $B$  do  
6: Find  $(s_{t_j}^{k\mathrm{NN}}, v_{t_j}^{k\mathrm{NN}})$  from  $\{\tau_{t_j}\}_{j=1}^{B}$  using max(||s - s'|, ||v - v'|)  
7: Compute distances  $\epsilon_s(t_j) \gets 2 \cdot ||s_{t_j} - s_{t_j}^{k\mathrm{NN}}||$  and  $\epsilon_v(t_j) \gets 2 \cdot ||v_{t_j} - v_{t_j}^{k\mathrm{NN}}||$   
8: Compute joint state-value space distance  $\epsilon(t_j) \gets \max(\epsilon_s(t_j), \epsilon_v(t_j))$   
9: Compute bias  $n_v(t_j)$  by counting  $\{v_i \mid v_i \in (v_{t_j} - \epsilon_v(t_j)/2, v_{t_j} + \epsilon_v(t_j)/2)\}$   
10: Compute intrinsic reward  $r_{t_j}^{\mathrm{VCSE}} \gets \psi(n_v(t_j) + 1)/ds + \log \epsilon(t_j)$   
11: Compute total reward  $r_{t_j}^{\mathrm{T}} \gets r_{t_j}^{\mathrm{e}} + \beta \cdot r_{t_j}^{\mathrm{VCSE}}$   
12: end for  
13: Update policy  $\pi$ , critic  $f_{v,\mathrm{T}}^{\pi}$  with  $\{(s_{t_j}, a_{t_j}, s_{t_j+1}, r_{t_j}^{\mathrm{T}})\}_{j=1}^{B}$   
14: Update extrinsic critic  $f_{v}^{\pi}$  with  $\{(s_{t_j}, a_{t_j}, s_{t_j+1}, r_{t_j}^{\mathrm{e}})\}_{j=1}^{B}$   
15: end for
corresponds to separately estimating the state entropies that are conditioned on the value estimates of each state and then maximizing their average. This intuitively can be seen as partitioning the visited state space with value estimates and averaging the state entropy of each partitioned space.
Estimation and intrinsic reward To estimate  $\mathcal{H}(S\mid V_{\pi}^{\mathrm{vs}})$ , we employ the KSG conditional entropy estimator in Equation 5. Specifically, we estimate the value-conditional state entropy as below:
$$
\widehat {H} _ {\mathrm {K S G}} (S \mid V) = \frac {1}{N} \sum_ {i = 1} ^ {N} \left[ \psi \left(n _ {v} (i) + 1\right) + d _ {V} \left(\log \epsilon (i) - \log \epsilon_ {v} (i)\right) + d _ {S} \log \epsilon (i) \right] + C \tag {6}
$$
where  $C$  denotes  $-\psi (k) + \log c_{d_S}$ . In practice, we maximize the lower bound of value-conditional state entropy because of its simplicity and ease of implementation as below:
$$
\widehat {H} _ {\mathrm {K S G}} (S \mid V) \geq \frac {1}{N} \sum_ {i = 1} ^ {N} \left[ \psi \left(n _ {v} (i) + 1\right) + d _ {S} \log \epsilon (i) \right] - \psi (k) + \log c _ {d _ {S}} \tag {7}
$$
We then define the intrinsic reward  $r_t^{\mathrm{VCSE}}$  similarly to Equation 2 by ignoring constant terms as below:
$$
r _ {t} ^ {\mathrm {V C S E}} = \frac {1}{d _ {S}} \psi \left(n _ {v} (i) + 1\right) + \log \epsilon (i) \text {w h e r e} \epsilon (i) = 2 \cdot \max  \left(\left| \left| s _ {i} - s _ {i} ^ {k \mathrm {N N}} \right| \right|, \left| \left| v _ {i} - v _ {i} ^ {k \mathrm {N N}} \right| \right|\right) \tag {8}
$$
To provide an intuitive explanation of how our reward encourages exploration, we note that a  $(s_j,v_j)$  pair whose value  $v_{j}$  largely differs from  $v_{i}$  is not likely to be chosen as  $k$  -th nearest neighbor because maximum-norm will be large due to large value norm  $||v_{i} - v_{j}||$ . This means that  $k$  -th nearest neighbor will be selected among the states with similar values, which corresponds to partitioning the state space with values. Then maximizing  $\epsilon (i)$  can be seen as maximizing the state entropy within each partitioned space. We provide an illustration of our method in Figure 1.
4.2 Training
Objective We train RL agents to solve the target tasks with the reward  $r_t^{\mathrm{T}} = r_t^{\mathsf{e}} + \beta \cdot r_t^{\mathrm{VCSE}}$  where  $\beta > 0$  is a scale hyperparameter that adjusts the balance between exploration and exploitation. We do not utilize a decay schedule for  $\beta$  unlike prior work (Seo et al., 2021) because our method is designed to avoid redundant exploration. We provide the pseudocode of our method in Algorithm 1.
Implementation detail To compute the intrinsic reward based on value estimates of expected return from environments, we train an extrinsic critic  $f_{v}^{\pi}$  to regress the expected return computed only with the extrinsic reward  $r_{t}^{\mathbf{e}}$ . Then we use the value estimates from  $f_{v}^{\pi}$  to compute the intrinsic reward in Equation 8. To train the agent to maximize  $r_{t}^{\mathrm{T}}$ , we train a critic function  $f_{v,\mathrm{T}}^{\pi}$  that regresses
Figure 2: Learning curves on six navigation tasks from MiniGrid (Chevalier-Boisvert et al., 2018) as measured on the success rate. The solid line and shaded regions represent the interquartile mean and standard deviation, respectively, across 16 runs.
the expected return computed with  $r_t^{\mathrm{T}}$ , which the policy  $\pi$  aims to maximize. To ensure that the distribution of value estimates be consistent throughout training, we normalize value estimates with their mean and standard deviation computed with samples within a mini-batch. When an environment is partially observable as a high-dimensional observation  $o_t \in \mathcal{O}$  is only available instead of a fully observable state, we follow a common practice (Mnih et al., 2015) that reformulates the problem as MDP by stacking observations  $\{o_t, o_{t-1}, o_{t-2}, \ldots\}$  to construct a state  $s_t$ . We also note that we use a replay buffer  $\mathcal{B}$  for entropy estimation following Liu & Abbeel (2021b); Seo et al. (2021) in that it explicitly encourages the policy to visit unseen, high-reward states which are not in the buffer.
We design our experiments to evaluate the generality of our maximum value-conditional state entropy (VCSE) exploration as a technique for improving the sample-efficiency of various RL algorithms (Mnih et al., 2016; Yarats et al., 2021a). We conduct extensive experiments on a range of challenging and high-dimensional domains, including partially-observable navigation tasks from MiniGrid (Chevalier-Boisvert
et al., 2018), pixel-based locomotion tasks from DeepMind Control Suite (DMC; Tassa et al. (2020)), and pixel-based manipulation tasks from Meta-World (Yu et al., 2020).
Figure 3: Examples of tasks from MiniGrid, DeepMind Control Suite, and Meta-World.
5.1 MiniGrid Experiments
Setup We evaluate our method on navigation tasks from MiniGrid benchmark (Chevalier-Boisvert et al., 2018) consisting of sparse reward goal-reaching tasks. This environment is partially observable as the agent has access to a  $7 \times 7 \times 3$  encoding of the  $7 \times 7$  grid in front of it instead of the full grid. As a baseline, we first consider RE3 (Seo et al., 2021) which trains Advantage Actor-Critic (A2C; Mnih et al. 2016) agent with state entropy (SE) exploration where the intrinsic reward is obtained using the representations from a random encoder. We build our method upon the official implementation of RE3 by modifying the SE reward with our VCSE reward. Unlike RE3, we do not normalize our intrinsic reward with standard deviation and also do not utilize a separate buffer for computing the intrinsic reward using the on-policy batch. We use  $k = 5$  for both SE and VCSE by following the original implementation. See Appendix A for more details.
4We provide additional results on fully-observable MiniGrid tasks in Appendix C, where trends are similar.