SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\mathrm{diag}(\mathbf{z})\in \mathbb{R}^{|S|\times |S|}$  denotes the diagonal matrix whose diagonal elements corresponds to the discrete stationary distribution of the MDP under the current policy.
Similar to how SF generalises SR, we propose the "Predecessor Feature" (PF) that generalises PR.
$$
\boldsymbol {\xi} ^ {\pi} (s) = \mathbb {E} \left[ \sum_ {k = 0} ^ {\infty} \gamma^ {k} \mu_ {t - k} | s _ {t + 1} = s \right] = \boldsymbol {\mu} \left(s _ {t + 1}\right) + \gamma \mathbb {E} \left[ \boldsymbol {\xi} ^ {\pi} \left(s _ {t}\right) | s _ {t + 1} = s, a _ {t} = a \right]. \tag {14}
$$
Similarly to the SF, the recursive definition of the PF again allows a simple expression of the TD error for gradient-based learning of the PF.
$$
\delta_ {t} ^ {\mathrm {P F}} = \mathbb {E} \left[ \left(\phi \left(s _ {t + 1}\right) + \gamma \xi \left(s _ {t}\right) - \xi \left(s _ {t + 1}\right)\right) \right], \tag {15}
$$
We utilise the norms of SF and PF to replace the row sums in discrete settings for tractable approximation to  $r_{\mathrm{SR - R}}$  in continuous state spaces. We use the same feature vector,  $\phi$ , for computing the SF and PF. In order to ensure the SF and PF are of similar scales across the state space, we normalise  $\phi (s)$  such that  $||\phi (s)||_2 = 1$  for all  $s$ . Contrary to how we define  $r_{\mathrm{SR - R}}$  as the difference between the SR and the column sum of the SR in discrete  $\mathrm{MDPs}^4$ , we find that setting the intrinsic reward as the difference between the reciprocal of the norms of the SF and the PF yields better empirical performance. We hence define the continuous Successor-Predecessor intrinsic reward as follows.
$$
r _ {\mathrm {S F - P F}} = \frac {1}{\left| \left| \phi \left(s _ {t + 1}\right) \right| \right| _ {1}} - \frac {1}{\left| \left| \psi \left(s _ {t} , a _ {t}\right) \right| \right| _ {1}} \tag {16}
$$
Details of deep RL implementation of  $r_{\mathrm{SF - PF}}$  We instantiate  $r_{\mathrm{SF - PF}}$  based on a Deep Q Network (DQN; [26]), with auxiliary predictive reconstruction task  $(\mathcal{L}_{\mathrm{recon}} = \mathbb{E}\left[(s_{t + 1} - \hat{s}_{t + 1})^2 |s_t\right]$  where  $\hat{s}_{t + 1}$  is the predicted next state), and separate heads for computing the q-values, the SF, and the PF, respectively (Figure 2). We call this model DQN-SF-PF. Note that, following Machado et al. [12], the intermediate feature representation  $\phi$  is trained given only the predictive reconstruction and value learning supervisions, and not updated given the TD error in the learning of the SF or the PF (the filled black circle in Figure 2 indicating the stop_gradient operation). We adopt the same set of hyperparameters and architecture for the DQN as reported in Oh et al. [27]. To make the comparison consistent, we utilise the
Figure 2: Graphical illustration of the neural network architecture of DQN-SF-PF for Atari games. Note that the state feature vector is L2-normalised,  $\phi (s) = \frac{\tilde{\phi}(s)}{||\tilde{\phi}(s)||_2}$ , where  $\tilde{\phi} (s)$  is the raw output of the convolutional encoder.
Note we refer to discrete/continuous MDP as an MDP with discrete/continuous state and action space.
Table 1: Evaluations SARSA-SRR and related baseline agents on RiverSwim and SixArms (averaged over 100 seeds, numbers in the parentheses represents standard errors).
SARSA	SARSA-SR	SARSA-FR	SARSA-SRR	SARSA-SR-PR
RiverSwim	25,075	1,197,075	1,547,243	2,547,156	2,857,324
(1,224)	(36,999)	(34,050)	(479,655)	(419,922)
SixArms	376,655	1,025,750	119,149	2,199,291	1,845,229
(8,449)	(49,095)	(42,942)	(1,024,726)	(1,032,050)
mixed Monte-Carlo return loss [28, 12], defined as following.
$$
\mathcal {L} _ {q} = \mathbb {E} \left[ ((1 - \tau) \delta_ {\mathrm {T D}} (s, a) + \tau \delta_ {\mathrm {M C}} (s, a)) ^ {2} \right],
$$
$$
\text {w h e r e} \delta_ {\mathrm {M C}} (s, a) = \sum_ {t = 0} ^ {\infty} \gamma^ {t} \left(r \left(s _ {t}, a _ {t}\right) + \beta r _ {\mathrm {S F - P F}} \left(s _ {t}, a _ {t}; \theta^ {-}\right)\right) - q (s, a; \theta), \tag {17}
$$
where  $\delta_{\mathrm{TD}}$  denotes the standard TD error for q-values (Eq. 2),  $\tau$  is the scaling factor controlling the contribution of the TD error and the Monte Carlo error, and  $\theta$  and  $\theta^{-}$  denote the parameters for the online and target DQN-SF-PF, respectively. Hence the overall loss objective for training DQN-SF-PF is as following.
$$
\mathcal {L} _ {\mathrm {D Q N - S F - P F}} = w _ {\mathrm {q}} \mathcal {L} _ {q} + w _ {\mathrm {S F}} \delta^ {\mathrm {S F}} + w _ {\mathrm {P F}} \delta^ {\mathrm {P F}} + w _ {\text {r e c o n}} \mathcal {L} _ {\text {r e c o n}}, \tag {18}
$$
where  $w_{\mathrm{q/SF/PF/recon}}$  denotes the scaling factors for the respective loss terms. The complete set of hyperparameters for DQN-SF-PF can be found in the Appendix.
Classical hard exploration tasks. We evaluate performance of the discrete SPIE agent (and other considered agents in Section 3: SARSA, SARSA-SR, SARSA-FR) on two classical hard exploration tasks commonly studied in the PAC-MDP literature, RiverSwim and SixArms [29] (appendix Figure 5). In both tasks, environment transition dynamics induce a bias towards states with low rewards, leaving high rewards in states that are harder to reach. Evaluation of the agents is based on the cumulative reward collected within 5000 training steps.
We observe from Table 1 that SARSA-SRR significantly outperforms all other considered agents. Moreover, in order to further justify the utility of  $\mathcal{R}_{\mathrm{SR - R}}$  in driving exploration, we run ablation studies by evaluating the performance of variants of SARSA-SRR (Appendix B.4). Ablation studies reveal the importance of combining both prospective and retrospective information for exploration, as well as the benefits of dynamic balancing exploring uncertain states and bottleneck states.
In order to validate the replacement of column norm of the SR with column norm of the PR in the construction of  $r_{\mathrm{SR - R}}$ , given the reciprocal relationship (Eq. 13), we empirically evaluate the performance of the SARSA agent with the alternative intrinsic reward,  $r_{\mathrm{SR - PR}}(s,a) = \hat{\mathbf{M}} [s,s^{\prime}] - ||\hat{\mathbf{N}} [s,s^{\prime}]\|_{1}$ . SARSA-SR-PR yields comparable performance as SARSA-SRR on both RiverSwim and SixArms (Table 1), empirically justifying the instantiation of SPIE with the PR for capturing the retrospective information.
Goal-oriented / sparse-reward tasks. We next evaluate the agents on grid world tasks with a single terminal goal state (Figure 1a;  $OF$ -small and Cluster-hard). All non-terminal transitions yield rewards of  $-1$ , and transitions into the goal state generates a reward of 0. Such goal-directed or sparse-reward tasks require efficient exploration. We examine both open-field and clustered grid-worlds. In  $OF$ -small and Cluster-hard tasks, SARSA-SRR outperforms both vanilla SARSA and SARSA-SR in terms of sample efficiency (Figure 3). In addition, SARSA-SRR yields more stable training and performance is more robust across different random seeds. Note that the navigation performance of SARSA-SR during training is highly unstable, which might attribute to its equivalence to count-based exploration given that visitation count is only a local measure for exploration. Somewhat surprisingly, the improvement for SARSA-SRR is more significant in open-field grid world ( $OF$ -small) rather than
(d)
Figure 3: Goal-oriented navigation in grid worlds. Evaluations of SARSA, SARSA-SR and SARSA-SRR on  $OF$ -small (a) and Cluster-hard (b) grid worlds (Figure 1a) with stationary reward structure, and on  $OF$ -small (c) and Cluster-hard (d) with non-stationary reward structures. The red dashed horizontal line represents the shorted path distance. The black dashed vertical lines represent the time point at which the goal change occurs.
the clustered grid world (Cluster-hard), in contrast to the pure exploration experiments (Figure 1b). Nevertheless, the improvement is strong and consistent.
In many real-world tasks, the environment is inherently dynamic, requiring continual exploration for adapting the optimal policy with respect to the non-stationary task structure. One such example is random foraging, where foods are depleted upon consumption, and new rewards appear in new locations. As argued in Section 3, SARSA-SRR yields "cycling" exploratory behaviour (Figure 6), hence could facilitate continual exploration that is potentially suitable for such non-stationary environments. To empirically justify the hypothesis, we consider the Non-Markovian Reward Decision Process (NMRDP; [23]), where the reward changes dynamically given the visited state sequence. We instantiate the NMRDPs in the grid worlds,  $OF$ -small and Cluster-hard, where there are three reward states ( $G$ ,  $G_1$ ,  $G_2$ ; Figure 1a) that are sequentially activated (and deactivated) every 30 episodes. As shown in Figure 3c and 3d, we observe that SARSA-SRR consistently outperforms SARSA and SARSA-SR, reaching the new goal states in increasingly shorter timescales. This supports our idea that SPIE provides a more ethologically plausible exploration strategy for dealing with non-stationarity. However, we note that the main focus of the current paper is on improved exploration within a single task, instead of over a stream of inter-related tasks. Here we provide preliminary evidence of potential applicability of SPIE in such continual exploration setting, and we leave more rigorous investigation in this direction for future work.
Linear function approximation for continuous state spaces. We next evaluate SPIE with function approximation. As a first step, we consider the linear features before moving onto the deep RL setting. We consider the MountainCar task (Figure 4a; [30]), with sparse reward structure, where we set the reward to 0 for all transitions into non-terminal states (the terminal state is indicated by the flag on the top of the right hill). We utilise Q-learning with linear function approximation, where we define the linear features to be the 128-dimensional random Fourier features (RFF [31]; Figure 4b). The SF and the PF are defined given the RFF, and are learned via standard TD-learning (Eq. 10; 15). The performance (over the first 1000 training episodes) of the resulting linear-Q agents with  $r_{\mathrm{SF}}$  and
(c)
Figure 4: Evaluation of SPIE with linear features in MountainCar. (a) Graphical demonstration of MountainCar environment; (b); Example random Fourier features; (c) Evaluations of Q-learning with linear function approximation with intrinsic rewards  $r_{\mathrm{SF}}$  and  $r_{\mathrm{SF-PF}}$  on MountainCar. Reported results are averaged over 10 random seeds.
Table 2: Evaluations of SPIE with deep RL implementation on hard-exploration Atari games (averaged over 10 random seeds, numbers in the parentheses are 1 standard errors).
DQN	DQNMMC	RND	DQNMMC-SR	DQNMMC-SF-PF
Freeway	32.4 (0.3)	29.5 (0.1)	28.2 (0.2)	29.4 (0.1)	27.5 (0.2)
Gravitar	118.5 (22.0)	1078.3 (254.1)	714.1 (105.9)	457.4 (120.3)	1223.0 (408.9)
Mont. Rev.	0.0 (0.0)	0.0 (0.0)	528 (314.0)	1395.4 (1121.8)	1530.0 (1072.1)
Private Eye	1447.4 (2,567.9)	113.4 (42.3)	61.3 (53.7)	104.4 (50.4)	488.2 (390.9)
Solaris	783.4 (55.3)	2132.6 (394.8)	1395.2 (401.7)	1890.1 (163.1)	2455.8 (262.0)
Venture	4.4 (5.4)	1220.1 (51.0)	953.7 (167.3)	1348.5 (56.5)	1274.0 (133.2)
$r_{\mathrm{SF - PF}}$  is shown in Figure 4c. The agent with  $r_{\mathrm{SF - PF}}$  outperforms the opposing agent significantly, empirically justifying the utility of SPIE in the linear function approximation regime.
Deep RL instantiation of SPIE in Atari games. We empirically evaluate  $\mathrm{DQN}_{\mathrm{SF - PF}}$  on 6 Atari games with sparse reward structures [32]: Freeway, Gravitar, Montezuma's Revenge, Private Eye, Solaris, and Venture. We follow the evaluation protocol as stated in Machado et al. [33], where we report the averaged evaluation scores over 10 random seeds given  $10^{8}$  training steps. The agent takes (stacked) raw pixel observations as inputs. Across all 4 games, the  $\beta$  values are set to 0.07 and the discounting factor  $\gamma = 0.995$ . We adopt the epsilon-annealing scheme as in [28], which linearly decreases  $\epsilon$  from 1.0 to 0.1 over the first  $10^{6}$  frames. We train the network with RMSprop, with standard hyperparameters, learning rate 0.00025,  $\epsilon = 0.001$  and decay equals 0.95 [26]. The discounting factors for value learning and online learning of the SF and the PF are set to 0.99. The scaling factors in Eq. 18 are set such that the different losses are on roughly similar scales:  $w_{q} = 1$ ,  $w_{\mathrm{SF}} = 1500$ ,  $w_{\mathrm{PF}} = 1500$ ,  $w_{\mathrm{recon}} = 0.001$ . More implementation details can be found in Appendix.
We compare DQN-SF-PF with vanilla DQN trained with standard TD error, vanilla DQN trained with the MMC loss  $(\mathcal{L}_q)$ , Random Network Distillation (RND; [20]), DQN-SR trained with the MMC loss [12] (Table 2). All agents are trained with the predictive reconstruction auxiliary task. By comparing with our main baseline, DQN-SR, we observe that DQN-SF-PF significantly outperforms DQN-SR on Four games (Gravitar, Montezuma's Revenge, Private Eye and Solaris), whilst yielding similar performance on the remaining two games (Freeway and Venture). Moreover, DQN-SF-PF outperforms RND, a state-of-the-art Deep RL algorithm for exploration, on all 6 games. The empirical difference is not only reflected in the asymptotic performance, but also in the sample efficiency of learning. Specifically, for Montezuma's Revenge, one of the hardest exploration games in the Atari suite, our agent achieves near asymptotic performance (defined as the score given  $10^{8}$  training steps) with only  $\sim 8 \times 10^{6}$  training frames, whereas the performance of DQN-SR saturates at  $\sim 2.4 \times 10^{7}$  training frames (with a lower score). We emphasise that the main aim of our empirical evaluations is to validate the utility of SPIE exploration objective as a simple modification to DQN. In principle, SPIE can be integrated with any state-of-the-art RL agent, and different instantiations of SPIE could be implemented to deal with the task at hand. We leave such investigation for future work.
The development of more efficient exploration algorithms is essential for practical implementation of RL agents in real-world environment where sample efficiency and optimality are vital to success. Here, we propose a general intrinsically motivated exploration framework, SPIE, where we construct intrinsic rewards by combining both prospective and retrospective information contained in past trajectories. The retrospective component provides information about the connectivity structure of the environment, facilitating more efficient targeted exploration between sub-regions of state space given structure awareness (e.g., robust identification of the bottleneck states; Figure 1a). SPIE yields more sample efficient exploration in discrete MDPs under complete absence of external reinforcement. Moreover, a side benefit we observe empirically is that SPIE exhibits ethologically plausible exploratory behaviour during exploration in grid worlds (i.e., cycling between different clusters of states). In continuous state space, we developed a novel generalization of the predecessor representation, the predecessor features, for capturing retrospective information in continuous spaces. Empirical evaluations on both discrete and continuous MDPs demonstrate that SPIE yields improvements over existing intrinsic exploration methods, in terms of sample efficiency of learning and asymptotic performance, and for adapting to non-stationary reward structures.
We instantiate SPIE using the SR and the PR, but we note that SPIE is a general framework that can be implemented with other formulations (e.g., predictive error in a temporally backward direction [34, 35]) and with more advanced neural architectures (including those currently unthought of). Although here we have examined the empirical properties of SPIE, the theoretical underpinnings for SPIE and the bottleneck seeking exploratory behavior bears further investigation. Specifically, more work needs to be done to probe the theoretical property of using SF and PF in continuous settings. Our definition of  $r_{\mathrm{SR-R}}$  overlaps with the successor contingency [25, 36], which has long been recognised for learning causal relationship between predictors and reward [37]. An interesting venue for future work is to investigate the implications of SPIE for causally guided exploration in RL. Another interesting direction for future work is to investigate the implications of SPIE in human exploration, where we could utilise SPIE to investigate how human balance local (e.g., visitation counts) versus global (e.g., environment structure) information for exploration in sequential decision tasks [38, 39].
We thank Franziska Brändle, James Heald, and Ted Moskovitz for useful discussions, and anonymous reviewers for valuable comments. This work is funded by the UKRI, DeepMind, the Gatsby Charitable Foundation, the Simons Foundation, the Wellcome Trust, and the Harvard Brain Initiative and by the Center for Brains, Minds and Machines (CBMM).
[1] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[2] Susan Amin, Maziar Gomrokchi, Harsh Satija, Herke van Hoof, and Doina Precup. A survey of exploration methods in reinforcement learning. arXiv preprint arXiv:2109.00157, 2021.
[3] Gerald Tesauro et al. Temporal difference learning and td-gammon. Communications of the ACM, 38(3):58-68, 1995.
[4] Will Dabney, Georg Ostrovski, and André Barreto. Temporally-extended  $\{\backslash$  epsilon\}-greedy exploration. arXiv preprint arXiv:2006.01782, 2020.
[5] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29, 2016.
[6] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pages 2778-2787. PMLR, 2017.
[27] Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in atari games. Advances in neural information processing systems, 28, 2015.
[28] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
[29] Alexander L Strehl and Michael L Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences, 74(8):1309-1331, 2008.
[30] Andrew William Moore. Efficient memory-based learning for robot control. Technical report, University of Cambridge, 1990.
[31] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007.
[32] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[33] Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
[34] Changmin Yu, Dong Li, Jianye Hao, Jun Wang, and Neil Burgess. Learning state representations via retracing in reinforcement learning. arXiv preprint arXiv:2111.12600, 2021.
[35] Tao Yu, Cuiling Lan, Wenjun Zeng, Mingxiao Feng, Zhizheng Zhang, and Zhibo Chen. Playvirtual: Augmenting cycle-consistent virtual trajectories for reinforcement learning. Advances in Neural Information Processing Systems, 34:5276-5289, 2021.
[36] Charles R Gallistel, Andrew R Craig, and Timothy A Shahan. Temporal contingency. Behavioural processes, 101:89-96, 2014.
[37] Herbert M Jenkins and William C Ward. Judgment of contingency between responses and outcomes. Psychological monographs: General and applied, 79(1):1, 1965.
[38] Daniel Acuna and Paul R Schrater. Structure learning in human sequential decision-making. Advances in neural information processing systems, 21, 2008.
[39] Franziska Brändle, Lena J Stocks, Joshua Tenenbaum, Samuel J Gershman, and Eric Schulz. Intrinsically motivated exploration as empowerment. 2022.
[40] Ronen I Brafman and Moshe Tennenholtz. R-max-a general polynomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research, 3(Oct):213-231, 2002.
A More Details on Predecessor Representation
Here we provide proofs of the reciprocal relationship between the SR and the PR.
Proposition A.1.  $\mathsf{Ndiag}(\mathbf{z}) = \mathrm{diag}(\mathbf{z})\mathsf{M}$ , where  $\mathrm{diag}(\mathbf{z})$  is the diagonal matrix with the diagonal elements as the vector  $\mathbf{z}$ , and  $\mathbf{z}$  is the vector of stationary distribution of  $\mathcal{P}^{\pi}$  (i.e.,  $\mathbf{z}[i] = \lim_{t\to \infty}\mathbb{E}_{\mathcal{P}^{\pi}}[s_t = i]$ .
Proof. Given the formal definition of the SR and the PR (Eq. 3; 11), we have the following analytical expressions.
$$
\mathbf {M} = (\mathbf {I} - \gamma \mathcal {P} ^ {\pi}) ^ {- 1}; \quad \mathbf {N} = (\mathbf {I} - \gamma \tilde {\mathcal {P}} ^ {\pi}) ^ {- 1}; \tag {19}
$$
where  $\tilde{\mathcal{P}}^{\pi}$  is the temporally reversed transition distribution. Assume matrix formulation of  $\mathcal{P}^{\pi}$  and  $\tilde{\mathcal{P}}^{\pi}$ ,  $\mathbf{P}$  and  $\tilde{\mathbf{P}}$  in  $\mathbb{R}^{|\mathcal{S}| \times |\mathcal{S}|}$ , we have the following.
$$
\begin{array}{l} \tilde {\mathbf {P}} _ {i j} = \mathbb {P} \left(s _ {t} = i \mid s _ {t + 1} = j\right) = \frac {\mathbb {P} \left(s _ {t + 1} = j \mid s _ {t} = i\right) \mathbb {P} \left(s _ {t} = i\right)}{\mathbb {P} \left(s _ {t + 1} = j\right)} = \frac {\mathbf {P} _ {i j} \mathbf {z} _ {i}}{\mathbf {z} _ {j}}, \tag {20} \\ \Rightarrow \tilde {\mathbf {P}} \operatorname {d i a g} (\mathbf {z}) = \operatorname {d i a g} (\mathbf {z}) \mathbf {P}, \\ \end{array}
$$
Substituting the reciprocal relationship between  $\tilde{\mathbf{P}}$  and  $\mathbf{P}$  into the definition of the PR, we have the following.
$$
\begin{array}{l} \mathbf {N} = \left(\mathbf {I} - \gamma \operatorname {d i a g} (\mathbf {z}) \mathbf {P} \operatorname {d i a g} (\mathbf {z}) ^ {- 1}\right) ^ {- 1}, \\ \mathbf {N} \operatorname {d i a g} (\mathbf {z}) = \left(\mathbf {I} - \gamma \operatorname {d i a g} (\mathbf {z}) \mathbf {P} \operatorname {d i a g} (\mathbf {z}) ^ {- 1}\right) ^ {- 1} \operatorname {d i a g} (\mathbf {z}) \\ = \left(\operatorname {d i a g} (\mathbf {z}) ^ {- 1} (\mathbf {I} - \gamma \operatorname {d i a g} (\mathbf {z}) \mathbf {P} \operatorname {d i a g} (\mathbf {z}) ^ {- 1})\right) ^ {- 1} \tag {21} \\ = \left(\left(\mathbf {I} - \gamma \mathbf {P}\right) \operatorname {d i a g} (\mathbf {z}) ^ {- 1}\right) ^ {- 1} \\ = \operatorname {d i a g} (\mathbf {z}) \left(\left(\mathbf {I} - \gamma \mathbf {P}\right)\right) ^ {- 1} \\ = \operatorname {d i a g} (\mathbf {z}) \mathbf {M} \\ \end{array}
$$
B Further results on tabular hard exploration tasks.
B.1 Graphical illustration of tabular hard-exploration tasks.
The demos of RiverSwim and SixArms is shown in Figure 5. In both tasks, the environmental transition dynamics impose asymmetry, biasing the agent towards low-rewarding states that are easier to reach, with greater rewards available in hard-to-reach states.
(a) RiverSwim
(b) SixArms
Figure 5: Discrete MDPs. Transition probabilities are denoted by  $\langle$  action, probability, reward  $\rangle$ . In RiverSwim (a), the agent starts in state 1 or 2. In SixArms (b), the agent starts in state 0.