# Long-Term Visitation Value for Deep Exploration in Sparse-Reward Reinforcement Learning
## 3 Long-Term Visitation Value for Deep Exploration
### Final Remarks  
In this section, we discuss the uniform count assumption used to derive the bounds, the differences between the proposed W-functions and intrinsic motivation, the benefit of using the count-based one, and potential issues in stochastic environments and continuous MDPs. **Uniform count assumption. ** In Section 3.3, we derived an initialization for \(W^{\beta}_{\textsc{cca}}\) and a bound for \(W^{\beta}_{\textsc{cca}}\) to guarantee that an action is not executed twice before all actions are executed once. For that, we assumed the visitation count to be uniform for all state-action pairs. We acknowledge that this assumption is not true in practice. First, the agent may need to revisit old states in order to visit new ones. Second, when an episode is over the state is reset and a new episode starts. This is common practice also for infinite horizon MDPs when usually the environment is reset after some steps. Thus, the agent's initial state will naturally have a higher visitation count. Nonetheless, in Section 4.2.1, we empirically show that our approach allows the agent to explore the environment as uniformly as possible. **W-function vs. auxiliary rewards. ** Using an auxiliary reward such as an exploration bonus or an intrinsic motivation signal has similarities with the proposed approach, especially with \(W^{\beta}_{\textsc{cca}}\), but the two methods are not equivalent. Consider augmenting the reward  
Figure 6: **Toy example (part 3). ** Behavior policies under uniform visitation count. In the top row, reward states (1,1) and (3,1) are terminal. In the bottom row, they are not and the MDP has an infinite horizon. All state-action pairs have been executed once, except for “left” in (1,3). Only the proposed methods (**d,e**) correctly identify such state-action pair as the most interesting for exploration and propagate this information to other states thanks to TD learning. When the action is executed, all policies converge to the greedy one (**a**). with the same visitation reward used to train the W-function, i.e., \(r_{t}^{+}=r_{t}+\alpha r_{t}^{W}\). Using TD learning, the augmented Q-function used for exploration can be rewritten as  
\[Q^{+}(s_{t},a_{t})=r_{t}+\alpha r_{t}^{W}+\gamma\max_{a}Q^{+}(s_{t+1},a). \tag{28}\]  
Then consider the behavior policy derived from \(W_{\text{ics}}^{\beta}\) in Equation (14), and decompose it  
\[Q^{\pi}(s_{t},a_{t})+\alpha W_{\text{ics}}^{\beta}(s_{t},a_{t})=r_{t}+\gamma \max_{a}Q^{\pi}(s_{t+1},a)+\alpha r_{t}^{W}+\alpha\gamma_{w}\max_{a}W_{\text{ics }}^{\beta}(s_{t+1},a), \tag{29}\]  
where we considered \(\alpha=\kappa(1-\gamma_{w})\). Red terms are equivalent, but blue terms are not because of the max operator, since \(\max_{x}\{f(x)+g(x)\}\
eq\max_{x}\{f(x)\}+\max_{x}\{g(x)\}\). This explicit separation between exploitation (\(Q\)) and exploration (\(W\)) has two important consequences. First, it is easy to implement the proposed behavior policy for any off-policy algorithm, as the exploration term is separate and independent from the Q-function estimates. This implies that we can propagate the exploration value independently from the action-value function. With this choice, we can select a discount factor \(\gamma_{w}\) that differs from the MDP discount \(\gamma\). By choosing a high exploration discount factor \(\gamma_{w}\), we do long-term exploration allowing us to find the optimal policy when exploration is crucial. By choosing a low exploration discount factor \(\gamma_{w}\), instead, we perform short-term exploration which may converge faster to a greedy policy. In the evaluation in Section 4.2, we show experiments for which the discount factors are the same, as well as experiments where they differ. Furthermore, we investigate the effect of changing the W-function discount while keeping the Q-function discount fixed. Second, auxiliary rewards are non-stationary, as the visitation count changes at every step. This leads to a non-stationary target for the Q-function.