SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathcal {B R} _ {\phi} (\text {V A P O R - l i t e}, T) \\ \leq \mathbb {E} \sum_ {t = 1} ^ {N} \left[ \mathcal {U} _ {\hat {\phi} ^ {t}} \left(\mathbb {E} ^ {t} \lambda^ {\pi^ {t}}, 1 / (S L t)\right) - \sum_ {l = 1} ^ {L} \sum_ {s, a} \mathbb {E} ^ {t} \lambda_ {l} ^ {\pi^ {t}} (s, a) \mathbb {E} ^ {t} r _ {l} (s, a) \right] + \sqrt {2} \hat {\sigma} _ {\max } \sum_ {t = 1} ^ {N} \frac {1}{t} \\ \leq \underbrace {\mathbb {E} \sum_ {t = 1} ^ {N} \sum_ {l = 1} ^ {L} \sum_ {s , a} \mathbb {E} ^ {t} \lambda_ {l} ^ {\pi^ {t}} (s , a) \sqrt {2} (1 + \log S L t) \hat {\sigma} _ {l} ^ {t} (s , a)} _ {:= Z _ {1}} \\ + \underbrace {\mathbb {E} \sum_ {t = 1} ^ {N} \sum_ {l = 1} ^ {L} \sum_ {s , a} \mathbb {E} ^ {t} \lambda_ {l} ^ {\pi^ {t}} (s , a) \sqrt {2} \left(1 + \log S L t\right) \sum_ {a ^ {\prime}} \left(- \hat {\sigma} _ {l} ^ {t} (s , a ^ {\prime}) \pi_ {l} ^ {t} (s , a ^ {\prime}) \log \pi_ {l} ^ {t} (s , a ^ {\prime})\right)} _ {:= Z _ {2}} \\ + \sqrt {2} \hat {\sigma} _ {\max } \sum_ {t = 1} ^ {N} \frac {1}{t}. \\ \end{array}
$$
To bound  $Z_{1}$  we apply the Cauchy-Schwarz inequality and the pigeonhole principle, as in the proof of Lemma 12, which gives that  $Z_{1} = \widetilde{O} (\hat{\sigma}_{\max}\sqrt{SAT})$ . To bound  $Z_{2}$ , we introduce the second moment of the information [33] of a discrete random variable  $X$  supported on finite set  $\mathcal{X}$  as
$$
\mathcal {H} ^ {(2)} (X) := \sum_ {x \in \mathcal {X}} \mathbb {P} (X = x) (- \log \mathbb {P} (X = x)) ^ {2},
$$
and note that  $\mathcal{H}^{(2)}(X) \leq \log^2(|\mathcal{X}|)$  for  $|\mathcal{X}| \geq 3$  [33, Proposition 8]. We bound
$$
\begin{array}{l} Z _ {2} \stackrel {\mathrm {(i)}} {\leq} \mathbb {E} \sum_ {t = 1} ^ {N} \sum_ {l = 1} ^ {L} \sum_ {s, a} \mathbb {E} ^ {t} \lambda_ {l} ^ {\pi^ {t}} (s, a) \sqrt {2} (1 + \log S L t) \hat {\sigma} _ {l} ^ {t} (s, a) \left(- \log \mathbb {E} ^ {t} \lambda_ {l} ^ {\pi^ {t}} (s, a)\right) \\ \stackrel {\text {(i i)}} {\leq} \mathbb {E} \sqrt {\sum_ {t = 1} ^ {N} \sum_ {l = 1} ^ {L} \mathcal {H} ^ {(2)} \left(\mathbb {E} ^ {t} \lambda_ {l} ^ {\pi^ {t}}\right)} \sqrt {\sum_ {t = 1} ^ {N} \sum_ {l = 1} ^ {L} \sum_ {s , a} 2 \mathbb {E} ^ {t} \lambda_ {l} ^ {\pi^ {t}} (s , a) (1 + \log S L t) ^ {2} \hat {\sigma} _ {l} ^ {t} (s , a) ^ {2}} \\ \stackrel {\mathrm {(i i i)}} {\leq} \widetilde {O} (\widehat {\sigma} _ {\max } \sqrt {S A T}), \\ \end{array}
$$
where (i) uses that  $\mathbb{E}^t\lambda_l^{\pi^t}(s,a)\leq \pi_l^t (s,a)$ , (ii) applies the Cauchy-Schwarz inequality and (iii) uses the pigeonhole principle (as in the  $Z_{1}$  bound) combined with the aforementioned logarithmic upper bound on  $\mathcal{H}^{(2)}$ . Finally, we bound the last term by using that  $\sum_{t = 1}^{N}\frac{1}{t}\leq 1 + \log N$ . Putting everything together and using that  $\hat{\sigma}_{\mathrm{max}} = O(L)$  yields the desired bound  $\mathcal{BR}_{\phi}(\mathrm{VAPOR - lite},T)\leq$ $\widetilde{O} (L\sqrt{SAT})$ .
Figure 7: Optimizing for a weighted maximum entropy objective results in faster coverage time and thus more aggressive exploration in a reward-free GridWorld.
Figure 8: Learning time of VAPOR-lite on DeepSea (one-hot pixel representation into neural net). Dashed line represents  $2^{L}$  where  $L$  denotes the depth.
I Experimental Details
I.1 Reward-Free GridWorld
We empirically complement our discussion in Section 6 on the connection of VAPOR to maximum entropy exploration. We consider a simple 4-room GridWorld domain with no rewards, known deterministic transitions, four cardinal actions and varying state space size. We measure the ability of the agent to cover the state-action space as quickly as possible, i.e., visit each state-action at least once. We consider the state-of-the-art algorithm of [79] for the original maximum entropy exploration objective  $\max_{\lambda \in \Lambda(P)} \mathcal{H}(\lambda)$ , and compare it to the same algorithm that optimizes for weighted entropy  $\max_{\lambda \in \Lambda(P)} \mathcal{H}_{\sigma}(\lambda)$ , where we define the uncertainty as  $\sigma(s,a) = \mathbb{1}\{n(s,a) = 0\}$ , with  $n(s,a)$  the visitation count to state-action  $(s,a)$ . We see in Figure 7 that optimizing for this weighted entropy objective results in faster coverage time. This illustrates the exploration benefits of the state-action weighted entropy regularization of VAPOR (e.g., weighted by the uncertainty as done in VAPOR-lite).
I.2 DeepSea (tabular)
We provide details on the experiment of Figure 3. For the transition function we use a prior Dirichlet  $(1 / \sqrt{S})$  and for rewards a standard normal  $\mathcal{N}(0,1)$ , as done by [53]. Similar to [62, 47, 40], we accelerate learning by imagining that each experienced transition  $(s,a,s',r)$  is repeated 100 times. Effectively, this strategy forces the MDP posterior to shrink faster without favoring any algorithm, making them all converge in fewer episodes.
VAPOR implementation. The VAPOR optimization problem is an exponential cone program that can be solved efficiently using modern optimization methods. We point out that we do not need to solve the two-player zero-sum game between  $\lambda$  and  $\tau$  since the minimization over  $\tau$  conveniently admits a closed-form solution, see Equation (3). In our experiments, we use CVXPY [11], specifically the ECOS solver [12] (with 1e-8 absolute tolerance). In terms of runtime, this took a few seconds on the largest DeepSeas, which was sufficient for our purposes. There is a natural trade-off between a less accurate optimization solution (i.e., better computational complexity) and a more accurate policy (i.e., better sample complexity), which can be balanced with the choice of CVXPY solver, its desired accuracy, its maximum number of iterations, etc. The implementation of the VAPOR optimization problem in CVXPY is straightforward using the library's atomic functions. Just note that although the expression  $x\sqrt{-\log x}$  that appears in the  $\nu_{\phi}$  function (3) is not an atomic function in CVXPY, a simple trick is to use the equivalence
$$
\begin{array}{l l} \max  _ {x} & x \sqrt {- \log x} \\ \text {s . t .} & x \geq 0 \end{array} \qquad \Longleftrightarrow \qquad \begin{array}{l l} \max  _ {x, y} & y \\ \text {s . t .} & x \geq 0, \quad y \geq 0, \quad - x \log x \geq y ^ {2} / x \end{array}
$$
where  $-x\log x$  (cvxpy.entr  $(x)$  ) and  $y^{2} / x$  (cvxpy.quad_over_lin  $(y,x)$  ) are atomic functions.
Common Hyperparameter	Value
Discount factor	0.995
Replay buffer size	1e5
Replay fraction	0.9
Replay prioritization exponent	1.0
Adam step size	1e-4
λ of V-Trace(λ)	0.9
Algorithm-specific Hyperparameter (Figure 10)	None	Fixed, scalar	Tuned, scalar	Tuned, state-action
Entropy regularization β	/	0.01	/	/
Uncertainty scale σscale	0.01	0.01	0.01	0.005
τmin	/	/	0.005	/
τmax	/	/	10	/
τinit	/	/	0.02	/
τstep size	/	/	1e-4	/
Table 2: Hyperparameters used in the Atari experiments.
I.3 DeepSea (neural network)
We consider the DeepSea domain where instead of using a tabular state representation, we feed a one-hot representation of the agent location into a neural network, using bsuite [58]. As discussed in prior works [62, 53, 49], agents that do not adequately perform deep exploration are far from being able to solve depths of up to 100 within  $10^{5}$  episodes. This includes vanilla actor-critic or Soft Q-learning (which can only solve depths up to around 14, showcasing a learning time that suffers from an exponential dependence on depth), but also agents with some exploration mechanisms such as optimistic actor-critic or Bootstrapped DQN (depths up to around 50). In contrast, Figure 8 shows that VAPOR-lite is able to solve DeepSea instances out to size 100 within  $4 \times 10^{4}$  episodes, without a clear performance degradation. We chose the exact same algorithmic configuration and hyperparameter choices as our experiments on Atari (Appendix I.4), except  $\sigma_{\mathrm{scale}} = 3.0$  and Replay fraction  $= 0.995$ . This experiment shows that VAPOR-lite is capable of deep exploration, as suggested by the regret analysis in Appendix H.
I.4 Atari
RL agent. Our setup involves actors generating experience and sending them back to a learner, which mixes online data and offline data from a replay buffer to update the network weights [30]. Our underlying agent is an actor-critic algorithm with use of replay data, thus we call it 'Replay Actor-Critic (RAC)'. We also made use of V-trace clipped importance sampling to the off-policy trajectories [15]. Replay was prioritized by TD-error and when sampling the replay prioritization exponent was 1.0 [72]. We note that, as commonly done on Atari, we consider the discounted RL setting (rather than the finite-horizon setting used to derive our theoretical results). We refer to Table 2 for hyperparameter details.
Uncertainty measure in VAPOR-lite. For the uncertainty measure  $\hat{\sigma}$ , we use an ensemble of reward predictors. Specifically, we set it to the standard deviation of  $H = 10$  randomly initialized reward prediction heads  $\hat{r}^{(i)}$  with randomized prior functions [56], i.e.,
$$
\hat {\sigma} (s, a) = \min  \left(\sigma_ {\mathrm {s c a l e}} \operatorname {s t d} _ {1 \leq i \leq H} (\{\hat {r} ^ {(i)} (s, a) \}), 1\right),
$$
with  $\sigma_{\mathrm{scale}} > 0$  a scaling hyperparameter. We also added a small amount of noise to the rewards in the replay buffer for the targets for the reward prediction ensemble (specifically, 0.1 times an independent standard normal distribution) [14], to prevent the ensemble from potentially collapsing due to the use of replay data.
Figure 9: Per-game human normalized score (averaged over 5 seeds).
Figure 10: Median human normalized score across 57 Atari games (with standard errors across 5 seeds) of an optimistic replay actor-critic agent, for different choices of entropy regularization. VAPOR-lite's 'tuned, state-action' performs best.
Baselines. In Figure 4, we compare VAPOR-lite, whose policy loss may be condensely expressed as
$$
\max  _ {\pi \in \Pi} \left(\mu^ {\pi} \pi\right) ^ {\top} (r + \hat {\sigma}) + \left(\mu^ {\pi}\right) ^ {\top} \left(\mathcal {H} _ {\hat {\sigma}} (\pi)\right), \tag {14}
$$
to a standard actor-critic objective
$$
\max  _ {\pi \in \Pi} \left(\mu^ {\pi} \pi\right) ^ {\top} r, \tag {15}
$$
as well as to an actor-critic with fixed scalar entropy regularization
$$
\max  _ {\pi \in \Pi} \left(\mu^ {\pi} \pi\right) ^ {\top} r + \left(\mu^ {\pi}\right) ^ {\top} (\beta \mathcal {H} (\pi)), \tag {16}
$$
where  $\beta > 0$  is a hyperparameter (e.g.,  $\beta = 0.01$  in [42], which is also used in Figure 4 as it performed best). We later compare to additional baselines (Figure 10).
On entropy regularization for policy improvement/evaluation. We refer to [84] for in-depth discussion on the difference of using entropy for policy improvement and/or for policy evaluation in entropy-regularized RL. Theory prescribes both [87] and some algorithms such as Soft Actor-Critic [26] implement both, yet as remarked by [84], omitting entropy for policy evaluation (i.e., not having entropy as an intrinsic reward) tends to result in more stable and efficient learning. We also observed the same and thus only use entropy for policy improvement.
Related works. We mention here a couple of related works to our proposed algorithm. [22] argue that standard policy entropy regularization can be harmful when the RL problem contains actions that are rarely useful, and propose a method that uses mutual-information regularization to optimize a prior action distribution. [27] discuss the limitation of policy entropy regularization being 'sample-unaware', and propose to regularize with respect to the entropy of a weighted sum of the policy action distribution and the sample action distribution from the replay buffer.
Per-game results. Figure 9 reports the performance of the Replay Actor-Critic (RAC) baseline compared to RAC augmented with VAPOR-lite, on each Atari game. We see that VAPOR-lite yields improvements in many games, especially in hard-exploration games such as Montezuma's Revenge or Venture.
Ablation on entropy regularization. VAPOR-lite advocates for two ingredients: optimism and weighted entropy regularization. While the first has already been investigated in deep RL for various choices of uncertainty bonus [4], the second is novel so we focus our attention on it here. For a fair comparison, we augment all agents with the optimistic reward component, to isolate the effect of the adaptive state-action entropy regularization of VAPOR-lite. We consider the following policy losses for various choices of entropy regularization
none:  $\max_{\pi \in \Pi} (\mu^{\pi} \pi)^{\top} (r + \widehat{\sigma})$
fixed, scalar:  $\max_{\pi \in \Pi} (\mu^{\pi} \pi)^{\top} (r + \widehat{\sigma}) + (\mu^{\pi})^{\top} (\beta \mathcal{H}(\pi)),$
tuned, scalar [49]:  $\max_{\pi \in \Pi} \min_{\tau > 0} (\mu^{\pi} \pi)^{\top} (r + \hat{\sigma}^{2} / (2\tau)) + (\mu^{\pi})^{\top} (\tau \mathcal{H}(\pi)),$
tuned, state-action (VAPOR-lite):  $\max_{\pi \in \Pi} (\mu^{\pi} \pi)^{\top} (r + \hat{\sigma}) + (\mu^{\pi})^{\top} (\mathcal{H}_{\hat{\sigma}}(\pi))$ .
In particular, we have further compared to an optimistic actor-critic objective with tuned, scalar entropy regularization, specifically the epistemic-risk-seeking objective of [49] which corresponds to solving a principled saddle-point problem between a policy player and a scalar temperature player  $\tau$  parameterized by a neural network. We refer to Table 2 for hyperparameter details. We observe in Figure 10 the isolated benefits of VAPOR-lite's tuned, state-action entropy regularization.
Future directions. Below we list some relevant directions for future empirical investigation:
- Although our simple  $\hat{\sigma}$  uncertainty measure using an ensemble of reward predictors worked well, more sophisticated domain-specific uncertainty signals could be used [63, 64, 7].
- A first 'looseness' of VAPOR-lite with respect to VAPOR is that it sets the temperatures  $\tau$  to the uncertainties  $\hat{\sigma}$  (which yields the same  $\tilde{O}$  regret bound). An interesting extension for VAPOR-lite could be to optimize directly the temperatures by minimizing VAPOR's saddle-point problem with a state-action dependent temperature player  $\tau$  parameterized by an independent neural network.
- A second 'looseness' of VAPOR-lite with respect to VAPOR is that it bypasses the challenging optimization of the entropy of the state visitation distribution, by only (weight-)regularizing with the entropy of the policy. Various attempts have turned to density models or non-parametric entropy estimation [29, 36, 43], and it could be relevant to incorporate such techniques to get closer to the VAPOR objective.
- Our agent is relatively simple compared to modern state-of-the-art Atari agents since it is missing components like model-based rollouts, distributional heads, auxiliary tasks, etc. An interesting extension would be to incorporate the techniques discussed in this paper into the most effective policy-based agents.