SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jean Tarbouriech
Google DeepMind jtarbouriech@google.com
Tor Lattimore  
Google DeepMind  
lattimore@google.com
Brendan O'Donoghue  
Google DeepMind  
bodonoghue@gmail.com
A popular perspective in Reinforcement learning (RL) casts the problem as probabilistic inference on a graphical model of the Markov decision process (MDP). The core object of study is the probability of each state-action pair being visited under the optimal policy. Previous approaches to approximate this quantity can be arbitrarily poor, leading to algorithms that do not implement genuine statistical inference and consequently do not perform well in challenging problems. In this work, we undertake a rigorous Bayesian treatment of the posterior probability of state-action optimality and clarify how it flows through the MDP. We first reveal that this quantity can indeed be used to generate a policy that explores efficiently, as measured by regret. Unfortunately, computing it is intractable, so we derive a new variational Bayesian approximation yielding a tractable convex optimization problem and establish that the resulting policy also explores efficiently. We call our approach VAPOR and show that it has strong connections to Thompson sampling, K-learning, and maximum entropy exploration. We conclude with some experiments demonstrating the performance advantage of a deep RL version of VAPOR.
Reinforcement learning (RL) is the problem of learning to control an unknown system by taking actions to maximize its cumulative reward through time [76]. As the agent navigates the environment, it receives noisy observations which it can use to update its (posterior) beliefs about the environment [21]. Unlike supervised learning, where the performance of an algorithm does not influence the data it will later observe, in RL the policy of the agent affects the data it will collect, which in turn affects the policy, and so on. As a result, an agent must sometimes take actions that lead to states where it has epistemic uncertainty about the value of those states, and sometimes take actions that lead to more certain payoff. The tension between these two modes is the exploration-exploitation trade-off [2, 35]. In this light, RL is a statistical inference problem wrapped in a control problem, and the two problems must be tackled simultaneously for good data efficiency.
It is natural to apply Bayesian probabilistic inference to the uncertain parameters in RL [21] and, since the goal of the agent is to find the optimal policy, a relevant object of study is the posterior probability of optimality for each state-action pair. In fact, the popular 'RL as inference' approach, most clearly summarized in the tutorial and review of [37], embeds the control problem into a graphical model by introducing optimality binary random variables that indicate whether a state-action is optimal. In the 'RL as inference' setup these optimality variables are assumed to be observed and take the value one with probability depending on the local reward. This surrogate potential seems like a peculiar and arbitrary choice, yet it has been widely accepted due to its mathematical convenience: for deterministic dynamics, the resulting probabilistic inference is equivalent to maximum entropy RL [80, 87]. However, this approximation leads to a critical shortcoming: The surrogate potential functions do not take into account the Bayesian (epistemic) uncertainty, and consequently do not perform genuine statistical inference on the unknowns in the MDP. This leads to 'posteriors' that are
in no way related to the true posteriors in the environment, and acting with those false posteriors leads to poor decision making and agents that require exponential time to solve even small problems [53].
Our contributions can be centered around a new quantity that we uncover as key for inference and control, denoted by  $\mathbb{P}_{\Gamma^{\star}}$ , which represents the posterior probability of each state-action pair being visited under the optimal policy.
- We reveal that  $\mathbb{P}_{\Gamma^{\star}}$  formalizes from a Bayesian perspective what it means for a 'state-action pair to be optimal', an event at the core of the RL as inference' framework which had never been properly analyzed.
- We establish that knowledge of  $\mathbb{P}_{\Gamma^{\star}}$  is sufficient to derive a policy that explores efficiently, as measured by regret (Section 3).
- Since computing  $\mathbb{P}_{\Gamma^{\star}}$  is intractable, we propose a variational optimization problem that tractably approximates it (Section 4).
- We first solve this optimization problem exactly, resulting in a new tabular model-based algorithm with a guaranteed regret bound (Section 5).
- We then solve a variant of this optimization problem using policy-gradient techniques, resulting in a new scalable model-free algorithm (Section 7).
- We show that both Thompson sampling [78, 75, 59, 71] and K-learning [47] can be directly linked to  $\mathbb{P}_{\Gamma^{\star}}$ , thus shedding a new light on these algorithms and tightly connecting them to our variational approach (Section 6).
- Our approach has the unique algorithmic feature of adaptively tuning optimism and entropy regularization for each state-action pair, which is empirically beneficial as our experiments on 'DeepSea' and Atari show (Section 8).
We model the RL environment as a finite state-action, time-inhomogeneous MDP given by the tuple  $\mathcal{M} \coloneqq \{\mathcal{S}, \mathcal{A}, L, P, R, \rho\}$ , where  $L$  is the horizon length,  $\mathcal{S} = \mathcal{S}_1 \cup \ldots \cup \mathcal{S}_L$  is the state space with cardinality  $S = \sum_{l=1}^{L} S_l$  where  $S_l = |\mathcal{S}_l|$ ,  $\mathcal{A}$  is the action space with  $A$  possible actions,  $P_l: \mathcal{S}_l \times \mathcal{A} \to \Delta(\mathcal{S}_{l+1})$  denotes the transition dynamics at step  $l$ ,  $R_l: \mathcal{S}_l \times \mathcal{A} \to \Delta(\mathbb{R})$  is the reward function at step  $l$  and  $\rho \in \Delta(\mathcal{S}_1)$  is the initial state distribution. Concretely, the initial state  $s_1 \in \mathcal{S}_1$  is sampled from  $\rho$ , then for steps  $l = 1, \ldots, L$  the agent is in state  $s_l \in \mathcal{S}_l$ , selects action  $a_l \in \mathcal{A}$ , receives a reward sampled from  $R_l(s_l, a_l)$  with mean  $r_l(s_l, a_l) \in \mathbb{R}$  and transitions to the next state  $s_{l+1} \in \mathcal{S}_{l+1}$  with probability  $P_l(s_{l+1} \mid s_l, a_l)$ . An agent following a policy  $\pi_l \in \Delta(\mathcal{A})^{S_l}$  at state  $s$  at step  $l$  selects action  $a$  with probability  $\pi_l(s, a)$ . The value functions are defined as
$$
Q _ {l} ^ {\pi} (s, a) = r _ {l} (s, a) + \sum_ {s ^ {\prime} \in \mathcal {S} _ {l + 1}} P _ {l} (s ^ {\prime} \mid s, a) V _ {l + 1} ^ {\pi} (s ^ {\prime}), \quad V _ {l} ^ {\pi} (s) = \sum_ {a \in \mathcal {A}} \pi_ {l} (s, a) Q _ {l} ^ {\pi} (s, a), \qquad V _ {L + 1} ^ {\pi} = 0,
$$
$$
Q _ {l} ^ {\star} (s, a) = r _ {l} (s, a) + \sum_ {s ^ {\prime} \in \mathcal {S} _ {l + 1}} P _ {l} (s ^ {\prime} \mid s, a) V _ {l + 1} ^ {\star} (s ^ {\prime}), \quad V _ {l} ^ {\star} (s) = \max _ {a \in \mathcal {A}} Q _ {l} ^ {\star} (s, a), \qquad \qquad V _ {L + 1} ^ {\star} = 0.
$$
An optimal policy  $\pi^{\star}$  satisfies  $V_{l}^{\pi^{\star}}(s) = V_{l}^{\star}(s)$  for each state  $s$  and step  $l$ . Let  $\{\pi_l^\star (s) = a\}$  be the event that action  $a$  is optimal for state  $s$  at step  $l$ , with ties broken arbitrarily so that only one action is optimal at each state. Let  $\mathbb{R}_+^{L,S,A} := \{\mathbb{R}_+^{S_l\times A}\}_{l = 1}^L$  be the set of functions  $\mathcal{S}_l\times \mathcal{A}\to \mathbb{R}_+$  for  $l\in [L]$ .
Occupancy measures. We denote the set of occupancy measures (a.k.a. stationary state-action distributions) with respect to the transition dynamics  $P$  as
$$
\Lambda (P) := \Big \{\lambda \in \mathbb {R} _ {+} ^ {L, S, A}: \sum_ {a} \lambda_ {1} (s, a) = \rho (s), \sum_ {a ^ {\prime}} \lambda_ {l + 1} (s ^ {\prime}, a ^ {\prime}) = \sum_ {s, a} P _ {l} (s ^ {\prime} \mid s, a) \lambda_ {l} (s, a) \Big \}.
$$
Note the correspondence between  $\Lambda(P)$  and the set of policies  $\Pi = \{\Delta(A)^{S_l}\}_{l=1}^L$  [66]. Any policy  $\pi \in \Pi$  induces  $\lambda^\pi \in \Lambda(P)$  such that  $\lambda_l^\pi(s, a)$  denotes the probability of reaching  $(s, a)$  at step  $l$  under  $\pi$ . Conversely, any  $\lambda \in \Lambda(P)$  induces  $\pi^\lambda \in \Pi$  given by  $\pi_l^\lambda(s, a) := \lambda_l(s, a) / (\sum_{a'} \lambda_l(s, a'))$  so long as  $\sum_{a'} \lambda_l(s, a') > 0$ ; otherwise  $\pi_l^\lambda(s, \cdot)$  can be any distribution, e.g., uniform.
Bayesian RL. We consider the Bayesian view of the RL problem, where the agent has some beliefs about the MDP represented by a distribution  $\phi$  on the space of all MDPs. We use the shorthand notation  $\mathbb{E}_{\phi}$  and  $\mathbb{P}_{\phi}$  to denote the expectation and probability under  $\phi$ . In the Bayesian view, the mean rewards  $r$  and transition dynamics  $P$  are random variables, and thus so are  $Q^{\star}, V^{\star}, \pi^{\star}$ .
2.1 Previous Approach to 'RL as Inference'
The popular line of research casting 'RL as inference' is most clearly summarized in the tutorial and review of Levine, 2018 [37]. It embeds the control problem into a graphical model, mirroring the dual relationship between optimal control and inference [81, 34, 68]. As shown in Figure 5a, this approach defines an additional optimality binary random variable denoted by  $\Gamma_l^{\star}$ , which indicates which state-action at timestep  $l$  is 'optimal', i.e., which state-action is visited under the (unknown) optimal policy. In these works the probability of  $\Gamma_l^{\star}$  is then modeled as proportional to the exponentiated reward.
Approximation 1 (Inference over exponentiated reward [37]). Denoting by  $\Gamma_l^\star(s, a)$  the event that state-action pair  $(s, a)$  is 'optimal' at timestep  $l$ , set  $\mathbb{P}(\Gamma_l^\star(s, a)) \propto \exp(r_l(s, a))$ .
Under Approximation 1, the optimality variables become observed variables in the graphical model, over which we can apply standard tools for inference. This approach has the advantage of being mathematically and computationally convenient. In particular, under deterministic dynamics it directly leads to RL algorithms with 'soft' Bellman updates and added entropy regularization, thus recovering the maximum-entropy RL framework [80, 87] and giving a natural, albeit heuristic exploration strategy. Many popular algorithms lie within this class [65, 25, 1] and have demonstrated strong empirical performance in domains where efficient exploration is not a bottleneck.
Despite its popularity and convenience, the potential function of Approximation 1 ignores epistemic uncertainty [53]. Due to this shortcoming, the inference procedure produces invalid and arbitrarily poor posteriors. It can be shown that the resulting algorithms fail even in basic bandit-like problems (e.g., [53, Problem 1]), as they underestimate the probability of optimal actions and take actions which have no probability of optimality under the true posterior. This implies that the resulting agents (e.g., Soft Q-learning [25]) can perform poorly in simple domains that require deep exploration [57]. The fact that the 'RL as inference' framework does not perform valid inference on the optimality variables leads us to consider how a proper Bayesian inference approach might proceed. This brings us to the first contribution of this manuscript.
3 A Principled Bayesian Inference Approach to RL
In this section, we provide a principled Bayesian treatment of statistical inference over 'optimality' variables in an MDP. Although it is the main quantity of interest of the popular 'RL as inference' framework, a rigorous definition of whether a state-action pair is optimal has remained elusive. We provide a recursive one below.
Definition 1 (State-action optimality). For any step  $l \in [L]$ , state-action  $(s, a) \in S_l \times \mathcal{A}$ , we define
$$
\Gamma_ {1} ^ {\star} (s) := \left\{s _ {1} = s \right\}, \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \Gamma_ {l + 1} ^ {\star} (s ^ {\prime}) := \bigcup_ {s \in \mathcal {S} _ {l}, a \in \mathcal {A}} \Gamma_ {l} ^ {\star} (s, a) \cap \left\{s _ {l + 1} = s ^ {\prime} \right\},
$$
$$
\Gamma_ {1} ^ {\star} (s, a) := \Gamma_ {1} ^ {\star} (s) \cap \{\pi_ {1} ^ {\star} (s) = a \}, \qquad \Gamma_ {l + 1} ^ {\star} (s ^ {\prime}, a ^ {\prime}) := \Gamma_ {l + 1} ^ {\star} (s ^ {\prime}) \cap \{\pi_ {l + 1} ^ {\star} (s ^ {\prime}) = a ^ {\prime} \}.
$$
In words,  $\Gamma_l^\star(s)$  is the random event that state  $s$  is reached at step  $l$  after executing optimal actions from steps 1 to  $l-1$ , and the event  $\Gamma_l^\star(s, a)$  further requires that the action  $a$  is optimal at state  $s$  in step  $l$ . Equipped with this definition, 'optimality' in an MDP flows both in a backward way (via action optimality  $\pi^\star$ , which can be computed using dynamic programming) and in a forward way (via state-action optimality  $\Gamma^\star$ ). We illustrate this bidirectional property in the simplified graphical model of Figure 5b (which isolates a single trajectory). In general, state-action optimality encapsulates all possible trajectories leading to a given state being optimal (and there may be exponentially many of them). Thanks to the MDP structure and Bayes' rule, we show in Lemma 1 that the posterior probability of state-action optimality lies in the space of occupancy measures. To simplify exposition, we first assume that the transition dynamics  $P$  are known (only rewards are unknown), and we extend our analysis to unknown  $P$  in Section 5.
Lemma 1. For known  $P$ , it holds that  $\mathbb{P}_{\Gamma^{\star}} := \{\mathbb{P}_{\phi}(\Gamma_{l}^{\star})\}_{l=1}^{L} \in \Lambda(P)$ .
The significance of Lemma 1 is that we can readily sample from  $\mathbb{P}_{\Gamma^{\star}}$  with the induced policy
$$
\pi_ {l} (s, a) = \frac {\mathbb {P} _ {\phi} \left(\Gamma_ {l} ^ {\star} (s , a)\right)}{\sum_ {a ^ {\prime} \in \mathcal {A}} \mathbb {P} _ {\phi} \left(\Gamma_ {l} ^ {\star} (s , a ^ {\prime})\right)} = \mathbb {P} _ {\phi} \left(\pi_ {l} ^ {\star} (s) = a \mid \Gamma_ {l} ^ {\star} (s)\right), \tag {1}
$$
Figure 1: Pair of MDPs  $\{\mathcal{M}^+, \mathcal{M}^-\}$  with uniform prior  $\phi = \left(\frac{1}{2}, \frac{1}{2}\right)$ .  $\mathcal{M}^+$  and  $\mathcal{M}^-$  only differ through their reward at state  $s_L$ , resp.  $+1$  and  $-1$ . Two actions are available at states  $s_1$  to  $s_{L-1}$ :  $\downarrow$  exits the chain and moves to an absorbing, zero-reward state, and  $\rightarrow$  moves right with small negative reward  $-\epsilon$ .
Policy	Agent starts at s1π1(s1,a) for a ∈ {↓,→}	Agent reaches sl, l > 1πl(sl,a) for a ∈ {↓,→}	Expected # episodes to reach sL
Bayes-optimal	(0,1)	(0,1)	1
Approximation 1 (e.g., Soft Q-Learning)	(0.5,0.5)	(0.5,0.5)	Ω(2^L)
Marginal Pφ(π*)	(0.5,0.5)	(0.5,0.5)	Ω(2^L)
Conditional Pφ(π* | Γ*)	(0.5,0.5)	(0,1)	2
Thompson Sampling	(0,1) w.p. 0.5	(0,1)	2
(1,0) w.p. 0.5
VAPOR	(0,1)	(0,1)	1
Table 1: Comparison of algorithm performance on MDP in Figure 1. Unprincipled approaches like Soft Q-learning take time exponential in  $L$  to reach the uncertain reward state. Naively using the marginal posterior probability of action optimality as the policy at each state also requires exponential time, however the policy derived from the posterior probability of state-action optimality  $\mathbb{P}_{\Gamma^{\star}}$  takes consistent actions and is therefore efficient. Similar good performance is achieved by Thompson sampling (implicit approximation of  $\mathbb{P}_{\Gamma^{\star}}$ , Section 6) and VAPOR (explicit approximation of  $\mathbb{P}_{\Gamma^{\star}}$ , Section 4).
which is the probability under the posterior that action  $a$  is optimal at state  $s$  conditioned on all actions taken before timestep  $l$  being optimal. It turns out that this policy explores efficiently, as measured by regret (see Corollary 1). Concretely, we have shown that the two facets of RL (inference and control) can be distilled into a single quantity:  $\mathbb{P}_{\Gamma^{\star}}$ . We use Bayesian inference to compute  $\mathbb{P}_{\Gamma^{\star}}$  and then use that to derive a good control policy. This is a principled and consistent Bayesian 'RL as inference' approach. The remaining challenge thus lies in computing  $\mathbb{P}_{\Gamma^{\star}}$ . Unfortunately, doing so involves computing several complicated integrals with respect to the posterior and is intractable in most cases. In Section 4, we introduce the second contribution of this manuscript, which is a way to approximate  $\mathbb{P}_{\Gamma^{\star}}$  using a computationally tractable variational approach.
3.1 Instructive Example
To gain intuition on the difference between action optimality (i.e.,  $\mathbb{P}_{\phi}(\pi^{\star}))$  and state-action optimality (i.e.,  $\mathbb{P}_{\Gamma^{\star}}$ ), we consider the simple decision problem of Figure 1. Each episode starts in the leftmost state and the agent can traverse a chain of states in order to reach the rightmost state, where the reward is always either  $+1$  or  $-1$  with equal probability. Once the agent reaches the rightmost state it will resolve its uncertainty. At each state the agent has the option to move  $\rightarrow$ , paying a cost of  $\epsilon$  where  $\epsilon L \ll 1$ , or move  $\downarrow$  with no cost. We see that there are only 2 consistent trajectories: either take action  $\downarrow$  at state  $s_1$  (optimal for  $\mathcal{M}^{-}$ ), or repeat actions  $\rightarrow$  to reach  $s_L$  (optimal for  $\mathcal{M}^{+}$ ). We detail how a selection of algorithms perform on this problem in Table 1.
It is worth examining in detail what has gone wrong for the policy that samples according to  $\mathbb{P}_{\phi}(\pi^{\star})$ . Even if the agent has traveled to the right a few times it will exit the chain with probability 0.5 at each new state, which leads to poor exploration. Taking action  $\rightarrow$  a few times and then exiting the chain has zero probability of being optimal under the posterior, so this sequence of actions is inconsistent with the posterior. Conditioning on the event  $\Gamma^{\star}(s)$  when at state  $s$  forces the policy to be consistent with the set of the beliefs that would get the agent to state  $s$  in the first place, which, as we prove later, yields deep exploration. In our chain example, the only belief that takes action  $\rightarrow$  is the one in which the end state has reward  $+1$ , so the only consistent policy conditioned on having taken  $\rightarrow$  once is to continue all the way to the end state. The same issue arises in Thompson sampling, where in order to achieve consistency the agent samples a policy from its posterior and keeps it fixed for the entire episode since resampling at every timestep leads to inefficient exploration [71]. As we shall show in Section 6, Thompson sampling is exactly equivalent to sampling from  $\mathbb{P}_{\Gamma^{\star}}$ . In conclusion:
action optimality:  $a_{l}\sim \mathbb{P}_{\phi}(\pi_{l}^{\star}(s_{l}) = \cdot)$ $\Rightarrow$  Poor exploration, state-action optimality:  $a_{l}\sim \mathbb{P}_{\phi}(\pi_{l}^{\star}(s_{l}) = \cdot |\Gamma_{l}^{\star}(s_{l}))\propto \mathbb{P}_{\phi}(\Gamma_{l}^{\star}(s_{l},\cdot))\Rightarrow$  Efficient exploration.
4 A Variational Bayesian Approach
Having access to  $\mathbb{P}_{\Gamma^{\star}}$  is sufficient to enable deep exploration, but computing this probability is intractable in general. In this section, we derive a variational, i.e., optimization-based, approach to approximate  $\mathbb{P}_{\Gamma^{\star}}$ . In most variational Bayesian approaches [18], the optimization objective is to find a surrogate probability measure that minimizes some dissimilarity metric (e.g., a KL-divergence) from the intractable probability measure of interest. Yet unlike standard variational inference techniques, we cannot minimize the KL-divergence to this distribution as we do not have access to samples. However, we do have a downstream control objective. That is to say, we are not simply interested in approximating the distribution, but also in our approximation doing well in the control problem. This insight provides an alternative objective that takes into account the rewards and uncertainties that the agent is likely to encounter. The optimal value function  $V^{\star}$  is a random variable under the beliefs  $\phi$ . Our next lemma relates the expected value of  $V^{\star}$  under  $\phi$  to our state-action optimality event  $\Gamma^{\star}$ .
Lemma 2. It holds that
$$
\mathbb {E} _ {s \sim \rho} \mathbb {E} _ {\phi} V _ {1} ^ {\star} (s) = \sum_ {l, s, a} \mathbb {P} _ {\phi} (\Gamma_ {l} ^ {\star} (s, a)) \mathbb {E} _ {\phi} \left[ r _ {l} (s, a) \mid \Gamma_ {l} ^ {\star} (s, a) \right].
$$
4.1 Information Theoretic Upper Bound
Lemma 2 depends on  $\mathbb{P}_{\Gamma^{\star}}$ , but the conditional expectation is not easy to deal with. To handle this, we upper bound each  $\mathbb{E}_{\phi}[r_l(s,a) \mid \Gamma_l^{\star}(s,a)]$  in Lemma 2 using tools from information theory. To simplify the exposition, we consider the following standard sub-Gaussian assumption [70, 61, 47] and we defer to Appendix F.3 the treatment of the general case. We say that  $X: \Omega \to \mathbb{R}$  is  $\upsilon$ -sub-Gaussian for  $\upsilon > 0$  if  $\mathbb{E} \exp(c(X - \mathbb{E}X)) \leq \exp(c^{2}\upsilon^{2}/2)$ , for all  $c \in \mathbb{R}$ .
Lemma 3. If  $r_l(s,a)$  is  $\sigma_l(s,a)$ -sub-Gaussian under  $\phi$  for  $l\in [L]$  and  $(s,a)\in S_l\times \mathcal{A}$ , then
$$
\begin{array}{l} \mathbb {E} _ {\phi} \left[ r _ {l} (s, a) \mid \Gamma_ {l} ^ {\star} (s, a) \right] \leq \mathbb {E} _ {\phi} r _ {l} (s, a) + \min  _ {\tau_ {l} (s, a) > 0} \left(\frac {\sigma_ {l} ^ {2} (s , a)}{2 \tau_ {l} (s , a)} - \tau_ {l} (s, a) \log \mathbb {P} _ {\phi} \left(\Gamma_ {l} ^ {\star} (s, a)\right)\right) \\ = \mathbb {E} _ {\phi} r _ {l} (s, a) + \sigma_ {l} (s, a) \sqrt {- 2 \log \mathbb {P} _ {\phi} (\Gamma_ {l} ^ {\star} (s , a))}. \\ \end{array}
$$
4.2 Optimization Problem
Finally, we combine Lemmas 2 and 3 to reveal a concave function in  $\mathbb{P}_{\Gamma^{\star}}$  that upper bounds the value function objective  $\mathbb{E}_{s\sim \rho}\mathbb{E}_{\phi}V_{1}^{\star}(s)$ . For any  $\tau \in \mathbb{R}_+^{L,S,A}$  and occupancy measure  $\lambda \in \Lambda (P)$ , we define the  $\tau$ -weighted entropy of  $\lambda$  (summed over steps  $l$ ) as  $\mathcal{H}_{\tau}(\lambda) := -\sum_{l,s,a}\tau_{l}(s,a)\lambda_{l}(s,a)\log \lambda_{l}(s,a)$ . In the following definition we take division and the square-root to be applied elementwise, and  $\circ$  denotes elementwise multiplication. With this we can define the following optimistic value functions
$$
\mathcal {V} _ {\phi} (\lambda , \tau) := \lambda^ {\top} \left(\mathbb {E} _ {\phi} r + \frac {\sigma^ {2}}{2 \tau}\right) + \mathcal {H} _ {\tau} (\lambda), \tag {2}
$$
$$
\mathcal {V} _ {\phi} (\lambda) := \min  _ {\tau \in \mathbb {R} _ {+} ^ {L, S, A}} \mathcal {V} _ {\phi} (\lambda , \tau) = \lambda^ {\top} \left(\mathbb {E} _ {\phi} r + \sigma \circ \sqrt {- 2 \log \lambda}\right). \tag {3}
$$
Lemma 4. For known  $P$  and  $\sigma$ -sub-Gaussian  $r$ , we have
$$
\mathbb {E} _ {s \sim \rho} \mathbb {E} _ {\phi} V _ {1} ^ {\star} (s) \leq \mathcal {V} _ {\phi} \left(\mathbb {P} _ {\Gamma^ {\star}}\right) \leq \underbrace {\max  _ {\lambda \in \Lambda (P)} \mathcal {V} _ {\phi} (\lambda)} _ {\text {V A P O R}}, \tag {4}
$$
where the VAPOR optimization problem is concave.
The above optimization problem is our variational objective, the solution of which yields an occupancy measure that approximates  $\mathbb{P}_{\Gamma^{\star}}$ . We call the approach 'VAPOR', for variational approximation of the posterior probability of optimality in RL. We discuss various properties of the optimization problem in Appendix E (its unconstrained dual problem and message passing interpretation).