SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] A. Abdolmaleki, J. T. Springenberg, Y. Tassa, R. Munos, N. Heess, and M. Riedmiller. Maximum a posteriori policy optimisation. arXiv preprint arXiv:1806.06920, 2018.
[2] P. Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research, 3(Nov):397-422, 2002.
[3] M. G. Azar, I. Osband, and R. Munos. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning, pages 263-272. PMLR, 2017.
[4] M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
[5] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[6] S. P. Boyd and L. Vandenberghe. Convex optimization. Cambridge university press, 2004.
[7] Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.12894, 2018.
[8] C. Bălcău. Weighted entropic perturbation for linear programming. Analele Universității București. Matematica-Informatica, 55, 01 2006.
[9] W. C. Cheung. Exploration-exploitation trade-off in reinforcement learning on online markov decision processes with global concave rewards. arXiv preprint arXiv:1905.06466, 2019.
[10] T. M. Cover and J. A. Thomas. Information theory and statistics. Elements of information theory, 1(1):279-335, 1991.
[11] S. Diamond and S. Boyd. CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research, 17(83):1-5, 2016.
[12] A. Domahidi, E. Chu, and S. Boyd. ECOS: An SOCP solver for embedded systems. In European Control Conference (ECC), pages 3071-3076, 2013.
[13] O. D. Domingues, P. Menard, E. Kaufmann, and M. Valko. Episodic reinforcement learning in finite mdps: Minimax lower bounds revisited. In Algorithmic Learning Theory, pages 578-598. PMLR, 2021.
[14] V. Dwaracherla, Z. Wen, I. Osband, X. Lu, S. M. Asghari, and B. Van Roy. Ensembles for uncertainty estimation: Benefits of prior functions and bootstrapping. arXiv preprint arXiv:2206.03633, 2022.
[15] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, pages 1407–1416. PMLR, 2018.
[16] S.-C. Fang and H.-S. J. Tsao. Linear programming with entropic perturbation. Zeitschrift für Operations Research, 37(2):171-186, 1993.
[17] M. Fellows, A. Mahajan, T. G. Rudner, and S. Whiteson. Virel: A variational inference framework for reinforcement learning. Advances in neural information processing systems, 32, 2019.
[18] C. W. Fox and S. J. Roberts. A tutorial on variational bayesian inference. Artificial intelligence review, 38:85-95, 2012.
[19] R. Fox, A. Pakman, and N. Tishby. Taming the noise in reinforcement learning via soft updates. arXiv preprint arXiv:1512.08562, 2015.
[82] M. J. Wainwright, M. I. Jordan, et al. Graphical models, exponential families, and variational inference. Foundations and Trends® in Machine Learning, 1(1-2):1-305, 2008.
[83] R. J. Williams and J. Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3(3):241-268, 1991.
[84] H. Yu, H. Zhang, and W. Xu. Do you need the entropy reward (in practice)? arXiv preprint arXiv:2201.12434, 2022.
[85] T. Zahavy, B. O'Donoghue, G. Desjardins, and S. Singh. Reward is enough for convex mdps. Advances in Neural Information Processing Systems, 34:25746-25759, 2021.
[86] Z. Zhang, Y. Zhou, and X. Ji. Almost optimal model-free reinforcement learning via reference-advantage decomposition. Advances in Neural Information Processing Systems, 33:15198–15207, 2020.
[87] B. D. Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010.
A Graphical Models 17
B Algorithmic Connections to  $\Gamma^*$  18
C Assumptions 18
D Limitations 19
E Properties of the VAPOR Optimization Problem 19
E.1 Dual Problem of the VAPOR Optimization Problem 19
E.2 Message Passing View of the VAPOR Optimization Problem 20
F Proofs 21
F.1 Proof of Lemma 1 21
F.2 Proof of Lemma 2 21
F.3 Proof of Lemma 3 22
F.4 Proof of Lemma 4 23
F.5 VAPOR without the Sub-Gaussian Assumption 23
F.6 Proof of Lemma 5 24
F.7 Proof of Theorem 1 and Corollary 1 25
F.8 Proof of Lemma 6 26
F.9 Proof of Lemma 7 28
F.10 Proof of Theorem 2 29
F.11 Proof of Lemma 8 29
F.12 Connection between K-learning and VAPOR 30
G Gaussian Sub-Gaussian Stochastic Optimism 31
H VAPOR-lite Analysis 33
I Experimental Details 35
I.1 Reward-Free GridWorld 35
I.2 DeepSea (tabular) 35
I.3 DeepSea (neural network) 36
I.4 Atari 36
(a) graphical model of the standard RL as inference' framework [37]
(b) graphical model for principled Bayesian 'RL as inference'
Figure 5: (a) In the standard 'RL as inference' framework, the binary random variables $^2$  of state-action optimality  $\Gamma^{\star}$  are independent and observed, equal one with probability proportional to the exponentiated reward, i.e.,  $\mathbb{P}(\Gamma_l^\star = 1)\propto \exp (\mathbf{r}_l)$ . This assumption is arbitrary and ignores the role of uncertainty and exploration. (b) In our model, only rewards  $\mathbf{r}$  are observed, with unobserved binary optimality variables  $\Gamma^{\star}$  and unobserved optimal values  $\mathbf{Q}^{\star}$ .  $\mathbf{Q}_l^{\star}$  depends on current reward  $\mathbf{r}_l$  and future  $\mathbf{Q}_{l + 1}^{\star}$ , and determines action optimality  $\pi_{l}^{\star}$ .  $\Gamma_l^\star$  depends on prior state-action optimality  $\Gamma_{l - 1}^{\star}$  and current action optimality  $\pi_l^\star$ . 'Optimality' thus propagates both in a backward way (via  $\mathbf{Q}_l^{\star}$ ) and a forward way (via  $\Gamma_l^{\star}$ ). Refraining from any modeling assumption on  $\Gamma^{\star}$ , we derive a variational approximation of the posterior probability of  $\Gamma^{\star}$  which yields an algorithm that explores efficiently.
A Graphical Models
We illustrate in Figure 5 the graphical models representing the standard<sup>1</sup> 'RL as inference' framework [37] and our principled, Bayesian 'RL as inference' framework. For ease of presentation, we consider here a simplified representation where we isolate a single trajectory  $(s_1, a_1, \ldots, s_4, a_4)$  and consider that  $\Gamma^{\star}$  does not encompass the randomness in the transition dynamics (which is e.g., the case for deterministic dynamics).
There exist other approaches to the RL as inference' perspective, such as the VIREL framework [17], that provide alternative benefits yet do not offer any principled guarantees either.
2In the main text, we use the terminology of events but note that events can be seen as binary random variables by taking the indicator function of the event.
Figure 6: VAPOR is an explicit, variational approximation of  $\mathbb{P}_{\Gamma^{\star}}$ , while Thompson sampling (TS) implicitly approximates  $\mathbb{P}_{\Gamma^{\star}}$  by sampling, similarly to the TS approximations Randomized value functions (RVFs), RLSVI [62]. A special case of VAPOR with equal temperatures approximately recovers K-learning and ERSAC [49]. We conjecture that there also exist connections between  $\Gamma^{\star}$  and information-theoretic approaches such as information-directed sampling (IDS), variance-IDS (V-IDS) [69, 28, 39].
B Algorithmic Connections to  $\Gamma^{\star}$
Figure 6 illustrates how numerous exploration algorithms can be linked to  $\Gamma^{\star}$ . This sheds a new light on existing algorithms, tightly connects them to our variational approach, and unifies these approaches within our principled Bayesian framework.
C Assumptions
In this section, we review and discuss the assumptions made throughout the paper. We stress that these assumptions are not required computationally for our variational Bayesian approach, but only for the analysis to obtain VAPOR's regret bound (Theorem 2). These are:
(A) Layered, time-inhomogeneous MDP,
(B) The reward noise is additive sub-Gaussian and the mean rewards are bounded almost surely with independent priors,
(C) The prior over transition functions is independent Dirichlet.
(A) This assumption is non-restrictive in the sense that any finite-horizon MDP with cycles can be transformed into an equivalent MDP without cycles by adding a timestep variable  $l \in [L]$  to the state space (picking up an additional factor of  $\sqrt{L}$  in the regret bound). It implies the transition function and the value function at the next state are conditionally independent, which is a technical property that simplifies the regret analysis.
(B) The sub-Gaussian assumption is standard in the literature and arises commonly in practice. For instance, it holds in any environment with bounded rewards (e.g., DeepSea, Atari). In Appendix F.3 (see Lemma 11) we extend Lemma 3 to the general case, and we refer to Appendix F.5 for the resulting VAPOR optimization problem (this would give a generic regret bound for the VAPOR learning algorithm which would depend on how fast the posteriors concentrate).
(C) The Dirichlet prior over the transition functions is the canonical prior for transitions in Bayesian RL, see [21]. This is because visiting a state-action pair simply increments by 1 the appropriate entry of the Dirichlet vector — note the ease of the update because the Dirichlet is the conjugate prior of the categorical distribution (which models transition probabilities of discrete-state-action-MDPs).
Note that we make the same assumptions as existing state-of-the-art Bayesian regret analyses, including PSRL [61], K-learning [47], RLSVI [62]. To the best of our knowledge, it is an open question how to obtain an algorithm with a  $\widetilde{O}(L\sqrt{SAT})$  Bayes regret bound without these assumptions.
<sup>3</sup>We point out that [61] originally considers time-homogeneous dynamics but there is a known mistake in the regret analysis, specifically in Lemma 3. We refer to e.g., these tutorial slides (see footnote of slide 65/91) and [67] for details. This issue can be fixed by assuming time-inhomogeneous dynamics.
We now discuss some limitations of our work, which constitute relevant directions for future investigation.
- The VAPOR convex optimization problem grows with  $L, S, A$ . Although large-scale exponential cone solvers exist that can handle this type of problem, it does not immediately lend itself to an online approximation.
- The regret guarantee of VAPOR relies on Assumptions 1 and 2 which, although standard (Appendix C), can be strong and it would be relevant to (partly) relax them.
- VAPOR-lite makes some upper-bound approximations of our variational optimization problem to use policy-gradient techniques, and there may be a tighter way to approximate VAPOR in the case where the policy is parameterized using a deep neural network that is updated using online stochastic gradients.
E Properties of the VAPOR Optimization Problem
The VAPOR optimization problem is an exponential cone program that can be solved efficiently using modern optimization methods [50, 12, 74, 46]. In Appendix E.1, we show that its dual is a convex optimization problem that is unconstrained in a 'value' variable and a 'temperature' variable. In Appendix E.2, we relate VAPOR to message passing, a popular technique to perform inference in graphical models [82], by showing that VAPOR can be cast as passing some forward and backward messages back and forth.
E.1 Dual Problem of the VAPOR Optimization Problem
We derive the dual problem of the VAPOR optimization problem (4) and show that it is a convex optimization problem in a 'value' variable  $V \in \mathbb{R}^{L,S} \coloneqq \{\mathbb{R}^{S_l}\}_{l=1}^L$  and a 'temperature' variable  $\tau \in \mathbb{R}_+^{L,S,A}$ . Even though the 'primal' admits flow constraints in  $\lambda$ , this dual is unconstrained in  $V$  (the only constraint is the non-negativity of  $\tau$ ). In the case of unknown  $P$  (Section 5), the dual is the same as in Lemma 9 except that  $P$  is replaced by  $\mathbb{E}_{\phi}P$  and the uncertainty  $\sigma$  is replaced by  $\hat{\sigma}$ .
Lemma 9. The  $\lambda^{*}\in \Lambda (P)$  solving the VAPOR optimization problem (4) is unique and satisfies
$$
\lambda_ {l} ^ {*} (s, a) = \exp \left(\frac {\delta_ {V ^ {*} , \tau^ {*} , l} (s , a)}{\tau_ {l} (s , a)} - 1\right),
$$
where we define the 'advantage' function
$$
\delta_ {V ^ {*}, \tau^ {*}, l} (s, a) = \mathbb {E} _ {\phi} r _ {l} (s, a) + \frac {\sigma_ {l} ^ {2} (s , a)}{2 \tau_ {l} ^ {*} (s , a)} + \sum_ {s ^ {\prime}} P _ {l} (s ^ {\prime} \mid s, a) V _ {l + 1} ^ {*} (s ^ {\prime}) - V _ {l} ^ {*} (s),
$$
and where  $V^{*},\tau^{*}$  minimize the (convex) dual function
$$
\min  _ {V \in \mathbb {R} ^ {L, S}} \min  _ {\tau \in \mathbb {R} _ {+} ^ {L, S, A}} \left\{\sum_ {s} \rho (s) V _ {1} (s) + \sum_ {l, s, a} \tau_ {l} (s, a) \exp \left(\frac {\delta_ {V , \tau , l} (s , a)}{\tau_ {l} (s , a)} - 1\right) \right\}. \tag {6}
$$
The induced policy  $\pi^{*}$  is uniquely defined as
$$
\pi_ {l} ^ {*} (s, a) = \frac {\exp \left(\frac {\delta_ {V ^ {*} , \tau^ {*} , l} (s , a)}{\tau_ {l} ^ {*} (s , a)} - 1\right)}{\sum_ {a ^ {\prime}} \exp \left(\frac {\delta_ {V ^ {*} , \tau^ {*} , l} (s , a ^ {\prime})}{\tau_ {l} ^ {*} (s , a ^ {\prime})} - 1\right)}
$$
if the denominator is positive, otherwise  $\pi_l^* (s,\cdot)$  can be any distribution.
Proof. The proof follows from applying the method of Lagrange multipliers to the VAPOR optimization problem using the definition of  $\mathcal{V}$  in (3). Derivations of dual problems of related optimization problems in the space of occupancy measures can be found in e.g., [65, 24]. Note that for a fixed temperature function  $\tau \in \mathbb{R}_{+}^{L,S,A}$ , VAPOR is an instance of linear programming with weighted entropic perturbation, which is a class of optimization problems shown to admit an unconstrained convex dual [8, 16].
Algorithm 2 Frank-Wolfe algorithm for the VAPOR optimization problem (Lemma 10)
Require: Accuracy level  $\varepsilon >0$    
Set the smoothing parameter  $\delta = \varepsilon /(\sigma_{\mathrm{max}}LSA)$    
Define  $\mathcal{V}_{\delta}(\lambda)\coloneqq \sum_{l,s,a}\lambda_l(s,a)\left[\mathbb{E}r_l(s,a) + \sigma_l(s,a)\sqrt{-2(\log(\lambda_l(s,a) + \delta) + \delta)}\right]$    
Set the number of iterations  $K = (\sigma_{\mathrm{max}}L\varepsilon^{-1})^{5}(SA)^{4}$    
Initialize any  $\lambda^{(0)}\in \Lambda$    
for iterations  $k = 1,2,\ldots ,K$  do Backward message pass: Compute the optimal policy  $\pi^{(k - 1)}$  for the reward  $\nabla \mathcal{V}_{\delta}(\lambda^{(k - 1)})$  Forward message pass: Compute  $d^{(k - 1)}$  the stationary state distribution induced by  $\pi^{(k - 1)}$  , i.e.,  $d_1^{(k - 1)}(s) = \rho (s),\quad d_{l + 1}^{(k - 1)}(s') = \sum_{s,a}P_l(s'\mid s,a)\pi_l^{(k - 1)}(s,a)d_l(s)$  Combine: Compute for step size  $\gamma_{k} = \frac{2}{k + 1}$  the occupancy measure  $\lambda_l^{(k)}(s,a) = (1 - \gamma_k)\lambda_l^{(k - 1)}(s,a) + \gamma_kd_l^{(k - 1)}(s)\pi_l^{(k - 1)}(s,a)$    
end for   
return  $\lambda^{(K)}$
E.2 Message Passing View of the VAPOR Optimization Problem
Message passing is a popular technique to perform inference in graphical models [82]. In fact, as shown in [37, Section 2.3], the standard 'RL as inference' framework (Section 2.1) can recover the optimal policy using standard sum-product message passing, relying on a single backward message pass. While VAPOR does not have such a simplified derivation, the next lemma shows that it can be cast as passing some forward and backward messages back and forth. In this view, a backward message amounts to computing the optimal policy for a given reward (via backward induction), and a forward message amounts to computing the associated stationary state distribution (via the flow equations). This view stems from iteratively solving VAPOR using the Frank-Wolfe (a.k.a. conditional gradient) algorithm, where an update is equivalent to solving an MDP [29, 77, 85].
Lemma 10. For any accuracy  $\varepsilon > 0$ , there exist forward-backward messages such that after  $\mathrm{poly}(S, A, L, \sigma_{\max}, \varepsilon^{-1})$  passes they output a  $\lambda^{MP} \in \Lambda(P)$  with  $\max_{\lambda \in \Lambda(P)} \mathcal{V}_{\phi}(\lambda) - \mathcal{V}_{\phi}(\lambda^{MP}) \leq \varepsilon$ .
Proof. We consider in Algorithm 2 the Frank-Wolfe algorithm applied to maximizing the smoothed proxy  $V_{\delta}$  over  $\lambda \in \Lambda$  (throughout the proof we write  $\Lambda = \Lambda(P)$  and omit the dependence on the beliefs  $\phi$  for ease of notation). Each iteration represents one forward-backward message pass: the backward message computes the optimal policy for the reward given by the gradient of the objective  $V_{\delta}$  evaluated at the current iterate (via backward induction), while the forward message computes the associated stationary state distribution (via the flow equations). Note that the chosen number of iterations  $K$  is polynomial in the inverse of the accuracy level  $\varepsilon^{-1}$ , the maximum uncertainty  $\sigma_{\max} \coloneqq \max_{l,s,a} \sigma_l(s,a)$  and the MDP parameters  $L, S, A$ . To prove the  $\varepsilon$ -convergence in the original objective  $\mathcal{V}$ , we decompose
$$
\begin{array}{l} \mathcal {V} \left(\lambda^ {*}\right) - \mathcal {V} \left(\lambda^ {(K)}\right) = \mathcal {V} \left(\lambda^ {*}\right) - \max  _ {\lambda \in \Lambda} V _ {\delta} (\lambda) + \max  _ {\lambda \in \Lambda} V _ {\delta} (\lambda) - \mathcal {V} _ {\delta} \left(\lambda^ {(K)}\right) + \mathcal {V} _ {\delta} \left(\lambda^ {(K)}\right) - \mathcal {V} \left(\lambda^ {(K)}\right) \\ \leq \underbrace {\mathcal {V} (\lambda^ {*}) - V _ {\delta} (\lambda^ {*})} _ {①} + \underbrace {\max  _ {\lambda \in \Lambda} V _ {\delta} (\lambda) - \mathcal {V} _ {\delta} (\lambda^ {(K)})} _ {②} + \underbrace {\mathcal {V} _ {\delta} (\lambda^ {(K)}) - \mathcal {V} (\lambda^ {(K)})} _ {③}. \\ \end{array}
$$
① and ③ can be bounded using the fact that for any  $\lambda \in \Lambda$ ,  $|\mathcal{V}_{\delta}(\lambda) - \mathcal{V}(\lambda)| \leq \sigma_{\max} LSA\delta$ . ② can be bounded from the Frank-Wolfe convergence rate for smooth objective functions [31], since the proxy  $\mathcal{V}_{\delta}$  is  $C_{\delta}$ -smooth by construction, with  $C_{\delta} = O(\sigma_{\max} \delta^{-4})$ . As a result, ②  $\leq M_{\delta} / (K + 2)$ , where  $M_{\delta}$  is the curvature constant of  $\mathcal{V}_{\delta}$  over  $\Lambda$ , with  $M_{\delta} \leq C_{\delta} \mathrm{diam}^2(\Lambda)$ , where  $\mathrm{diam}^2(\Lambda) \leq 2L$ . Therefore, choosing  $\delta = \varepsilon / (\sigma_{\max} LSA)$  and  $K = (\sigma_{\max} L\varepsilon^{-1})^5 (SA)^4$  gives
$$
\mathcal {V} \left(\lambda^ {*}\right) - \mathcal {V} \left(\lambda^ {(K)}\right) \leq O \left(\sigma_ {\max } L S A \delta + \frac {\sigma_ {\max } L}{\delta^ {4} K}\right) = O (\varepsilon).
$$
Proof. We omit the dependence on the beliefs  $\phi$  for ease of notation. At the initial step,
$$
\mathbb {P} \left(\Gamma_ {1} ^ {\star} (s, a)\right) = \mathbb {P} \left(\left\{s _ {1} = s \right\} \cap \left\{\pi_ {1} ^ {\star} (s) = a \right\}\right) = \rho (s) \mathbb {P} \left(\pi_ {1} ^ {\star} (s) = a\right),
$$
and from Bayes' rule and the fact that  $P$  is assumed known, we have the recursive equation
$$
\begin{array}{l} \mathbb {P} \left(\Gamma_ {l + 1} ^ {\star} \left(s ^ {\prime}, a ^ {\prime}\right)\right) = \mathbb {P} \left(\Gamma_ {l + 1} ^ {\star} \left(s ^ {\prime}\right) \cap \left\{\pi_ {l + 1} ^ {\star} \left(s ^ {\prime}\right) = a ^ {\prime} \right\}\right) \\ = \mathbb {P} \left(\pi_ {l + 1} ^ {\star} (s ^ {\prime}) = a ^ {\prime} \mid \Gamma_ {l + 1} ^ {\star} (s ^ {\prime})\right) \mathbb {P} \left(\Gamma_ {l + 1} ^ {\star} (s ^ {\prime})\right) \\ = \mathbb {P} \left(\pi_ {l + 1} ^ {\star} \left(s ^ {\prime}\right) = a ^ {\prime} \mid \Gamma_ {l + 1} ^ {\star} \left(s ^ {\prime}\right)\right) \sum_ {s, a} \mathbb {P} \left(\Gamma_ {l} ^ {\star} (s, a) \cap \left\{s _ {l + 1} = s ^ {\prime} \right\}\right) \\ = \mathbb {P} \left(\pi_ {l + 1} ^ {\star} (s ^ {\prime}) = a ^ {\prime} \mid \Gamma_ {l + 1} ^ {\star} (s ^ {\prime})\right) \sum_ {s, a} \mathbb {P} \left(\left\{s _ {l + 1} = s ^ {\prime} \right\} \mid \Gamma_ {l} ^ {\star} (s, a)\right) \mathbb {P} \left(\Gamma_ {l} ^ {\star} (s, a)\right) \\ = \mathbb {P} \left(\pi_ {l + 1} ^ {\star} \left(s ^ {\prime}\right) = a ^ {\prime} \mid \Gamma_ {l + 1} ^ {\star} \left(s ^ {\prime}\right)\right) \sum_ {s, a} P _ {l} \left(s ^ {\prime} \mid s, a\right) \mathbb {P} \left(\Gamma_ {l} ^ {\star} (s, a)\right). \tag {7} \\ \end{array}
$$
Summing (7) over  $a^\prime \in \mathcal{A}$  implies the non-negativity and flow conservation properties of  $\mathbb{P}_{\Gamma^{\star}}$  Moreover, when  $\sum_{a'\in \mathcal{A}}\mathbb{P}(\Gamma_l^\star (s,a')) > 0$  , it holds that
$$
\mathbb {P} \left(\pi_ {l} ^ {\star} (s) = a \mid \Gamma_ {l} ^ {\star} (s)\right) = \frac {\mathbb {P} \left(\Gamma_ {l} ^ {\star} (s , a)\right)}{\sum_ {a ^ {\prime} \in \mathcal {A}} \mathbb {P} \left(\Gamma_ {l} ^ {\star} (s , a ^ {\prime})\right)},
$$
which corresponds to (1).
F.2 Proof of Lemma 2
Proof. Denote by  $Z^{\star}$  the random variable of the total reward accumulated along a single rollout of the optimal policy  $\pi^{\star}$ , and let  $Z_{l}^{\star}(s)$  be the total reward accumulated by  $\pi^{\star}$  from state  $s$  at layer  $l$  and  $Z_{l}^{\star}(s, a)$  be the total reward accumulated by  $\pi^{\star}$  from state  $s$  at layer  $l$  after taking action  $a$ . We can write the total reward accumulated as
$$
Z ^ {\star} = \sum_ {s} Z _ {1} ^ {\star} (s) \mathbb {1} \left\{s _ {1} = s \right\} = \sum_ {s} Z _ {1} ^ {\star} (s) \mathbb {1} \left\{\Gamma_ {1} ^ {\star} (s) \right\}.
$$
Now for arbitrary  $l\in \{1,\ldots ,L - 1\}$
$$
\begin{array}{l} \sum_ {s} Z _ {l} ^ {\star} (s) \mathbb {1} \{\Gamma_ {l} ^ {\star} (s) \} = \sum_ {s, a} Z _ {l} ^ {\star} (s, a) \mathbb {1} \{\pi_ {l} ^ {\star} (s) = a \} \mathbb {1} \{\Gamma_ {l} ^ {\star} (s) \} \\ = \sum_ {s, a} Z _ {l} ^ {\star} (s, a) \mathbb {1} \left\{\Gamma_ {l} ^ {\star} (s, a) \right\} \\ = \sum_ {s, a} R _ {l} (s, a) \mathbb {1} \left\{\Gamma_ {1} ^ {\star} (s, a) \right\} + \sum_ {s, a} \mathbb {1} \left\{\Gamma_ {l} ^ {\star} (s, a) \right\} \sum_ {s ^ {\prime}} Z _ {l + 1} ^ {\star} (s ^ {\prime}) \mathbb {1} \left\{s _ {l + 1} = s ^ {\prime} \right\} \\ = \sum_ {s, a} R _ {l} (s, a) \mathbb {1} \left\{\Gamma_ {l} ^ {\star} (s, a) \right\} + \sum_ {s ^ {\prime}} Z _ {l + 1} ^ {\star} (s ^ {\prime}) \mathbb {1} \left\{\Gamma_ {l + 1} ^ {\star} (s ^ {\prime}) \right\}, \\ \end{array}
$$
and unrolling we obtain
$$
Z ^ {\star} = \sum_ {l, s, a} R _ {l} (s, a) \mathbb {1} \left\{\Gamma_ {l} ^ {\star} (s, a) \right\}.
$$
Denoting by  $\mathcal{F}_r$  the sigma-algebra generated by  $r$ , it holds from the tower property of conditional expectation that
$$
\begin{array}{l} \mathbb {E} _ {\phi} \left[ R _ {l} (s, a) \mathbb {1} \left\{\Gamma_ {l} ^ {\star} (s, a) \right\} \right] = \mathbb {E} _ {\phi} \left[ \mathbb {E} _ {\phi} \left[ R _ {l} (s, a) \mathbb {1} \left\{\Gamma_ {l} ^ {\star} (s, a) \right\} \mid \mathcal {F} _ {r} \right] \right] \\ = \mathbb {E} _ {\phi} \left[ \mathbb {E} _ {\phi} \left[ R _ {l} (s, a) \mid \mathcal {F} _ {r} \right] \mathbb {E} _ {\phi} \left[ \mathbb {1} \left\{\Gamma_ {l} ^ {\star} (s, a) \right\} \mid \mathcal {F} _ {r} \right] \right] \\ = \mathbb {E} _ {\phi} \left[ r _ {l} (s, a) \mathbb {1} \left\{\Gamma_ {l} ^ {\star} (s, a) \right\} \right]. \\ \end{array}
$$
$$
\mathbb {E} _ {s \sim \rho} \mathbb {E} _ {\phi} V ^ {\star} (s) = \sum_ {l, s, a} \mathbb {E} _ {\phi} \left[ r _ {l} (s, a) 1 \left\{\Gamma_ {l} ^ {\star} (s, a) \right\} \right].
$$