SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof (Lemma D.17). Let  $\pi : \mathcal{S} \to \Delta(\mathcal{A})$  be explicitly given. First, we construct a single-action MDP  $M' = (\mathcal{S}, s_0, \{0\}, P', r', \gamma)$  whose optimal (and only) state-action function  $Q'$  is precisely the value function  $V^{\pi}$  of  $\pi$ .
Let the state space  $\mathcal{S}$  and initial state  $s_0$  remain the same but replace the action space with a singleton "0". We identify each new state-action pair with the state since only one action exists. Next, define the transition probability
$$
P ^ {\prime} (s _ {1} \mid s _ {0}) := \sum_ {a \in \mathcal {A} ^ {s _ {0}}} \pi (s _ {0}, a) \cdot P (s _ {1} \mid s _ {0}, a).
$$
We can simulate a sample from  $P'(\cdot \mid s_0)$  by sampling (for free) from the policy, say  $a \sim \pi(s_0)$ , and then sampling from the generative model  $s_1 \sim P(\cdot \mid s_0, a)$ . Note this costs a single sample from the generative model. In addition, define the reward function as the expected reward at state  $s_0$
$$
r ^ {\prime} (s _ {0}) := \sum_ {a \in \mathcal {A} ^ {s _ {0}}} \pi (s _ {0}, a) \cdot r (s _ {0}, a).
$$
This value can be directly computed given  $\pi, r$ . Finally, the discount factor  $\gamma$  remains the same.
We remark that  $Q_{M'}^*(s) = V_M^\pi(s)$  for all  $s \in S$  by construction. Thus the deterministic (non-replicable)  $Q$ -function estimation algorithm from Theorem D.2 is in fact a deterministic (non-replicable) algorithm that yields an estimate  $\widehat{V}^\pi$  of  $V^\pi$  such that with probability at least  $1 - \delta$ ,
$$
\begin{array}{l} \left\| \widehat {V} ^ {\pi} - V ^ {\pi} \right\| _ {\infty} \leq \frac {\varepsilon^ {\prime}}{2 N} \\ \varepsilon^ {\prime} := \frac {\varepsilon (\rho - 2 \delta)}{\rho + 1 - 2 \delta}. \\ \end{array}
$$
Moreover, the sample complexity is
$$
\tilde {O} \left(\frac {| \mathcal {S} | N ^ {2}}{(1 - \gamma) ^ {3} \varepsilon^ {2} \rho^ {2}} \log \frac {1}{\delta}\right)
$$
since the state-action space of  $M'$  has size  $|S|$ . Without loss of accuracy, assume we clip the estimates to lie in the feasible range  $[0, 1/(1 - \gamma)]$ .
In order to replicably estimate  $Q^{\pi}$ , we use the following relation
$$
Q ^ {\pi} (s, a) = r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ V ^ {\pi} \left(s ^ {\prime}\right) \right].
$$
Note that we only have access to an estimate  $\widehat{V}^{\pi}(s')$  and not  $V^{\pi}(s')$ , thus we instead estimate
$$
\bar {Q} ^ {\pi} (s, a) := r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ \widehat {V} ^ {\pi} \left(s ^ {\prime}\right) \right].
$$
But by Hölder's inequality, we are still guaranteed that with probability at least  $1 - \delta$ ,
$$
\begin{array}{l} \left. \left| \bar {Q} ^ {\pi} (s, a) - Q ^ {\pi} (s, a) \right| \leq \gamma \| P (\cdot | s, a) \| _ {1} \cdot \left\| \widehat {V} ^ {\pi} - V ^ {\pi} \right\| _ {\infty} \right. \\ = \gamma \left\| \widehat {V} ^ {\pi} - V ^ {\pi} \right\| _ {\infty} \\ \leq \frac {\varepsilon^ {\prime}}{2 N} \\ \end{array}
$$
for all  $s\in \mathcal{S},a\in \mathcal{A}^s$
Each expectation of the form  $\mathbb{E}_{s' \sim P(\cdot \mid s, a)}\left[\widehat{V}^{\pi}(s')\right]$  is simply a bounded statistical query. By an Hoeffding bound, drawing  $m$  samples from the generative model  $s_j' \sim P(\cdot \mid s, a)$  implies
$$
\mathbb {P} \left\{\left| \frac {1}{m} \sum_ {j = 1} ^ {m} \widehat {V} ^ {\pi} \left(s _ {j} ^ {\prime}\right) - \mathbb {E} _ {s ^ {\prime}} \left[ \widehat {V} ^ {\pi} \left(s ^ {\prime}\right) \right] \right| > \frac {\varepsilon^ {\prime}}{2 N} \right\} \leq 2 \exp \left(- \frac {2 m (1 - \gamma) ^ {2} \varepsilon^ {\prime 2}}{4 N ^ {2}}\right).
$$
Thus with
$$
m = \tilde {O} \left(\frac {N ^ {2}}{(1 - \gamma) ^ {2} \varepsilon^ {\prime 2}} \ln \frac {1}{\delta}\right)
$$
trials, the empirical average estimates a single query  $\mathbb{E}_{s' \sim P(\cdot | s, a)}\left[\widehat{V}^{\pi}(s')\right]$  to an accuracy of  $\varepsilon'/2N$  with probability at least  $1 - \delta$ . The total number of samples is thus  $Nm$ .
Combining the  $V^{\pi}$  estimation and Hoeffding bound yields an estimate  $\bar{Q}^{\pi}$  such that
$$
\left\| \bar {Q} ^ {\pi} - Q ^ {\pi} \right\| \leq \frac {\varepsilon^ {\prime}}{N}
$$
with probability at least  $1 - 2\delta$ . Thus we can use the locally random  $\rho$ -replicable rounding scheme from Theorem C.2 to compute an estimate  $\widehat{Q}^{\pi}$  such that with probability at least  $1 - 2\delta$
$$
\left\| \widehat {Q} ^ {\pi} - Q ^ {\pi} \right\| _ {\infty} \leq \varepsilon .
$$
All in all, we described an algorithm that replicably estimates  $Q^{\pi}$  given a policy. The algorithm is locally random since the only internal randomness used to estimate  $Q^{\pi}(s,a)$  occurs at the last locally replicable rounding step. Finally, the total sample complexity of this algorithm is
$$
\tilde {O} \left(\left(\frac {| \mathcal {S} | N ^ {2}}{(1 - \gamma) ^ {3} \varepsilon^ {2} \rho^ {2}} + \frac {N ^ {3}}{(1 - \gamma) ^ {2} \varepsilon^ {2} \rho^ {2}}\right) \log \frac {1}{\delta}\right)
$$
With Lemma D.17 in hand, we are now ready to prove Theorem D.11, which reduces replicable policy estimation to replicable  $Q$ -function estimation. We restate the result below for convenience.
Proof (Theorem D.11). Run  $\mathcal{A}$  to output a  $\rho$ -replicable policy  $\pi$  such that
$$
\left| V ^ {\pi} (s) - V ^ {\star} (s) \right| \leq \varepsilon
$$
with probability at least  $1 - \delta / 12$  for each  $s \in S$ .
Applying the algorithm from Lemma D.17 with replicability parameter  $\rho$ , failure rate  $\delta / 12$ , and  $\pi$  as input yields some estimate  $\widehat{Q}^{\pi}$  such that
$$
\left| \widehat {Q} ^ {\pi} (s, a) - Q ^ {\pi} (s, a) \right| \leq \varepsilon
$$
with probability at least  $1 - \delta / 12$  for each  $s \in S, a \in \mathcal{A}^s$ .
Using the relationship
$$
Q ^ {\pi} (s, a) = r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} [ V ^ {\pi} (s) ],
$$
the triangle inequality, as well as a union bound, we realize that
$$
\left| \widehat {Q} ^ {\pi} (s, a) - Q ^ {\star} (s, a) \right| \leq 2 \varepsilon
$$
with probability at least  $1 - \delta /6$  for each  $s\in S,a\in \mathcal{A}^s$ . Moreover,  $\pi$  is identical across two executions with probability at least  $1 - \rho$  by assumption and thus  $\widehat{Q}^{\pi}$  will be identical across two executions with probability at least  $1 - 2\rho$ .
Remark that the scheme we derived above is a locally random  $2\rho$ -replicable algorithm for  $Q$ -estimation. It is locally random since the only source of internal randomness comes from the locally random subroutine in Lemma D.17. Thus the algorithm satisfies the conditions of Theorem D.9 and the lower bound from that theorem applies. In particular, if  $m$  denotes the sample complexity of  $\mathcal{A}$ , then
$$
m + \tilde {o} \left(\frac {N ^ {3}}{(1 - \gamma) ^ {3} \varepsilon^ {2} \rho^ {2}}\right) \geq \Omega \left(\frac {N ^ {3}}{(1 - \gamma) ^ {3} \varepsilon^ {2} \rho^ {2}}\right).
$$
The LHS is the sample complexity of the scheme we derived courtesy of Lemma D.17 and the RHS is the lower bound from Theorem D.9. It follows that
$$
m \geq \Omega \left(\frac {N ^ {3}}{(1 - \gamma) ^ {3} \varepsilon^ {2} \rho^ {2}}\right)
$$
E Omitted Details from Section 4
We now restate and prove Theorem 4.2, a  $\rho$ -TV indistinguishable SQ oracle for multiple queries.
Proof (Theorem 4.2). We first argue about the correctness of our algorithm. By assumption, with probability at least  $1 - \delta /2$ , the oracles provide estimates  $\widehat{\mu}_i$  's such that
$$
\max  _ {i \in [ d ]} | \widehat {\mu} _ {i} - v _ {i} | \leq \frac {\varepsilon \rho}{2 \sqrt {8 d \cdot \log (4 d / \delta)}} \leq \frac {\varepsilon}{2}.
$$
We call this event  $\mathcal{E}_1$ . Fix some  $i \in [d]$  and consider the second step where we add Gaussian noise  $\mathcal{N}(0, \varepsilon^2)$  to the estimate  $\widehat{\mu}_i$  to obtain noisy estimates  $\widehat{v}_i$ 's. From standard concentration bounds, we know that for  $X \sim \mathcal{N}(\mu, \sigma^2), r > 0$
$$
\mathbb {P} \left\{\left| X - \mu \right| > r \sigma \right\} \leq 2 e ^ {- r ^ {2} / 2}.
$$
Plugging in the values
$$
\sigma^ {2} = \frac {\varepsilon^ {2}}{8 \log (4 d / \delta)} \mathrm {,} r = \sqrt {2 \log \left(\frac {4 d}{\delta}\right)},
$$
$$
\mathbb {P} \{\left| \widehat {v} _ {i} - \widehat {\mu} _ {i} \right| > \varepsilon / 2 \} \leq \frac {\delta}{2 d}.
$$
Thus, taking a union bound over  $i \in [d]$  we have that
$$
\mathbb {P} \left\{\max  _ {i \in [ d ]} | \widehat {v} _ {i} - \widehat {\mu} _ {i} | > \varepsilon / 2 \right\} > \frac {\delta}{2}.
$$
We call this event  $\mathcal{E}_2$ . By taking a union bound over  $\mathcal{E}_1, \mathcal{E}_2$  and applying the triangle inequality, we see that
$$
\mathbb {P} \bigg \{\max _ {i \in [ d ]} | \widehat {v} _ {i} - v _ {i} | > \varepsilon \bigg \} \leq \delta .
$$
We now move on to prove the  $\rho$ -TV indistinguishability guarantees. Consider two executions of the algorithm and let  $\widehat{\mu}^{(1)}, \widehat{\mu}^{(2)} \in \mathbb{R}^d$  be the estimates after the first step of the algorithm. We know that, with probability at least  $1 - \delta$  over the calls to the SQ oracle across the two executions it holds that
$$
\left\| \widehat {\mu} _ {1} - \widehat {\mu} _ {2} \right\| _ {\infty} \leq \frac {\varepsilon \rho}{\sqrt {8 d \cdot \log (4 d / \delta)}}.
$$
We call this event  $\mathcal{E}_3$  and we condition on it for the rest of the proof. Standard computations [Gupta, 2020] reveal that the KL-divergence between two  $d$ -dimensional Gaussians  $p = \mathcal{N}(\mu_p,\Sigma_p),q = \mathcal{N}(\mu_q,\Sigma_q)$  can be written as
$$
D _ {\mathrm {K L}} (p \| q) = \frac {1}{2} \left(\log \frac {| \Sigma_ {q} |}{| \Sigma_ {p} |} - d + (\mu_ {p} - \mu_ {q}) \Sigma_ {q} ^ {- 1} (\mu_ {p} - \mu_ {q}) + \operatorname {t r a c e} (\Sigma_ {q} ^ {- 1} \Sigma_ {p})\right)
$$
In our setting we have
$$
p = \mathcal {N} \left(\widehat {\mu} ^ {(1)}, \frac {\varepsilon^ {2}}{8 \log (4 d / \delta)} \cdot I _ {d}\right), \qquad q = \mathcal {N} \left(\widehat {\mu} ^ {(2)}, \frac {\varepsilon^ {2}}{8 \log (4 d / \delta)} \cdot I _ {d}\right).
$$
Plugging these in we have that
$$
D _ {\mathrm {K L}} (p \| q) = \frac {8 \log (4 d / \delta)}{\varepsilon^ {2}} \left\| \widehat {\mu} ^ {(1)} - \widehat {\mu} ^ {(2)} \right\| _ {2} ^ {2} \leq \rho^ {2}.
$$
Thus, using Pinsker's inequality we see that
$$
d _ {\mathrm {T V}} (p, q) \leq \frac {\rho}{\sqrt {2}}
$$
Hence, we can bound the expected TV-distance as
$$
\mathbb {E} \left[ d _ {\mathrm {T V}} (p, q) \right] \leq \frac {\rho}{\sqrt {2}} + \delta <   \rho .
$$
Next, we prove Theorem 4.8, an implementation of the  $\rho$ -TV indistinguishable SQ oracle for multiple queries whose internal randomness is designed in a way that also ensures  $\rho$ -replicability. The result is restated below for convenience.
Proof (Theorem 4.8). Consider a call to Algorithm A.1 with TV indistinguishability parameter  $\rho /2$ . Let us call this algorithm  $\mathcal{A}$ . Notice that given a particular sample,  $\mathcal{A}$  is Gaussian and therefore absolutely continuous with respect to the Lebesgue measure. Let  $\mathcal{P}$  be the Lebesgue measure over  $\mathbb{R}^d$ . Let  $\mathcal{R}$  be a Poisson point process with intensity  $\mathcal{P} \times \mathrm{Leb} \times \mathrm{Leb}$ , where  $\mathrm{Leb}$  is the Lebesgue measure over  $\mathbb{R}_+$  (cf. Theorem C.4).
The learning rule  $\mathcal{A}'$  is defined as in Algorithm A.2. For every input  $S$  of  $\mathcal{A}$ , let  $f_S$  be the conditional density of  $\mathcal{A}$  given the input  $S$ , let  $r := \{(x_i, y_i, t_i)\}_{i \in \mathbb{N}}$  be an infinite sequence of the Poisson point process  $\mathcal{R}$ , and let  $i^{\star} := \arg \min_{i \in \mathbb{N}} \{t_i : f_S(x_i) > y_i\}$ . In words, we consider the pdf of the output conditioned on the input, an infinite sequence drawn from the described Poisson point process, and we focus on the points of this sequence whose  $y$ -coordinate falls below the curve. The output of  $\mathcal{A}'$  is  $x_i^{\star}$  and we denote it by  $\mathcal{A}'(S; r)$ .
We will shortly explain why this is well-defined, except for a measure zero event. The fact that  $\mathcal{A}'$  is equivalent to  $\mathcal{A}$  follows from the coupling guarantees of this process (cf. Theorem C.4), applied to the single random vector  $\mathcal{A}(S)$ . We can now observe that, except for a measure zero event, (i) since  $\mathcal{A}$  is absolutely continuous with respect to  $\mathcal{P}$ , there exists such a density  $f_{S}$ , (ii) the set over which we are taking the minimum is not empty, (iii) the minimum is attained at a unique point. This means
that  $\mathcal{A}'$  is well-defined, except on an event of measure zero $^8$ , and by the correctness of the rejection sampling process [Angel and Spinka, 2019],  $\mathcal{A}'(S)$  has the desired probability distribution.
We now prove that  $\mathcal{A}'$  is replicable. Since  $\mathcal{A}$  is  $\rho/2$ -TV indistinguishable, it follows that
$$
\mathbb {E} _ {S, S ^ {\prime}} \left[ d _ {\mathrm {T V}} \left(\mathcal {A} (S), \mathcal {A} \left(S ^ {\prime}\right)\right) \right] \leq \rho / 2.
$$
We have shown that  $\mathcal{A}'$  is equivalent to  $\mathcal{A}$ , so we can see that  $\mathbb{E}_{S,S'}[d_{\mathrm{TV}}(\mathcal{A}'(S),\mathcal{A}'(S'))]\leq \rho /2$ . Thus, using the guarantees of Theorem C.4, we have that for any datasets  $S,S'$
$$
\mathbb {P} _ {r \sim \mathcal {R}} \{\mathcal {A} ^ {\prime} (S; r) \neq \mathcal {A} ^ {\prime} (S ^ {\prime}; r) \} \leq \frac {2 d _ {\mathrm {T V}} (\mathcal {A} ^ {\prime} (S) , \mathcal {A} ^ {\prime} (S ^ {\prime}))}{1 + d _ {\mathrm {T V}} (\mathcal {A} ^ {\prime} (S) , \mathcal {A} ^ {\prime} (S ^ {\prime}))}.
$$
By taking the expectation over  $S, S'$ , we get that
$$
\begin{array}{l} \mathbb {E} _ {S, S ^ {\prime}} \left[ \mathbb {P} _ {r \sim \mathcal {R}} \{\mathcal {A} ^ {\prime} (S, r) \neq \mathcal {A} ^ {\prime} (S ^ {\prime}, r) \} \right] \leq \mathbb {E} _ {S, S ^ {\prime}} \left[ \frac {2 d _ {\mathrm {T V}} (\mathcal {A} ^ {\prime} (S) , \mathcal {A} ^ {\prime} (S ^ {\prime}))}{1 + d _ {\mathrm {T V}} (\mathcal {A} ^ {\prime} (S) , \mathcal {A} ^ {\prime} (S ^ {\prime}))} \right] \\ \leq \frac {2 \mathbb {E} _ {S , S ^ {\prime}} [ d _ {\mathrm {T V}} (\mathcal {A} ^ {\prime} (S) , \mathcal {A} ^ {\prime} (S ^ {\prime})) ]}{1 + \mathbb {E} _ {S , S ^ {\prime}} [ d _ {\mathrm {T V}} (\mathcal {A} ^ {\prime} (S) , \mathcal {A} ^ {\prime} (S ^ {\prime})) ]} \\ \leq \frac {\rho}{1 + \rho / 2} \\ \leq \rho , \\ \end{array}
$$
where the first inequality follows from Theorem C.4 and taking the expectation over  $S, S'$ , the second inequality follows from Jensen's inequality, and the third inequality follows from the fact that  $f(x) = 2x / (1 + x)$  is increasing. Now notice that since the source of randomness  $\mathcal{R}$  is independent of  $S, S'$ , we have that
$$
\mathbb {E} _ {S, S ^ {\prime}} \left[ \mathbb {P} _ {r \sim \mathcal {R}} \left\{\mathcal {A} ^ {\prime} (S; r) \neq \mathcal {A} ^ {\prime} (S ^ {\prime}; r) \right\} \right] = \mathbb {P} _ {S, S ^ {\prime}, r \sim \mathcal {R}} \left\{\mathcal {A} ^ {\prime} (S; r) \neq \mathcal {A} ^ {\prime} (S ^ {\prime}; r) \right\}.
$$
$$
\mathbb {P} _ {S, S ^ {\prime}, r \sim \mathcal {R}} \left\{\mathcal {A} ^ {\prime} (S; r) \neq \mathcal {A} ^ {\prime} (S ^ {\prime}; r) \right\} \leq \rho ,
$$
and so the algorithm  $\mathcal{A}'$  is  $\rho$ -replicable, concluding the proof.
Remark E.1. The coupling we described in the previous proof is not computable, since it requires an infinite sequence of samples. However, we can execute it approximately in the following way. We can first truncate the tails of the distribution and consider a large enough  $d$ -dimensional box that encloses the pdf of the truncated distribution. Then, by sampling  $\widetilde{O}(\exp(d))$  many points we can guarantee that, with high probability, there will be at least one that falls below the pdf. Even though this approximate coupling is computable, it still requires exponential time.
Remark E.2 (Coordinate-Wise Coupling). Since our algorithms add independent Gaussian noise to each of the estimates, a first approach to achieve the coupling using only shared randomness would be to construct a pairwise coupling between each estimate. In the context of multiple statistical query estimation, this would mean that we couple the estimate of the  $i$ -th query in the first execution with the estimate of the  $i$ -th query in the second execution. Unfortunately, even though this coupling is computationally efficient to implement it does not give us the desired sample complexity guarantees. To see that, notice that when the TV-distance of estimates across each coordinate is  $O(\rho)$ , under this pairwise coupling the probability that at least one of the estimates will be different across the two executions is  $O(d \cdot \rho)$ . However, the TV distance of the  $d$ -dimensional Gaussians is  $O(\sqrt{d} \cdot \rho)$  and this is the reason why the more complicated coupling we propose achieves better sample complexity guarantees. Our results reaffirm the observation that was made by Kalavasis et al. [2023] that the replicability property and the sample complexity of an algorithm are heavily tied to the implementation of its internal randomness, which can lead to a substantial computational overhead.
Under the measure zero event that at least one of these three conditions does not hold, we let  $\mathcal{A}'(S; r)$  be some arbitrary  $d$ -dimensional vector in  $[0, 1]^d$ .
F Omitted Details from Section 5
We now fill in the missing technical details from Section 5, which proposes an approximately replicable algorithm for optimal policy estimation.
Algorithm F.1 Approximately Replicable Policy Estimation
1:  $\delta \gets \min \{\delta, \rho_2 / 2\}$ .
2: Run Sublinear Randomized QVI [Sidford et al., 2018a] with confidence  $\delta$ , discount factor  $\gamma$ , and error  $\rho_1 \cdot \varepsilon \cdot (1 - \gamma) / (8 \log(|A|))$ .
3: Let  $\widehat{Q}$  be the output of the previous step.
4:  $\lambda \gets \frac{\log(|\mathcal{A}|)}{\varepsilon / 2 \cdot (1 - \gamma)}$ .
5: For every  $s \in S$ , let  $\pi(s, a) = \frac{\exp\left\{\lambda \widehat{Q}(s, a)\right\}}{\sum_{a' \in A} \exp\left\{\lambda \widehat{Q}(s, a')\right\}}$ .
6: Output  $\pi$ .
F.1 Different Soft-Max Rules
Instead of using the exponential soft-max rule we described in Section 5, one could use a more sophisticated soft-max rule like the piecewise-linear soft-max rule that was developed in Epasto et al. [2020]. The main advantage of this approach is that it achieves a worst-case  $\varepsilon$ -approximation, i.e., it never picks an action that will lead to a cumulative reward that is  $\varepsilon$  worse than the optimal one (cf. Definition 5.3). We also remark that this leads to sparse policies when there is only a small number of near-optimal actions.
F.2 Deferred Proofs
F2.1 Proof of Lemma 5.5
We now restate and prove Lemma 5.5, a useful result which quantifies the performance of the soft-max policy in comparison to the optimal policy.
Proof (Lemma 5.5). Using the guarantees of Lemma 5.4 we have that
$$
\max  _ {a \in \mathcal {A}} \widehat {Q} (s, a) \leq \sum_ {a \in \mathcal {A}} \widehat {\pi} (\bar {s}, a) \cdot \widehat {Q} (s, a) + \varepsilon_ {2}.
$$
We can assume without loss of generality that  $\pi^{\star}$  is a deterministic policy due to the fundamental theorem of RL. Fix an arbitrary  $s\in S$ . By the fact that  $\max_{s\in S,a\in \mathcal{A}}|\widehat{Q} (s,a) - Q^{\star}(s,a)|\leq \varepsilon_1$ , we see that
$$
\begin{array}{l} Q ^ {\star} (s, \pi (s)) \leq \max  _ {a \in \mathcal {A}} \widehat {Q} (s, a) + \varepsilon_ {1} \\ \leq \sum_ {a \in \mathcal {A}} \widehat {\pi} (\bar {s}, a) \cdot \widehat {Q} (s, a) + \varepsilon_ {1} + \varepsilon_ {2} \\ \leq \sum_ {a \in \mathcal {A}} \widehat {\pi} (\bar {s}, a) \cdot Q ^ {\star} (s, a) + 2 \varepsilon_ {1} + \varepsilon_ {2} \\ \end{array}
$$
An equivalent way to write the previous inequality is
$$
\begin{array}{l} r (s, \pi^ {\star} (s)) + \gamma \sum_ {s ^ {\prime} \in \mathcal {S}} V ^ {\star} (s ^ {\prime}) \cdot P _ {M} (s ^ {\prime} | s, \pi^ {\star} (s)) \\ \leq \sum_ {a \in \mathcal {A}} \widehat {\pi} (s, a) \cdot \left(r (s, a) + \gamma \sum_ {s ^ {\prime} \in \mathcal {S}} V ^ {\star} (s ^ {\prime}) \cdot P _ {M} (s ^ {\prime} | s, a)\right) + 2 \varepsilon_ {1} + \varepsilon_ {2} \\ \end{array}
$$
$$
\begin{array}{l} r (s, \pi^ {\star} (s)) - \sum_ {a \in \mathcal {A}} \widehat {\pi} (s, a) \cdot r (s, a) \\ \leq \gamma \left(\sum_ {a \in \mathcal {A}} \widehat {\pi} (s, a) \cdot \left(\sum_ {s ^ {\prime} \in \mathcal {S}} V ^ {\star} \left(s ^ {\prime}\right) \cdot P _ {M} \left(s ^ {\prime} \mid s, a\right)\right) - \sum_ {s ^ {\prime} \in \mathcal {S}} V ^ {\star} \left(s ^ {\prime}\right) \cdot P _ {M} \left(s ^ {\prime} \mid s, \pi^ {\star} (s)\right)\right) \\ + 2 \varepsilon_ {1} + \varepsilon_ {2}. \tag {1} \\ \end{array}
$$
Let  $\bar{s}\in \arg \max_{s\in S}V^{\star}(s) - V^{\widehat{\pi}}(s)$ . Then, we have that
$$
\begin{array}{l} V ^ {\star} (\bar {s}) - V ^ {\widehat {\pi}} (\bar {s}) \\ \leq r (\bar {s}, \pi^ {\star} (\bar {s})) - \sum_ {a \in \mathcal {A}} \widehat {\pi} (\bar {s}, a) \cdot r (\bar {s}, a) \\ + \gamma \left(\sum_ {s ^ {\prime} \in \mathcal {S}} V ^ {\star} (s ^ {\prime}) \cdot P _ {M} (s ^ {\prime} | \bar {s}, \pi^ {\star} (\bar {s})) - \sum_ {a \in \mathcal {A}} \widehat {\pi} (s, a) \cdot \left(\sum_ {s ^ {\prime} \in \mathcal {S}} V ^ {\widehat {\pi}} (s ^ {\prime}) \cdot P _ {M} (s ^ {\prime} | \bar {s}, a)\right)\right). \\ \end{array}
$$
Bounding the first two terms of the RHS using Equation (1) for  $s = \bar{s}$  we see that
$$
\begin{array}{l} V ^ {\star} (\bar {s}) - V ^ {\widehat {\pi}} (\bar {s}) \\ \leq \gamma \left(\sum_ {a \in \mathcal {A}} \widehat {\pi} (\bar {s}, a) \cdot \left(\sum_ {s ^ {\prime} \in \mathcal {S}} V ^ {\star} (s ^ {\prime}) \cdot P _ {M} (s ^ {\prime} | \bar {s}, a)\right) - \sum_ {s ^ {\prime} \in \mathcal {S}} V ^ {\star} (s ^ {\prime}) \cdot P _ {M} (s ^ {\prime} | \bar {s}, \pi^ {\star} (\bar {s}))\right) \\ + \gamma \left(\sum_ {s ^ {\prime} \in \mathcal {S}} V ^ {\star} \left(s ^ {\prime}\right) \cdot P _ {M} \left(s ^ {\prime} \mid \bar {s}, \pi^ {\star} (\bar {s})\right) - \sum_ {a \in \mathcal {A}} \widehat {\pi} (\bar {s}, a) \cdot \left(\sum_ {s ^ {\prime} \in \mathcal {S}} V ^ {\widehat {\pi}} \left(s ^ {\prime}\right) \cdot P _ {M} \left(s ^ {\prime} \mid \bar {s}, a\right)\right)\right) \\ + 2 \varepsilon_ {1} + \varepsilon_ {2}. \\ \end{array}
$$
After we cancel out some terms on the RHS we conclude that
$$
\begin{array}{l} V ^ {\star} (\bar {s}) - V ^ {\widehat {\pi}} (\bar {s}) \\ \leq \gamma \left(\sum_ {a \in \mathcal {A}} \widehat {\pi} (\bar {s}, a) \cdot \left(\sum_ {s ^ {\prime} \in \mathcal {S}} \left(V ^ {\star} (s ^ {\prime}) - V ^ {\widehat {\pi}} (s ^ {\prime})\right) \cdot P _ {M} (s ^ {\prime} | \bar {s}, a)\right)\right) + 2 \varepsilon_ {1} + \varepsilon_ {2} \\ \leq \gamma \left(\sum_ {a \in \mathcal {A}} \widehat {\pi} (\bar {s}, a) \cdot \left(\sum_ {s ^ {\prime} \in \mathcal {S}} \left(V ^ {\star} (\bar {s}) - V ^ {\widehat {\pi}} (\bar {s})\right) \cdot P _ {M} (s ^ {\prime} | \bar {s}, a)\right)\right) + 2 \varepsilon_ {1} + \varepsilon_ {2} \\ = \gamma \left(V ^ {\star} (\bar {s}) - V ^ {\widehat {\pi}} (\bar {s})\right) + 2 \varepsilon_ {1} + \varepsilon_ {2}. \\ \end{array}
$$
The second inequality follows from the fact that  $\bar{s} \in \arg \max_{s \in \mathcal{S}} V^{\star}(s) - V^{\widehat{\pi}}(s)$  and the equality follows from the fact that  $\sum_{a \in \mathcal{A}} \widehat{\pi}(\bar{s}, a) = 1$ ,  $\sum_{s' \in \mathcal{S}} P_M(s' | \bar{s}, a) = 1$ ,  $\forall a \in \mathcal{A}$ . Rearranging, we see that
$$
V ^ {\star} (\bar {s}) - V ^ {\hat {\pi}} (\bar {s}) \leq \frac {2 \varepsilon_ {1} + \varepsilon_ {2}}{1 - \gamma}.
$$
But the choice of  $s \in S$  was arbitrary, concluding the proof.
F.2.2 Proof of Theorem 5.6