SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We assume that the agent does not know the attacker's manipulations and the presence of the attacker. The attacker's manipulations on rewards are stationary. We can consider the combination of the attacker and the environment  $\mathrm{MG}(\mathcal{S},\{\mathcal{A}_i\}_{i = 1}^m,H,P,\{R_i\}_{i = 1}^m)$  as a new environment  $\widetilde{\mathrm{MG}} (\mathcal{S},\{\mathcal{A}_i\}_{i = 1}^m,H,P,\{\widetilde{R}_i\}_{i = 1}^m)$ , and the agents interact with the new environment. We define  $\widetilde{Q}_i$  and  $\widetilde{V}_i$  as the  $Q$ -values and value functions of the new environment  $\widetilde{\mathrm{MG}}$  that each agent  $i$  observes.
We introduce some special strategy modifications  $\{\phi_{i,h}^{\dagger}\}_{h=1}^{H}$ . For any  $h' \geq h$ , we have  $\phi_{i,h}^{\dagger} \diamond \pi_{i,h'}(s) = \pi_{i,h'}^{\dagger}(s)$  and for any  $h' < h$ , we have  $\phi_{i,h}^{\dagger} \diamond \pi_{i,h'}(s) = \pi_{i,h'}(s)$ . Thus,
$$
\widetilde {V} _ {i, 1} ^ {\pi_ {i} ^ {\dagger} \times \pi_ {- i}} (s _ {1}) - \widetilde {V} _ {i, 1} ^ {\pi} (s _ {1}) = \sum_ {h = 1} ^ {H} \left[ \widetilde {V} _ {i, 1} ^ {\phi_ {i, h} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} (s _ {1}) - \widetilde {V} _ {i, 1} ^ {\phi_ {i, h + 1} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} (s _ {1}) \right]. \tag {41}
$$
We have that for any policy  $\pi$
$$
\begin{array}{l} \left[ \widetilde {V} _ {i, 1} ^ {\phi_ {i, h} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} \left(s _ {1}\right) - \widetilde {V} _ {i, 1} ^ {\phi_ {i, h + 1} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} \left(s _ {1}\right) \right] \\ = \mathbb {E} _ {\pi , \mathbb {A}, P} \left[ \left(\widetilde {V} _ {i, h} ^ {\phi_ {i, h} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} (s _ {h}) - \widetilde {V} _ {i, h} ^ {\phi_ {i, h + 1} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} (s _ {h})\right) \right] \\ = \mathbb {E} _ {\pi , \mathbb {A}, P} \left[ \left(\mathbb {D} _ {\phi_ {i, h} ^ {\dagger} \diamond \pi_ {i, h} \times \pi_ {- i, h}} - \mathbb {D} _ {\phi_ {i, h + 1} ^ {\dagger} \diamond \pi_ {i, h} \times \pi_ {- i, h}}\right) \left[ \widetilde {Q} _ {i, h} ^ {\phi_ {i, h + 1} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} \right] (s _ {h}) \right] \tag {42} \\ = \mathbb {E} _ {\pi , \mathbb {A}, P} \left[ \left(\mathbb {D} _ {\pi_ {i, h} ^ {\dagger} \times \pi_ {- i, h}} - \mathbb {D} _ {\pi_ {h}}\right) \left[ \widetilde {R} _ {i, h} + \mathbb {P} _ {h} \widetilde {V} _ {i, h + 1} ^ {\phi_ {i, h + 1} ^ {i} \diamond \pi_ {i} \times \pi_ {- i}} \right] (s _ {h}) \right]. \\ \end{array}
$$
Since  $\widetilde{R}_{i,h}(s,\pmb {a}) = R_{i,h}(s,\pi^{\dagger}(s)) - (\eta +(H - h)\Delta_R)\mathbb{1}(a_i\neq \pi_{i,h}^{\dagger}(s))$  from  $\eta$ -gap attack strategy and  $(H - h)\min_{s^{\prime}\times a^{\prime}\times h^{\prime}}R_{i,h^{\prime}}(s^{\prime},a^{\prime}) < \mathbb{P}_{h}\widetilde{V}_{i,h + 1}^{\pi}(s^{\prime},a^{\prime})\leq (H - h)\max_{s^{\prime}\times a^{\prime}\times h^{\prime}}R_{i,h^{\prime}}(s^{\prime},a^{\prime})$  for any  $s$  and  $a$ , we have
$$
\begin{array}{l} \left[ \widetilde {V} _ {i, 1} ^ {\phi_ {i, h} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} \left(s _ {1}\right) - \widetilde {V} _ {i, 1} ^ {\phi_ {i, h + 1} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} \left(s _ {1}\right) \right] \\ = \mathbb {E} _ {\pi , \mathbb {A}, P} \left[ \sum_ {\boldsymbol {a}} \pi_ {h} (\boldsymbol {a} | s _ {h}) (\eta + (H - h) \Delta_ {R}) \mathbb {1} \left(a _ {i} \neq \pi_ {i, h} ^ {\dagger} \left(s _ {h}\right)\right) \right] \\ + \mathbb {E} _ {\pi , \mathbb {A}, P} \left[ \sum_ {\boldsymbol {a}} \pi_ {h} (\boldsymbol {a} | s _ {h}) \mathbb {1} (a _ {i} \neq \pi_ {i, h} ^ {\dagger} (s _ {h})) \left(\mathbb {P} _ {h} \widetilde {V} _ {i, h + 1} ^ {\phi_ {i, h + 1} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} (s _ {h}, \pi_ {i, h} ^ {\dagger} (s _ {h}) \times \boldsymbol {a} _ {- i}) - \mathbb {P} _ {h} \widetilde {V} _ {i, h + 1} ^ {\phi_ {i, h + 1} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} (s _ {h}, \boldsymbol {a})\right) \right] \\ \geq \mathbb {E} _ {\pi , \mathbb {A}, P} \left[ \sum_ {\boldsymbol {a}} \pi_ {h} (\boldsymbol {a} | s _ {h}) \eta \mathbb {1} \left(a _ {i} \neq \pi_ {i, h} ^ {\dagger} (s)\right) \right] \\ = \mathbb {E} _ {\pi , \mathbb {A}, P} \left[ \mathbb {1} \left(a _ {i, h} \neq \pi_ {i, h} ^ {\dagger} (s)\right) \right] \eta . \tag {43} \\ \end{array}
$$
$$
\widetilde {V} _ {i, 1} ^ {\pi_ {i} ^ {\dagger} \times \pi_ {- i}} (s _ {1}) - \widetilde {V} _ {i, 1} ^ {\pi} (s _ {1}) \geq \mathbb {E} _ {\pi , \mathbb {A}, P} \left[ \sum_ {h = 1} ^ {H} \mathbb {1} \left(a _ {i, h} \neq \pi_ {i, h} ^ {\dagger} (s _ {h})\right) \right] \eta \geq 0. \tag {44}
$$
The target policy is an NE, and then also is an CE or CCE.
If there exists an CCE  $\pi'$  under  $\eta$ -gap attack, we have  $\max_{i \in [m]} (\widetilde{V}_{i,1}^{\dagger, \pi_{-i}^{\prime}}(s) - \widetilde{V}_{i,1}^{\pi^{\prime}}(s)) = 0$  for any initial state  $s$ . If  $\pi_{i,h}^{\prime}(\cdot | s) \neq \pi_{i,h}^{\dagger}(\cdot | s)$  at a reachable state  $s$ , we have that  $\mathbb{E}_{\pi', \mathbb{A}, P}\left[\sum_{h=1}^{H} \mathbb{1}(a_{i,h} \neq \pi_{i,h}^{\dagger}(s_h))\right]$ . Thus,
$$
\widetilde {V} _ {i, 1} ^ {\pi_ {i} ^ {\dagger} \times \pi_ {- i} ^ {\prime}} (s _ {1}) - \widetilde {V} _ {i, 1} ^ {\pi^ {\prime}} (s _ {1}) \geq \mathbb {E} _ {\pi^ {\prime}, \mathbb {A}, P} \left[ \sum_ {h = 1} ^ {H} \mathbb {1} \left(a _ {i, h} \neq \pi_ {i, h} ^ {\dagger} (s _ {h})\right) \right] \eta > 0, \tag {45}
$$
and  $\pi^{\prime}$  is not an CCE. In summary, the target policy  $\pi_i^\dagger$  is the unique {NE, CE, CCE}.
H.2 Proof of Theorem 6
From the definition of the best-in-hindsight regret and (51), we have
$$
\begin{array}{l} \operatorname {R e g} _ {i} (K, H) = \max  _ {\pi_ {i} ^ {\prime}} \sum_ {k = 1} ^ {K} \left[ \widetilde {V} _ {i, 1} ^ {\pi_ {i} ^ {\prime} \times \pi_ {- i} ^ {k}} \left(s _ {1} ^ {k}\right) - \widetilde {V} _ {i, 1} ^ {\pi^ {k}} \left(s _ {1} ^ {k}\right) \right] \tag {46} \\ \geq \sum_ {k = 1} ^ {K} \left[ \widetilde {V} _ {i, 1} ^ {\pi_ {i} ^ {\dagger} \times \pi_ {- i} ^ {k}} \left(s _ {1} ^ {k}\right) - \widetilde {V} _ {i, 1} ^ {\pi^ {k}} \left(s _ {1} ^ {k}\right) \right]. \\ \end{array}
$$
From (44), we have
$$
\operatorname {R e g} _ {i} (K, H) \geq \sum_ {k = 1} ^ {K} \mathbb {E} _ {\pi^ {k}, \mathrm {A}, P} \left[ \sum_ {h = 1} ^ {H} \mathbb {1} \left[ a _ {i, h} ^ {k} \neq \pi_ {i, h} ^ {\dagger} \left(s _ {h} ^ {k}\right) \right] \right] \eta \tag {47}
$$
$$
\sum_ {i = 1} ^ {m} \operatorname {R e g} _ {i} (K, H) = \eta \mathbb {E} [ \text {L o s s 1} (K, H) ]. \tag {48}
$$
If the best-in-hindsight regret  $\operatorname{Reg}(K, H)$  of each agent's algorithm is bounded by a sub-linear bound  $\mathcal{R}(T)$ , then the attack loss is bounded by  $\mathbb{E}[\operatorname{Loss1}(K, H)] \leq m\mathcal{R}(T) / \eta$ .
The  $\eta$ -gap attack strategy attacks all agents when any agent  $i$  chooses an non-target action. Note that the rewards are bounded in  $[0,1]$ . We have
$$
\begin{array}{l} \operatorname {C o s t} (K, H) = \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \sum_ {i = 1} ^ {m} \left(\mathbb {1} \left(\widetilde {a} _ {i, h} ^ {k} \neq a _ {i, h} ^ {k}\right) + \left| \widetilde {r} _ {i, h} ^ {k} - r _ {i, h} ^ {k} \right|\right) \tag {49} \\ \leq \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \sum_ {i = 1} ^ {m} \mathbb {1} \left[ a _ {i, h} ^ {k} \neq \pi_ {i, h} ^ {\dagger} \left(s _ {h} ^ {k}\right) \right] m. \\ \end{array}
$$
Hence, the attack cost is bounded by  $m\mathbb{E}[\mathrm{Loss1}(K,H)]\leq m^2\mathcal{R}(T) / \eta$
I Analysis of the gray-box attacks
I.1 Proof of Theorem 7
We assume that the agent does not know the attacker's manipulations and the presence of the attacker. The attacker's manipulations on actions are stationary. We can consider the combination of the attack and the environment  $\mathrm{MG}(\mathcal{S},\{\mathcal{A}_i\}_{i = 1}^m,H,P,\{R_i\}_{i = 1}^m)$  as a new environment
$\widetilde{\mathrm{MG}} (\mathcal{S},\{\mathcal{A}_i\}_{i = 1}^m,H,\widetilde{P},\{\widetilde{R}_i\}_{i = 1}^m)$ , and the agents interact with the new environment. We define  $\widetilde{Q}_i$  and  $\widetilde{V}_i$  as the  $Q$ -values and value functions of the new environment  $\widetilde{\mathrm{MG}}$  that each agent  $i$  observes.
We first prove that the best response of each agent  $i$  towards any policy  $\pi_{-i}$  is  $\pi_i^\dagger$ .
From the mixed attack strategy, we have
$$
\widetilde {Q} _ {i, h} ^ {\pi} (s, \boldsymbol {a}) = \mathbb {1} \left[ a _ {i} = \pi_ {i, h} ^ {\dagger} (s) \right] R _ {i, h} \left(s, \pi_ {h} ^ {\dagger} (s)\right) + \mathbb {P} _ {h} \widetilde {V} _ {i, h + 1} ^ {\pi} \left(s, \pi_ {h} ^ {\dagger} (s)\right). \tag {50}
$$
Consider an arbitrary policy  $\pi$  and an arbitrary initial state  $s_1$ . We have
$$
\begin{array}{l} \widetilde {V} _ {i, 1} ^ {\pi_ {i} ^ {\dagger} \times \pi_ {- i}} (s _ {1}) - \widetilde {V} _ {i, 1} ^ {\pi} (s _ {1}) \\ = \widetilde {V} _ {i, 1} ^ {\pi_ {i} ^ {\dagger} \times \pi_ {- i}} (s _ {1}) - \mathbb {D} _ {\pi} [ \widetilde {Q} _ {i, 1} ^ {\pi_ {i} ^ {\dagger} \times \pi_ {- i}} ] (s _ {1}) + \mathbb {D} _ {\pi} [ \widetilde {Q} _ {i, 1} ^ {\pi_ {i} ^ {\dagger} \times \pi_ {- i}} ] (s _ {1}) - \widetilde {V} _ {i, 1} ^ {\pi} (s _ {1}) \\ = \mathbb {E} _ {a _ {i, 1} \sim \pi_ {i, 1} (\cdot | s _ {1})} \left[ \mathbb {1} [ a _ {i, 1} \neq \pi_ {i, 1} ^ {\dagger} (s _ {1}) ] R _ {i, 1} (s _ {1}, \pi_ {1} ^ {\dagger} (s _ {1})) \right] + \mathbb {P} _ {1} \widetilde {V} _ {i, 2} ^ {\pi_ {i} ^ {\dagger} \times \pi_ {- i}} (s _ {1}, \pi_ {1} ^ {\dagger} (s _ {1})) - \mathbb {P} _ {1} \widetilde {V} _ {i, 2} ^ {\pi} (s _ {1}, \pi_ {1} ^ {\dagger} (s _ {1})) \\ = \mathbb {E} _ {a _ {i, 1} \sim \pi_ {i, 1} (\cdot | s _ {1})} \left[ \mathbb {1} [ a _ {i, 1} \neq \pi_ {i, 1} ^ {\dagger} (s _ {1}) ] R _ {i, 1} (s _ {1}, \pi_ {1} ^ {\dagger} (s _ {1})) \right] + \mathbb {P} _ {1} [ \widetilde {V} _ {i, 2} ^ {\pi_ {i} ^ {\dagger} \times \pi_ {- i}} - \widetilde {V} _ {i, 2} ^ {\pi} ] (s _ {1}, \pi_ {1} ^ {\dagger} (s _ {1})) \\ = \dots = \mathbb {E} _ {\pi , \mathbb {A}, P} \left[ \sum_ {h = 1} ^ {H} \left(1 - \pi_ {i, h} \left(\pi_ {i, h} ^ {\dagger} (s _ {h}) | s _ {h}\right)\right) R _ {i, h} \left(s _ {h}, \pi_ {h} ^ {\dagger} (s _ {h})\right) \right] \geq 0. \tag {51} \\ \end{array}
$$
Since  $R_{i,h}(s_h,\pi_h^\dagger (s_h)) > 0$ ,  $\widetilde{V}_{i,1}^{\pi_i^\dagger \times \pi_{-i}}(s_1) - \widetilde{V}_{i,1}^\pi (s_1) = 0$  holds if and only if  $\pi_i^\dagger = \pi_i$  holds for the states that are reachable under policy  $\pi^{\dagger}$ . We conclude that the best response of each agent  $i$  towards any policy  $\pi_{-i}$  is  $\pi_i^\dagger$  under the mixed attack strategy. The target policy  $\pi^{\dagger}$  is an NE, CE, CCE under the mixed attack strategy.
Now we prove that the target policy  $\pi_i^\dagger$  is the unique {NE, CE, CCE} under the mixed attack strategy, when every state  $s\in S$  is reachable at every step  $h\in [H]$  under the target policy.
If there exists an CCE  $\pi'$  under the mixed attack strategy, we have  $\max_{i\in [m]}\bigl (\widetilde{V}_{i,1}^{\dagger,\pi_{-i}^{\prime}}(s)- \widetilde{V}_{i,1}^{\pi^{\prime}}(s)\bigr) = 0$  for any initial state  $s$ .
From (51), we have that if  $\pi_{i,h}^{\prime}(\cdot |s)\neq \pi_{i,h}^{\dagger}(\cdot |s)$  at a reachable state  $s$ ,  $\widetilde{V}_{i,1}^{\pi_i^\dagger \times \pi_{-i}^\prime}(s_1) - \widetilde{V}_{i,1}^{\pi '}(s_1) > 0$ . Then  $\widetilde{V}_{i,1}^{\dagger,\pi_{-i}^{\prime}}(s_1) > \widetilde{V}_{i,1}^{\pi_i^\dagger \times \pi_{-i}^\prime}(s_1) > \widetilde{V}_{i,1}^{\pi '}(s_1)$ .  $\pi'$  is not an CCE in this case.
We can conclude that  $\pi' = \pi^\dagger$  for the states that are reachable under policy  $\pi'$ . If every state  $s \in S$  is reachable at every step  $h \in [H]$  under the target policy,  $\pi^\dagger$  is the unique  $\{\mathrm{NE}, \mathrm{CE}, \mathrm{CCE}\}$ .
I.2 Proof of Theorem 8
We set  $R_{min} = \min_{h\in [H]}\min_{s\in S}\min_{i\in [m]}R_{i,h}(s,\pi_h^\dagger (s))$ . From the definition of the best-in-hindsight regret and (51), we have
$$
\begin{array}{l} \operatorname {R e g} _ {i} (K, H) = \max  _ {\pi_ {i} ^ {\prime}} \sum_ {k = 1} ^ {K} \left[ \widetilde {V} _ {i, 1} ^ {\pi_ {i} ^ {\prime} \times \pi_ {- i} ^ {k}} \left(s _ {1} ^ {k}\right) - \widetilde {V} _ {i, 1} ^ {\pi^ {k}} \left(s _ {1} ^ {k}\right) \right] \\ \geq \sum_ {k = 1} ^ {K} \left[ \widetilde {V} _ {i, 1} ^ {\pi_ {i} ^ {\dagger} \times \pi_ {- i} ^ {k}} \left(s _ {1} ^ {k}\right) - \widetilde {V} _ {i, 1} ^ {\pi^ {k}} \left(s _ {1} ^ {k}\right) \right] \\ = \sum_ {k = 1} ^ {K} \mathbb {E} _ {\pi^ {k}, \mathbb {A}, P} \left[ \sum_ {h = 1} ^ {H} \left(1 - \pi_ {i, h} ^ {k} \left(\pi_ {i, h} ^ {\dagger} \left(s _ {h} ^ {k}\right) \mid s _ {h} ^ {k}\right)\right) R _ {i, h} \left(s _ {h} ^ {k}, \pi_ {h} ^ {\dagger} \left(s _ {h} ^ {k}\right)\right) \right] \tag {52} \\ = \sum_ {k = 1} ^ {K} \mathbb {E} _ {\pi^ {k}, \mathbb {A}, P} \left[ \sum_ {h = 1} ^ {H} \mathbb {1} [ a _ {i, h} ^ {k} \neq \pi_ {i, h} ^ {\dagger} (s _ {h} ^ {k}) ] R _ {i, h} (s _ {h} ^ {k}, \pi_ {h} ^ {\dagger} (s _ {h} ^ {k})) \right] \\ \geq R _ {m i n} \sum_ {k = 1} ^ {K} \mathbb {E} _ {\pi^ {k}, \mathbb {A}, P} \left[ \sum_ {h = 1} ^ {H} \mathbb {1} \left[ a _ {i, h} ^ {k} \neq \pi_ {i, h} ^ {\dagger} \left(s _ {h} ^ {k}\right) \right] \right] \\ \end{array}
$$
$$
\sum_ {i = 1} ^ {m} \operatorname {R e g} _ {i} (K, H) \geq R _ {\min } \mathbb {E} [ \text {L o s s} 1 (K, H) ]. \tag {53}
$$
If the best-in-hindsight regret  $\operatorname{Reg}(K, H)$  of each agent's algorithm is bounded by a sub-linear bound  $\mathcal{R}(T)$  under the mixed attack strategy, then the attack loss is bounded by  $\mathbb{E}[\operatorname{Loss1}(K, H)] \leq m\mathcal{R}(T)/R_{min}$ .
The mixed attack strategy only attacks agent  $i$  when agent  $i$  chooses a non-target action. We have
$$
\begin{array}{l} \operatorname {C o s t} (K, H) = \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \sum_ {i = 1} ^ {m} \left(\mathbb {1} \left(\tilde {a} _ {i, h} ^ {k} \neq a _ {i, h} ^ {k}\right) + \left| \tilde {r} _ {i, h} ^ {k} - r _ {i, h} ^ {k} \right|\right) \\ \leq \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \sum_ {i = 1} ^ {m} \mathbb {1} \left[ \widetilde {a} _ {i, h} ^ {k} \neq a _ {i, h} ^ {k} \right] (1 + 1). \tag {54} \\ \end{array}
$$
Then, the attack cost is bounded by  $2\mathbb{E}[\mathrm{Loss1}(K,H)] \leq 2m\mathcal{R}(T) / R_{min}$ .
J Analysis of the black-box attacks
J.1 Proof of Lemma 1
We denote by  $\overline{Q}_{\dagger,h}^k, \underline{Q}_{\dagger,h}^k, \overline{V}_{\dagger,h}^k, \underline{V}_{\dagger,h}^k, N_h^k, \hat{\mathbb{P}}_h^k, \pi_h^k$  and  $\hat{R}_{\dagger,h}^k$  the observations of the approximate mixed attacker at the beginning of episode  $k$  and time step  $h$ . As before, we begin with proving that the estimations are indeed upper and lower bounds of the corresponding  $Q$ -values and state value functions. We use  $\pi^*$  to denote the optimal policy that maximizes the attacker's rewards, i.e.  $V_{\dagger,1}^{\pi^*}(s) = \max_{\pi} V_{\dagger,1}^{\pi}(s)$ .
Lemma 2 With probability  $1 - p$ , for any  $(s, \mathbf{a}, h)$  and  $k \leq \tau$ ,
$$
\bar {Q} _ {\dagger , h} ^ {k} (s, \boldsymbol {a}) \geq Q _ {\dagger , h} ^ {\pi^ {*}} (s, \boldsymbol {a}), \underline {{Q}} _ {\dagger , h} ^ {k} (s, \boldsymbol {a}) \leq Q _ {\dagger , h} ^ {\pi^ {k}} (s, \boldsymbol {a}), \tag {55}
$$
$$
\bar {V} _ {\dagger , h} ^ {k} (s) \geq V _ {\dagger , h} ^ {\pi^ {*}} (s), \underline {{V}} _ {\dagger , h} ^ {r} (s) \leq V _ {\dagger , h} ^ {\pi^ {k}} (s). \tag {56}
$$
Proof. For each fixed  $k$ , we prove this by induction from  $h = H + 1$  to  $h = 1$ . For the step  $H + 1$ , we have  $\overline{V}_{\dagger, H + 1}^{k} = \underline{V}_{\dagger, H + 1}^{k} = Q_{\dagger, H + 1}^{\pi^{*}} = \mathbf{0}$ . Now, we assume inequality (56) holds for the step  $h + 1$ . By the definition of  $Q$ -values and Algorithm 1, we have
$$
\begin{array}{l} \overline {{Q}} _ {\dagger , h} ^ {k} (s, \boldsymbol {a}) - Q _ {\dagger , h} ^ {\pi^ {*}} (s, \boldsymbol {a}) \\ = \hat {R} _ {\dagger , h} ^ {k} (s, \boldsymbol {a}) - R _ {\dagger , h} ^ {k} (s, \boldsymbol {a}) + \hat {\mathbb {P}} _ {h} ^ {k} \bar {V} _ {\dagger , h + 1} ^ {k} (s, \boldsymbol {a}) - \mathbb {P} _ {h} V _ {i, h} ^ {\pi^ {*}} (s, \boldsymbol {a}) + B \left(N _ {h} ^ {k} (s, \boldsymbol {a})\right) \tag {57} \\ = \hat {\mathbb {P}} _ {h} ^ {k} (\overline {{V}} _ {\dagger , h + 1} ^ {k} - V _ {\dagger , h} ^ {\pi^ {*}}) (s, \pmb {a}) + (\hat {R} _ {\dagger , h} ^ {k} - R _ {\dagger , h} ^ {k}) (s, \pmb {a}) + (\hat {\mathbb {P}} _ {h} ^ {k} - \mathbb {P} _ {h}) V _ {\dagger , h} ^ {\pi^ {*}} (s, \pmb {a}) + B (N _ {h} ^ {k} (s, \pmb {a})). \\ \end{array}
$$
Recall that  $B(N) = (H\sqrt{S} + 1)\sqrt{\log(2AH\tau / p) / (2N)}$ . By Azuma-Hoeffding inequality, we have that with probability  $1 - 2p / SAH$ ,
$$
\forall k \leq \tau , \left| \hat {R} _ {\dagger , h} ^ {k} (s, \boldsymbol {a}) - R _ {\dagger , h} ^ {k} (s, \boldsymbol {a}) \right| \leq \sqrt {\frac {\log (2 S A H \tau / p)}{2 N _ {h} ^ {k} (s , \boldsymbol {a})}}, \tag {58}
$$
$$
\forall k \leq \tau , \left| \left(\hat {\mathbb {P}} _ {h} ^ {k} - \mathbb {P} _ {h}\right) V _ {\dagger , h} ^ {\pi^ {*}} (s, \boldsymbol {a}) \right| \leq H \sqrt {\frac {S \log (2 S A H \tau / p)}{2 N _ {h} ^ {k} (s , \boldsymbol {a})}}. \tag {59}
$$
Putting everything together, we have  $\overline{Q}_{\dagger,h}^{k}(s,\pmb{a}) - Q_{\dagger,h}^{\pi^{*}}(s,\pmb{a}) \geq \hat{\mathbb{P}}_{h}^{k}(\overline{V}_{\dagger,h+1}^{k} - V_{\dagger,h}^{\pi^{*}})(s,\pmb{a}) \geq 0$ . Similarly,  $\underline{Q}_{\dagger,h}^{k}(s,\pmb{a}) \leq Q_{\dagger,h}^{\pi^{k}}(s,\pmb{a})$ .
Now we assume inequality (55) holds for the step  $h$ . As discussed above, if inequality (56) holds for the step  $h + 1$ , inequality (55) holds for the step  $h$ . By Algorithm 1, we have
$$
\bar {V} _ {\dagger , h} ^ {k} (s) = \bar {Q} _ {\dagger , h} ^ {k} \left(s, \pi_ {h} ^ {k} (s)\right) \geq \bar {Q} _ {\dagger , h} ^ {k} \left(s, \pi_ {h} ^ {*} (s)\right) \geq Q _ {\dagger , h} ^ {\pi^ {*}} \left(s, \pi_ {h} ^ {*} (s)\right) = V _ {\dagger , h} ^ {\pi^ {*}} (s). \tag {60}
$$
Similarly,  $\underline{V}_{\dagger,h}^{k}(s) \leq V_{\dagger,h}^{\pi^{k}}(s)$ .
Now, we are ready to prove Lemma 1. By Azuma-Hoeffding inequality, we have that with probability  $1 - 2p$ ,
$$
\left| \left(\mathbb {E} _ {s _ {1} \sim P _ {0} (\cdot)} - \mathbb {E} _ {s _ {1} \sim \hat {\mathbb {P}} _ {0} (\cdot)}\right) \left[ V _ {\dagger , 1} ^ {\pi^ {*}} (s _ {1}) - V _ {\dagger , 1} ^ {\pi^ {k}} (s _ {1}) \right] \right| \leq H \sqrt {\frac {S \log (2 \tau / p)}{2 k}}, \tag {61}
$$
and  $\forall k\leq \tau$
$$
\left| \sum_ {k ^ {\prime} = 1} ^ {k} \left(\mathbb {E} _ {s _ {1} \sim P _ {0} (\cdot)} - \mathbb {1} \left(s _ {1} = s _ {1} ^ {k ^ {\prime}}\right)\right) \left[ V _ {\dagger , 1} ^ {\pi^ {*}} (s _ {1}) - V _ {\dagger , 1} ^ {\pi^ {k ^ {\prime}}} (s _ {1}) \right] \right| \leq H \sqrt {\frac {S \log (2 \tau / p)}{2 k}}. \tag {62}
$$
Thus, for any  $k \leq \tau$ ,
$$
\begin{array}{l} \mathbb {E} _ {s _ {1} \sim P _ {0} (\cdot)} \left[ V _ {\dagger , 1} ^ {\pi^ {*}} (s _ {1}) - V _ {\dagger , 1} ^ {\pi^ {k}} (s _ {1}) \right] \\ \leq \mathbb {E} _ {s _ {1} \sim \hat {\mathrm {P}} _ {0} ^ {k} (\cdot)} \left[ V _ {\dagger , 1} ^ {\pi^ {*}} (s _ {1}) - V _ {\dagger , 1} ^ {\pi^ {k}} (s _ {1}) \right] + H \sqrt {S \log (2 \tau / p) / (2 k)} \tag {63} \\ \leq \mathbb {E} _ {s _ {1} \sim \hat {\mathbb {P}} _ {0} ^ {k} (\cdot)} \left[ \bar {V} _ {\dagger , 1} ^ {k} (s _ {1}) - \underline {{V}} _ {\dagger , 1} ^ {k} (s _ {1}) \right] + H \sqrt {S \log (2 \tau / p) / (2 k)}. \\ \end{array}
$$
According to (61) and (62), we have
$$
\begin{array}{l} \sum_ {k = 1} ^ {\tau} \left(\mathbb {E} _ {s _ {1} \sim \hat {\mathbb {P}} _ {0} ^ {k} (\cdot)} \left[ \overline {{V}} _ {\dagger , 1} ^ {k} (s _ {1}) - \underline {{V}} _ {\dagger , 1} ^ {k} (s _ {1}) \right] + H \sqrt {S \log (2 \tau / p) / (2 k)}\right) \\ \leq \sum_ {k = 1} ^ {\tau} \left(\bar {V} _ {\dagger , 1} ^ {k} \left(s _ {1} ^ {k}\right) - \underline {{V}} _ {\dagger , 1} ^ {k} \left(s _ {1} ^ {k}\right)\right) + \sum_ {k = 1} ^ {\tau} 3 H \sqrt {S \log (2 \tau / p) / (2 k)}. \tag {64} \\ \end{array}
$$
We define  $\Delta V_h^k(s) = \overline{V}_{\dagger,h}^k(s) - \underline{V}_{\dagger,h}^k(s)$ ,  $\Delta Q_h^k(s, \boldsymbol{a}) = \overline{Q}_{\dagger,h}^k(s, \boldsymbol{a}) - \underline{Q}_{\dagger,h}^k(s, \boldsymbol{a})$ . By the update equations in Algorithm 1, we have  $\Delta Q_h^k(s, \boldsymbol{a}) \leq \hat{\mathbb{P}}_h^k \Delta V_{h+1}^k(s, \boldsymbol{a}) + 2B(N_h^k(s, \boldsymbol{a}))$  and  $\Delta V_h^k(s) = \Delta Q_h^k(s, \pi_h^k(s))$ . We define  $\psi_h^k = \Delta V_h^k(s_h^k) = \Delta Q_h^k(s_h^k, a_h^k)$ . From (59) and (66), we have
$$
\begin{array}{l} \psi_ {h} ^ {k} \leq \hat {\mathbb {P}} _ {h} ^ {k} \Delta V _ {h + 1} ^ {k} \left(s _ {h} ^ {k}, \boldsymbol {a} _ {h} ^ {k}\right) + 2 B \left(N _ {h} ^ {k} \left(s _ {h} ^ {k}, \boldsymbol {a} _ {h} ^ {k}\right)\right) \\ \leq \mathbb {P} _ {h} ^ {k} \Delta V _ {h + 1} ^ {k} \left(s _ {h} ^ {k}, \boldsymbol {a} _ {h} ^ {k}\right) + 3 B \left(N _ {h} ^ {k} \left(s _ {h} ^ {k}, \boldsymbol {a} _ {h} ^ {k}\right)\right) \tag {65} \\ \leq \mathbb {P} _ {h} ^ {k} \Delta V _ {h + 1} ^ {k} \left(s _ {h} ^ {k}, \boldsymbol {a} _ {h} ^ {k}\right) - \psi_ {h + 1} ^ {k} + \psi_ {h + 1} ^ {k} + 3 B \left(N _ {h} ^ {k} \left(s _ {h} ^ {k}, \boldsymbol {a} _ {h} ^ {k}\right)\right). \\ \end{array}
$$
By Azuma-Hoeffding inequality, we have that with probability  $1 - p / H,\forall k\leq \tau$
$$
\left| \sum_ {k ^ {\prime} = 1} ^ {k} \left| \mathbb {P} _ {h} ^ {k ^ {\prime}} \Delta V _ {h + 1} ^ {k} \left(s _ {h} ^ {k ^ {\prime}}, \boldsymbol {a} _ {h} ^ {k ^ {\prime}}\right) - \psi_ {h + 1} ^ {k ^ {\prime}} \right| \right| \leq H \sqrt {\frac {S \log (2 H \tau / p)}{2 k}}. \tag {66}
$$
Since  $\psi_{H + 1}^k = 0$  for all  $k$ , we have
$$
\begin{array}{l} \sum_ {k = 1} ^ {\tau} \psi_ {1} ^ {k} \leq \sum_ {k = 1} ^ {\tau} \sum_ {h = 1} ^ {H} | \mathbb {P} _ {h} ^ {k} \Delta V _ {h + 1} ^ {k} \left(s _ {h} ^ {k}, \boldsymbol {a} _ {\boldsymbol {h}} ^ {k}\right) - \psi_ {h + 1} ^ {k} | + \sum_ {k = 1} ^ {\tau} \sum_ {h = 1} ^ {H} 3 B \left(N _ {h} ^ {k} \left(s _ {h} ^ {k}, \boldsymbol {a} _ {h} ^ {k}\right)\right) \\ \leq \sum_ {h = 1} ^ {H} H \sqrt {\frac {S \log (2 H \tau / p)}{2 \tau}} + \sum_ {h = 1} ^ {H} \sum_ {(s, \boldsymbol {a})} \sum_ {n = 1} ^ {N _ {h} ^ {\tau} (s, \boldsymbol {a})} (H \sqrt {S} + 1) \sqrt {\frac {\log (2 S A H \tau / p)}{2 n}} \tag {67} \\ \leq H ^ {2} \sqrt {\frac {S \log (2 H \tau / p)}{2 \tau}} + H (H \sqrt {S} + 1) \sqrt {2 S A \tau \log (2 S A H \tau / p)} \\ \end{array}
$$
$$
\sum_ {k = 1} ^ {\tau} \left(\mathbb {E} _ {s _ {1} \sim \hat {\mathbb {P}} _ {0} ^ {k} (\cdot)} \left[ \bar {V} _ {\dagger , 1} ^ {k} (s _ {1}) - \underline {{V}} _ {\dagger , 1} ^ {k} (s _ {1}) \right] + H \sqrt {S \log (2 \tau / p) / (2 k)}\right) \tag {68}
$$
$$
\leq H ^ {2} \sqrt {\frac {S \log (2 H \tau / p)}{2 \tau}} + 3 H \sqrt {2 S \tau \log (2 \tau / p)} + H (H \sqrt {S} + 1) \sqrt {2 S A \tau \log (2 S A H \tau / p)}.
$$
$$
\pi^ {\dagger} = \min  _ {\pi_ {k}} \left(\mathbb {E} _ {s _ {1} \sim \hat {\mathbb {P}} _ {0} ^ {k} (\cdot)} \left[ \sum_ {i = 1} ^ {m} \left(V _ {\dagger , 1} ^ {\pi^ {*}} (s _ {1}) - V _ {\dagger , 1} ^ {\pi^ {k}} (s _ {1})\right) \right] + H \sqrt {S \log (2 \tau / p) / (2 k)}\right), \tag {69}
$$