SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\widetilde {Q} _ {i, h} ^ {\pi} (s, \boldsymbol {a}) = \widetilde {R} _ {i, h} (s, \boldsymbol {a}) + \frac {d _ {h} (s , \boldsymbol {a})}{m} \mathbb {P} _ {h} \widetilde {V} _ {i, h + 1} ^ {\pi} (s, \pi_ {h} ^ {\dagger} (s)) + \left(1 - \frac {d _ {h} (s , \boldsymbol {a})}{m}\right) \mathbb {P} _ {h} \widetilde {V} _ {i, h + 1} ^ {\pi} (s, \pi_ {h} ^ {-} (s)), \tag {14}
$$
$$
\widetilde {R} _ {i, h} (s, \boldsymbol {a}) = \frac {d _ {h} (s , \boldsymbol {a})}{m} R _ {i, h} (s, \pi_ {h} ^ {\dagger} (s)) + \left(1 - \frac {d _ {h} (s , \boldsymbol {a})}{m}\right) R _ {i, h} (s, \pi_ {h} ^ {-} (s)). \tag {15}
$$
Since the attacker does not attack when the agents follow the target policy, we have  $\widetilde{V}_{i,h+1}^{\pi^\dagger}(s) = V_{i,h+1}^{\pi^\dagger}(s)$ . Then,
$$
\widetilde {Q} _ {i, h} ^ {\pi^ {\dagger}} (s, \boldsymbol {a}) = \frac {d _ {h} (s , \boldsymbol {a})}{m} Q _ {i, h} ^ {\pi^ {\dagger}} (s, \pi_ {h} ^ {\dagger} (s)) + \left(1 - \frac {d _ {h} (s , \boldsymbol {a})}{m}\right) Q _ {i, h} ^ {\pi^ {\dagger}} (s, \pi_ {h} ^ {-} (s)). \tag {16}
$$
If  $a_i \neq \pi_{i,h}^\dagger(s)$ , we have
$$
\widetilde {Q} _ {i, h} ^ {\pi^ {\dagger}} (s, \pi_ {i, h} ^ {\dagger} (s) \times \boldsymbol {a} _ {- i}) - \widetilde {Q} _ {i, h} ^ {\pi^ {\dagger}} (s, \boldsymbol {a}) = \frac {1}{2 m} \left(Q _ {i, h} ^ {\pi^ {\dagger}} (s, \pi_ {h} ^ {\dagger} (s)) - Q _ {i, h} ^ {\pi^ {\dagger}} (s, \pi_ {h} ^ {-} (s))\right) \geq \frac {\Delta_ {m i n}}{2 m}. \tag {17}
$$
We have that policy  $\pi_i^\dagger$  is best-in-hindsight policy towards the target policy  $\pi_{-i}^\dagger$  at step  $h$  in the observation of each agent  $i$ , i.e.  $\widetilde{V}_{i,h+1}^{\pi_i^\dagger \times \pi_{-i}^\dagger}(s) = \widetilde{V}_{i,h+1}^{\dagger, \pi_{-i}^\dagger}(s)$  for any agent  $i$ , any state  $s$  and any policy  $\pi_{-i}$ .
Since the above argument works for any step  $h \in [H]$ , we have that the best response of each agent  $i$  towards the target product policy  $\pi_{-i}^{\dagger}$  is  $\pi_i^\dagger$  and the target policy is an {NE, CE, CCE} under  $d$ -portion attack.
Now we prove that the target policy  $\pi_i^\dagger$  is the unique {NE, CE, CCE}, when every state  $s\in S$  is reachable at every step  $h\in [H]$  under the target policy.
If there exists an CCE  $\pi'$  under  $d$ -portion attack, we have  $\max_{i \in [m]} (\widetilde{V}_{i,1}^{\dagger, \pi_{-i}^{\prime}}(s) - \widetilde{V}_{i,1}^{\pi^{\prime}}(s)) = 0$  for any initial state  $s$ .
At the step  $H$ ,  $\widetilde{Q}_{i,H}^{\pi}(s,\pmb{a}) = \widetilde{R}_{i,H}(s,\pmb{a})$ . Since  $R_{i,H}(s,\pi_H^\dagger(s)) \geq R_{i,H}(s,\pi_H^- (s)) + \Delta_{min}$  with  $\Delta_{min} > 0$ , the policy  $\pi_{i,H}^{\dagger}$  is the unique best response towards any policy  $\pi_{-i,H}$ , i.e.  $\widetilde{V}_{i,H}^{\pi_{i,H},\pi_{-i,H}}(s) = \widetilde{V}_{i,H}^{\dagger,\pi_{-i,H}}(s)$  and  $\widetilde{V}_{i,H}^{\pi_{i,H},\pi_{-i,H}}(s) > \widetilde{V}_{i,H}^{\pi_{i,H},\pi_{-i,H}}(s)$  for any  $\pi_{i,H}(\cdot |s) \neq \pi_{i,H}^{\dagger}(\cdot |s)$ . Thus, we have  $\pi_H'(s_H) = \pi_H^\dagger(s_H)$  for any state  $s_H$  that is reachable at the time step  $H$  under policy  $\pi'$ . We assume that every state  $s \in S$  is reachable at every step  $h \in [H]$  under the target policy. Under  $d$ -portion attacks, the post-attack action  $\widetilde{\pmb{a}}_h = \pi_h^\dagger(s)$  with probability more than 0.5. Thus, every state  $s \in S$  is reachable at every step  $h \in [H]$  under any policy  $\pi$  and  $d$ -portion attacks.
Recall that for any  $a_i \neq \pi_{i,h}^\dagger(s)$ ,
$$
\widetilde {Q} _ {i, h} ^ {\pi^ {\dagger}} (s, \pi_ {i, h} ^ {\dagger} (s) \times \boldsymbol {a} _ {- i}) - \widetilde {Q} _ {i, h} ^ {\pi^ {\dagger}} (s, \boldsymbol {a}) \geq \Delta_ {\min } / 2 m. \tag {18}
$$
Suppose  $\{\pi_{h'}'\}_{h' = h + 1}^H = \{\pi_{h'}^\dagger\}_{h' = h + 1}^H$  for any states. If  $\pi_{i,h}'(\cdot |s) \neq \pi_{i,h}^\dagger(\cdot |s)$  at a reachable state  $s$ , we have
$$
\begin{array}{l} \widetilde {V} _ {i, h} ^ {\pi^ {\prime}} (s) = \mathbb {E} _ {\boldsymbol {a} \sim \pi_ {h} ^ {\prime} (\cdot | s)} [ \widetilde {Q} _ {i, h} ^ {\pi^ {\prime}} (s, \boldsymbol {a}) ] \\ = \mathbb {E} _ {\boldsymbol {a} \sim \pi_ {h} ^ {\prime} (\cdot | s)} \left[ \widetilde {Q} _ {i, h} ^ {\pi^ {\dagger}} (s, \boldsymbol {a}) \right] (19) \\ \leq \mathbb {E} _ {\boldsymbol {a} _ {- i} \sim \pi_ {- i, h} ^ {\prime} (\cdot | s)} \left[ \widetilde {Q} _ {i, h} ^ {\pi^ {\dagger}} (s, \pi_ {i, h} ^ {\dagger} (s) \times \boldsymbol {a} _ {- i}) \right] - c \Delta_ {m i n} / 2 m (19) \\ = \widetilde {V} _ {i, h} ^ {\pi^ {\dagger} \times \pi_ {- i} ^ {\prime}} (s) - c \Delta_ {m i n} / m \\ \end{array}
$$
with some constant  $c > 0$ . Then,  $\pi'$  is not an CCE in such a case.
From induction on  $h = H, H - 1, \dots, 1$ ,  $\pi' = \pi^\dagger$  for any states. If every state  $s \in S$  is reachable at every step  $h \in [H]$  under the target policy,  $\pi^\dagger$  is the unique {NE, CE, CCE}.
G.2 Proof of Theorem 4
Consider an arbitrary Markov policy  $\pi$ . From  $d$ -portion attack strategy, we have
$$
\begin{array}{l} \widetilde {Q} _ {i, h} ^ {\pi} (s, \boldsymbol {a}) = \frac {d _ {h} (s , \boldsymbol {a})}{m} R _ {i, h} (s, \pi_ {h} ^ {\dagger} (s)) + \frac {m - d _ {h} (s , \boldsymbol {a})}{m} R _ {i, h} (s, \pi_ {h} ^ {-} (s)) \\ + \frac {d _ {h} (s , \boldsymbol {a})}{m} \mathbb {P} _ {h} \widetilde {V} _ {i, h + 1} ^ {\pi} (s, \pi_ {h} ^ {\dagger} (s)) + \frac {m - d _ {h} (s , \boldsymbol {a})}{m} \mathbb {P} _ {h} \widetilde {V} _ {i, h + 1} ^ {\pi} (s, \pi_ {h} ^ {-} (s)) \\ = \frac {d _ {h} (s , \boldsymbol {a}) - m}{m} \left(R _ {i, h} \left(s, \pi_ {h} ^ {\dagger} (s)\right) - R _ {i, h} \left(s, \pi_ {h} ^ {-} (s)\right)\right) \tag {20} \\ + \frac {d _ {h} (s , \boldsymbol {a}) - m}{m} \left(\mathbb {P} _ {h} \widetilde {V} _ {i, h + 1} ^ {\pi} \left(s, \pi_ {h} ^ {\dagger} (s)\right) - \mathbb {P} _ {h} \widetilde {V} _ {i, h + 1} ^ {\pi} \left(s, \pi_ {h} ^ {-} (s)\right)\right) \\ + R _ {i, h} (s, \pi_ {h} ^ {\dagger} (s)) + \mathbb {P} _ {h} \widetilde {V} _ {i, h + 1} ^ {\pi} (s, \pi_ {h} ^ {\dagger} (s)) \\ \end{array}
$$
$$
\begin{array}{l} \widetilde {V} _ {i, h} ^ {\pi} (s) = \mathbb {D} _ {\pi_ {h}} [ \widetilde {Q} _ {i, h} ^ {\pi} ] (s) \\ = \frac {\mathbb {D} _ {\pi_ {h}} [ d ] (s) - m}{m} \left(R _ {i, h} \left(s, \pi_ {h} ^ {\dagger} (s)\right) - R _ {i, h} \left(s, \pi_ {h} ^ {-} (s)\right)\right) \tag {21} \\ + \frac {\mathbb {D} _ {\pi_ {h}} [ d ] (s) - m}{m} \left(\mathbb {P} _ {h} \widetilde {V} _ {i, h + 1} ^ {\pi} (s, \pi_ {h} ^ {\dagger} (s)) - \mathbb {P} _ {h} \widetilde {V} _ {i, h + 1} ^ {\pi} (s, \pi_ {h} ^ {-} (s))\right) \\ + R _ {i, h} (s, \pi_ {h} ^ {\dagger} (s)) + \mathbb {P} _ {h} \widetilde {V} _ {i, h + 1} ^ {\pi} (s, \pi_ {h} ^ {\dagger} (s)). \\ \end{array}
$$
Now we bound the difference between  $\widetilde{V}_{i,h}^{\pi}(s)$  and  $\widetilde{V}_{i,h}^{\pi^{\dagger}}(s)$  for any policy  $\pi$ .
$$
\widetilde {V} _ {i, h} ^ {\pi^ {\dagger}} (s) - \widetilde {V} _ {i, h} ^ {\pi} (s) = \underbrace {\widetilde {V} _ {i , h} ^ {\pi^ {\dagger}} (s) - \mathbb {D} _ {\pi_ {h}} [ \widetilde {Q} _ {i , h} ^ {\pi^ {\dagger}} ] (s)} _ {(a)} + \underbrace {\mathbb {D} _ {\pi_ {h}} [ \widetilde {Q} _ {i , h} ^ {\pi^ {\dagger}} ] (s) - \widetilde {V} _ {i , h} ^ {\pi} (s)} _ {(b)}. \tag {22}
$$
For term (a), from equations (20) and (21), we have
$$
\begin{array}{l} \widetilde {V} _ {i, h} ^ {\pi^ {\dagger}} (s) - \mathbb {D} _ {\pi_ {h}} [ \widetilde {Q} _ {i, h} ^ {\pi^ {\dagger}} ] (s) \\ = \frac {m - \mathbb {D} _ {\pi_ {h}} [ d ] (s)}{m} \left(R _ {i, h} \left(s, \pi_ {h} ^ {\dagger} (s)\right) - R _ {i, h} \left(s, \pi_ {h} ^ {-} (s)\right)\right) \tag {23} \\ + \frac {m - \mathbb {D} _ {\pi_ {h}} [ d ] (s)}{m} \left(\mathbb {P} _ {h} \widetilde {V} _ {i, h + 1} ^ {\pi^ {\dagger}} (s, \pi_ {h} ^ {\dagger} (s)) - \mathbb {P} _ {h} \widetilde {V} _ {i, h + 1} ^ {\pi^ {\dagger}} (s, \pi_ {h} ^ {-} (s))\right). \\ \end{array}
$$
Since the attacker does not attack when the agents follow the target policy, we have  $\widetilde{V}_{i,h+1}^{\pi^{\dagger}}(s) = V_{i,h+1}^{\pi^{\dagger}}(s)$ .
$$
\widetilde {V} _ {i, h} ^ {\pi^ {\dagger}} (s) - \mathbb {D} _ {\pi_ {h}} [ \widetilde {Q} _ {i, h} ^ {\pi^ {\dagger}} ] (s) = \frac {m - \mathbb {D} _ {\pi_ {h}} [ d ] (s)}{m} \left(Q _ {i, h} ^ {\pi^ {\dagger}} (s, \pi_ {h} ^ {\dagger} (s)) - Q _ {i, h} ^ {\pi^ {\dagger}} (s, \pi_ {h} ^ {-} (s))\right). \tag {24}
$$
Denote  $\Delta_{i,h}^{\dagger -}(s) = Q_{i,h}^{\pi^{\dagger}}(s,\pi_{h}^{\dagger}(s)) - Q_{i,h}^{\pi^{\dagger}}(s,\pi_{h}^{-}(s))$ . We have
$$
\widetilde {V} _ {i, h} ^ {\pi^ {\dagger}} (s) - \mathbb {D} _ {\pi_ {h}} [ \widetilde {Q} _ {i, h} ^ {\pi^ {\dagger}} ] (s) = \frac {\Delta_ {i , h} ^ {\ddagger -} (s)}{2 m} \mathbb {E} _ {\boldsymbol {a} \sim \pi_ {h} (\cdot | s)} \left[ \sum_ {i = 1} ^ {m} \mathbb {1} \left(a _ {i} \neq \pi_ {i, h} ^ {\dagger} (s)\right) \right]. \tag {25}
$$
For term (b), from equations (20) and (21), we have
$$
\begin{array}{l} \mathbb {D} _ {\pi_ {h}} [ \widetilde {Q} _ {i, h} ^ {\pi^ {\dagger}} ] (s) - \widetilde {V} _ {i, h} ^ {\pi} (s) \\ = \frac {\mathbb {D} _ {\pi_ {h}} [ d ] (s)}{m} \mathbb {P} _ {h} \widetilde {V} _ {i, h + 1} ^ {\pi^ {\dagger}} (s, \pi_ {h} ^ {\dagger} (s)) + \frac {m - \mathbb {D} _ {\pi_ {h}} [ d ] (s)}{m} \mathbb {P} _ {h} \widetilde {V} _ {i, h + 1} ^ {\pi^ {\dagger}} (s, \pi_ {h} ^ {-} (s)) \\ - \frac {\mathbb {D} _ {\pi_ {h}} [ d ] (s)}{m} \mathbb {P} _ {h} \widetilde {V} _ {i, h + 1} ^ {\pi} \left(s, \pi_ {h} ^ {\dagger} (s)\right) - \frac {m - \mathbb {D} _ {\pi_ {h}} [ d ] (s)}{m} \mathbb {P} _ {h} \widetilde {V} _ {i, h + 1} ^ {\pi} \left(s, \pi_ {h} ^ {-} (s)\right) \tag {26} \\ = \frac {\mathbb {D} _ {\pi_ {h}} [ d ] (s)}{m} \mathbb {P} _ {h} \left[ \widetilde {V} _ {i, h + 1} ^ {\pi^ {\dagger}} - \widetilde {V} _ {i, h + 1} ^ {\pi} \right] \left(s, \pi_ {h} ^ {\dagger} (s)\right) \\ + \frac {m - \mathbb {D} _ {\pi_ {h}} [ d ] (s)}{m} \mathbb {P} _ {h} [ \widetilde {V} _ {i, h + 1} ^ {\pi^ {\dagger}} - \widetilde {V} _ {i, h + 1} ^ {\pi} ] (s, \pi_ {h} ^ {-} (s)) \\ = \mathbb {E} _ {s ^ {\prime} \sim P _ {h} (\cdot | s, \bar {a}), \bar {a} \sim \mathbb {A} _ {h} (\cdot | s, a), a \sim \pi_ {h} (\cdot | s)} \left[ \widetilde {V} _ {i, h + 1} ^ {\pi^ {\dagger}} (s ^ {\prime}) - \widetilde {V} _ {i, h + 1} ^ {\pi} (s ^ {\prime}) \right]. \\ \end{array}
$$
By combining terms (a) and (b), we have
$$
\begin{array}{l} \widetilde {V} _ {i, h} ^ {\pi^ {\dagger}} (s _ {h}) - \widetilde {V} _ {i, h} ^ {\pi} (s _ {h}) \\ \begin{array}{l} = \frac {\Delta_ {i , h} ^ {\dagger -} (s _ {h})}{2 m} \mathbb {E} _ {\boldsymbol {a} \sim \pi_ {h} (\cdot | s _ {h})} \left[ \mathbb {1} \left(a _ {i} \neq \pi_ {i, h} ^ {\dagger} (s _ {h})\right) \right] \\ + \mathbb {E} _ {s _ {h + 1} \sim P _ {h} (\cdot | s _ {h}, \widetilde {\boldsymbol {a}}), \widetilde {\boldsymbol {a}} \sim \mathbb {A} _ {h} (\cdot | s _ {h}, \boldsymbol {a}), \boldsymbol {a} \sim \pi_ {h} (\cdot | s _ {h})} \left[ \widetilde {V} _ {i, h + 1} ^ {\pi^ {\dagger}} (s _ {h + 1}) - \widetilde {V} _ {i, h + 1} ^ {\pi} (s _ {h + 1}) \right] \end{array} \tag {27} \\ = \dots = \mathbb {E} _ {\pi , \mathbb {A}, P} \left[ \sum_ {h ^ {\prime} = h} ^ {H} \sum_ {i = 1} ^ {m} \mathbb {1} \left(a _ {i, h ^ {\prime}} \neq \pi_ {i, h ^ {\prime}} ^ {\dagger} \left(s _ {h ^ {\prime}}\right)\right) \frac {\Delta_ {i , h ^ {\prime}} ^ {\dagger -} \left(s _ {h ^ {\prime}}\right)}{2 m} \right]. \\ \end{array}
$$
From the definition of the best-in-hindsight regret and (27), we have
$$
\begin{array}{l} \operatorname {R e g} _ {i} (K, H) = \max  _ {\pi_ {i} ^ {\prime}} \sum_ {k = 1} ^ {K} \left[ \widetilde {V} _ {i, 1} ^ {\pi_ {i} ^ {\prime} \times \pi_ {- i} ^ {k}} \left(s _ {1} ^ {k}\right) - \widetilde {V} _ {i, 1} ^ {\pi^ {k}} \left(s _ {1} ^ {k}\right) \right] \tag {28} \\ \geq \sum_ {k = 1} ^ {K} \left[ \widetilde {V} _ {i, 1} ^ {\pi_ {i} ^ {\dagger} \times \pi_ {- i} ^ {k}} \left(s _ {1} ^ {k}\right) - \widetilde {V} _ {i, 1} ^ {\pi^ {k}} \left(s _ {1} ^ {k}\right) \right]. \\ \end{array}
$$
Now, we bound  $\sum_{i=1}^{m}[\widetilde{V}_{i,1}^{\pi_i^\dagger \times \pi_{-i}}(s_1) - \widetilde{V}_{i,1}^\pi(s_1)]$  for any policy  $\pi$ . We introduce some special strategy modifications  $\{\phi_{i,h}^\dagger\}_{h=1}^H$ . For any  $h' \geq h$ , we have  $\phi_{i,h}^\dagger \diamond \pi_{i,h'}(s) = \pi_{i,h'}^\dagger(s)$  and for any  $h' < h$ , we have  $\phi_{i,h}^\dagger \diamond \pi_{i,h'}(s) = \pi_{i,h'}(s)$ . Thus,
$$
\begin{array}{l} \sum_ {i = 1} ^ {m} \left[ \widetilde {V} _ {i, 1} ^ {\pi_ {i} ^ {\dagger} \times \pi_ {- i}} \left(s _ {1}\right) - \widetilde {V} _ {i, 1} ^ {\pi} \left(s _ {1}\right) \right] \tag {29} \\ = \sum_ {h = 1} ^ {H} \sum_ {i = 1} ^ {m} \left[ \widetilde {V} _ {i, 1} ^ {\phi_ {i, h} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} (s _ {1}) - \widetilde {V} _ {i, 1} ^ {\phi_ {i, h + 1} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} (s _ {1}) \right]. \\ \end{array}
$$
When  $h = H$ , we have
$$
\begin{array}{l} \sum_ {i = 1} ^ {m} \left(\widetilde {V} _ {i, 1} ^ {\phi_ {i, H} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} (s _ {1}) - \widetilde {V} _ {i, 1} ^ {\phi_ {i, H + 1} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} (s _ {1})\right) \\ = \mathbb {E} _ {\pi , \mathrm {A}, P} \left[ \sum_ {i = 1} ^ {m} \left(\widetilde {V} _ {i, H} ^ {\phi_ {i, H} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} \left(s _ {H}\right) - \widetilde {V} _ {i, H} ^ {\phi_ {i, H + 1} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} \left(s _ {H}\right)\right) \right] \tag {30} \\ = \mathbb {E} _ {\pi , \mathbb {A}, P} \left[ \sum_ {i = 1} ^ {m} \left(\widetilde {V} _ {i, H} ^ {\pi_ {i} ^ {\dagger} \times \pi_ {- i}} (s _ {H}) - \widetilde {V} _ {i, H} ^ {\pi} (s _ {H})\right) \right] \\ = \mathbb {E} _ {\pi , \mathrm {A}, P} \left[ \sum_ {i = 1} ^ {m} \mathbb {1} \left(a _ {i, H} \neq \pi_ {i, H} ^ {\dagger} \left(s _ {H}\right)\right) \frac {\Delta_ {i , H} ^ {\dagger -} \left(s _ {H}\right)}{2 m} \right]. \\ \end{array}
$$
For  $h <   H$  , we have
$$
\begin{array}{l} \sum_ {i = 1} ^ {m} \left(\widetilde {V} _ {i, 1} ^ {\phi_ {i, h} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} (s _ {1}) - \widetilde {V} _ {i, 1} ^ {\phi_ {i, h + 1} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} (s _ {1})\right) \\ = \mathbb {E} _ {\pi , \mathbb {A}, P} \left[ \sum_ {i = 1} ^ {m} \left(\widetilde {V} _ {i, h} ^ {\phi_ {i, h} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} (s _ {h}) - \widetilde {V} _ {i, h} ^ {\phi_ {i, h + 1} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} (s _ {h})\right) \right] \\ = \mathbb {E} _ {\pi , \mathbb {A}, P} \left[ \sum_ {i = 1} ^ {m} \left(\mathbb {D} _ {\phi_ {i, h} ^ {\dagger} \diamond \pi_ {i, h} \times \pi_ {- i, h}} - \mathbb {D} _ {\phi_ {i, h + 1} ^ {\dagger} \diamond \pi_ {i, h} \times \pi_ {- i, h}}\right) \left[ \widetilde {Q} _ {i, h} ^ {\phi_ {i, h + 1} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} \right] (s _ {h}) \right] \tag {31} \\ = \mathbb {E} _ {\pi , \mathbb {A}, P} \left[ \sum_ {i = 1} ^ {m} \frac {1 - \pi_ {i , h} (s _ {h} , \pi_ {i , h} ^ {\dagger} (s _ {h}))}{2 m} (\mathbb {D} _ {\pi^ {\dagger}} - \mathbb {D} _ {\pi^ {-}}) \left[ R _ {i, h} + \mathbb {P} _ {h} \widetilde {V} _ {i, h + 1} ^ {\phi_ {i, h + 1} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} \right] (s _ {h}) \right] \\ \end{array}
$$
where the second equation holds as  $\phi_{i,h}^{\dagger}\diamond \pi_{i}\times \pi_{-i} = \phi_{i,h + 1}^{\dagger}\diamond \pi_{i}\times \pi_{-i}$  at any time step  $h^\prime >h$  and the last equation holds from equation (20).
Note that  $Q_{i,h}^{\pi^{\dagger}} = R_{i,h} + \mathbb{P}_{h}V_{i,h + 1}^{\pi^{\dagger}} = R_{i,h} + \mathbb{P}_{h}\widetilde{V}_{i,h + 1}^{\pi^{\dagger}}$ . From equation (31), we have
$$
\begin{array}{l} \sum_ {i = 1} ^ {m} \left(\widetilde {V} _ {i, 1} ^ {\phi_ {i, h} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} (s _ {1}) - \widetilde {V} _ {i, 1} ^ {\phi_ {i, h + 1} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} (s _ {1})\right) \\ = \underbrace {\mathbb {E} _ {\pi , \mathbb {A} , P} \left[ \sum_ {i = 1} ^ {m} \frac {1 - \pi_ {i , h} \left(s _ {h} , \pi_ {i , h} ^ {\dagger} (s _ {h})\right)}{2 m} \left(\mathbb {D} _ {\pi^ {\dagger}} - \mathbb {D} _ {\pi^ {-}}\right) \left[ Q _ {i , h} ^ {\pi^ {\dagger}} \right] (s _ {h}) \right]} _ {①} \\ + \underbrace {\mathbb {E} _ {\pi , \mathrm {A} , P} \left[ \sum_ {i = 1} ^ {m} \frac {1 - \pi_ {i , h} \left(s _ {h} , \pi_ {i , h} ^ {\dagger} \left(s _ {h}\right)\right)}{2 m} \left(\mathbb {D} _ {\pi^ {\dagger}} - \mathbb {D} _ {\pi^ {-}}\right) \left[ \mathbb {P} _ {h} \widetilde {V} _ {i , h + 1} ^ {\phi_ {i , h + 1} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} - \mathbb {P} _ {h} \widetilde {V} _ {i , h + 1} ^ {\pi^ {\dagger}} \right] (s _ {h}) \right]} _ {\text {②}}. \tag {32} \\ \end{array}
$$
Denote  $\Delta_{i,h}^{\dagger -}(s) = Q_{i,h}^{\pi^{\dagger}}(s,\pi_{h}^{\dagger}(s)) - Q_{i,h}^{\pi^{\dagger}}(s,\pi_{h}^{-}(s))$ . Thus,
$$
① = \mathbb {E} _ {\pi , \mathbb {A}, P} \left[ \sum_ {i = 1} ^ {m} \frac {1 - \pi_ {i , h} \left(s _ {h} , \pi_ {i , h} ^ {\dagger} \left(s _ {h}\right)\right)}{2 m} \Delta_ {i, h} ^ {\dagger -} \left(s _ {h}\right) \right]. \tag {33}
$$
Now, we bound item ②. If  $(\mathbb{D}_{\pi^{\dagger}} - \mathbb{D}_{\pi^{-}})\left[\mathbb{P}_h\widetilde{V}_{i,h + 1}^{\phi_{i,h + 1}^{\dagger}\diamond \pi_i\times \pi_{-i}} - \mathbb{P}_h\widetilde{V}_{i,h + 1}^{\pi^{\dagger}}\right](s_h)\geq 0,$
$$
\begin{array}{l} \left(\mathbb {D} _ {\pi^ {\dagger}} - \mathbb {D} _ {\pi^ {-}}\right) \left[ \mathbb {P} _ {h} \widetilde {V} _ {i, h + 1} ^ {\phi_ {i, h + 1} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} - \mathbb {P} _ {h} \widetilde {V} _ {i, h + 1} ^ {\pi^ {\dagger}} \right] (s _ {h}) \\ \geq \frac {2 \mathbb {D} _ {\pi_ {h}} [ d ] (s _ {h})}{m} \mathbb {D} _ {\pi^ {\dagger}} \mathbb {P} _ {h} \left[ \widetilde {V} _ {i, h + 1} ^ {\phi_ {i, h + 1} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} - \widetilde {V} _ {i, h + 1} ^ {\pi^ {\dagger}} \right] (s _ {h}) \tag {34} \\ + \frac {2 (m - \mathbb {D} _ {\pi_ {h}} [ d ] (s _ {h}))}{m} \mathbb {D} _ {\pi^ {-}} \mathbb {P} _ {h} [ \widetilde {V} _ {i, h + 1} ^ {\phi_ {i, h + 1} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} - \widetilde {V} _ {i, h + 1} ^ {\pi^ {\dagger}} ] (s _ {h}) \\ = 2 \mathbb {E} _ {\pi , \mathbb {A}, P} \left[ \widetilde {V} _ {i, h + 1} ^ {\phi_ {i, h + 1} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} (s _ {h + 1}) - \widetilde {V} _ {i, h + 1} ^ {\pi^ {\dagger}} (s _ {h + 1}) \right], \\ \end{array}
$$
because the RHS of the inequality is smaller or equal to 0.
$$
\begin{array}{l} \text {I f} \left(\mathbb {D} _ {\pi^ {\dagger}} - \mathbb {D} _ {\pi^ {-}}\right) \left[ \mathbb {P} _ {h} \widetilde {V} _ {i, h + 1} ^ {\phi_ {i, h + 1} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} - \mathbb {P} _ {h} \widetilde {V} _ {i, h + 1} ^ {\pi^ {\dagger}} \right] (s _ {h}) \leq 0, \\ \left(\mathbb {D} _ {\pi^ {\dagger}} - \mathbb {D} _ {\pi^ {-}}\right) \left[ \mathbb {P} _ {h} \widetilde {V} _ {i, h + 1} ^ {\phi_ {i, h + 1} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} - \mathbb {P} _ {h} \widetilde {V} _ {i, h + 1} ^ {\pi^ {\dagger}} \right] (s _ {h}) \\ \geq \frac {2 \mathbb {D} _ {\pi_ {h}} [ d ] (s _ {h})}{m} (\mathbb {D} _ {\pi^ {\dagger}} - \mathbb {D} _ {\pi^ {-}}) \left[ \mathbb {P} _ {h} \widetilde {V} _ {i, h + 1} ^ {\phi_ {i, h + 1} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} - \mathbb {P} _ {h} \widetilde {V} _ {i, h + 1} ^ {\pi^ {\dagger}} \right] (s _ {h}) \\ \geq \frac {2 \mathbb {D} _ {\pi_ {h}} [ d ] (s _ {h})}{m} \mathbb {D} _ {\pi^ {\dagger}} \mathbb {P} _ {h} \left[ \widetilde {V} _ {i, h + 1} ^ {\phi_ {i, h + 1} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} - \widetilde {V} _ {i, h + 1} ^ {\pi^ {\dagger}} \right] (s _ {h}) \tag {35} \\ + \frac {2 \left(m - \mathbb {D} _ {\pi_ {h}} [ d ] \left(s _ {h}\right)\right)}{m} \mathbb {D} _ {\pi^ {-}} \mathbb {P} _ {h} \left[ \widetilde {V} _ {i, h + 1} ^ {\phi_ {i, h + 1} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} - \widetilde {V} _ {i, h + 1} ^ {\pi^ {\dagger}} \right] (s _ {h}) \\ = 2 \mathbb {E} _ {\pi , \mathbb {A}, P} \left[ \widetilde {V} _ {i, h + 1} ^ {\phi^ {\dagger} _ {i, h + 1} \diamond \pi_ {i} \times \pi_ {- i}} (s _ {h + 1}) - \widetilde {V} _ {i, h + 1} ^ {\pi^ {\dagger}} (s _ {h + 1}) \right]. \\ \end{array}
$$
From (27), we have
$$
\begin{array}{l} \widetilde {V} _ {i, h + 1} ^ {\pi^ {\dagger}} \left(s _ {h + 1}\right) - \widetilde {V} _ {i, h + 1} ^ {\phi_ {i, h + 1} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} \left(s _ {h + 1}\right) \\ = \mathbb {E} _ {\phi_ {i, h + 1} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}, \mathbb {A}, P} \left[ \sum_ {h ^ {\prime} = h + 1} ^ {H} \sum_ {i = 1} ^ {m} \mathbb {1} \left(a _ {i, h ^ {\prime}} \neq \pi_ {i, h ^ {\prime}} ^ {\dagger} \left(s _ {h ^ {\prime}}\right)\right) \frac {\Delta_ {i , h ^ {\prime}} ^ {\dagger -} \left(s _ {h ^ {\prime}}\right)}{2 m} \right] \tag {36} \\ \leq \sum_ {h ^ {\prime} = h + 1} ^ {H} (m - 1) \max  _ {s \in S, i \in [ m ]} \frac {\Delta_ {i , h ^ {\prime}} ^ {\dagger -} (s)}{2 m} \\ \leq \frac {(m - 1)}{2 m} \Delta_ {i, h} ^ {\dagger -} (s _ {h}), \\ \end{array}
$$
where the last inequality holds when  $\min_{s\in \mathcal{S},i\in [m]}\Delta_{i,h}^{\dagger -}(s)\geq \sum_{h' = h + 1}^{H}\max_{s\in \mathcal{S},i\in [m]}\Delta_{i,h'}^{\dagger -}(s)$ . Combine the above inequalities, we have
$$
\geq - \mathbb {E} _ {\pi , \mathrm {A}, P} \left[ \sum_ {i = 1} ^ {m} \frac {1 - \pi_ {i , h} \left(s _ {h} , \pi_ {i , h} ^ {\dagger} \left(s _ {h}\right)\right)}{2 m} \frac {(m - 1)}{m} \Delta_ {i, h} ^ {\dagger -} \left(s _ {h}\right) \right], \tag {37}
$$
$$
\begin{array}{l} \sum_ {i = 1} ^ {m} \left(\widetilde {V} _ {i, 1} ^ {\phi_ {i, h} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} (s _ {1}) - \widetilde {V} _ {i, 1} ^ {\phi_ {i, h + 1} ^ {\dagger} \diamond \pi_ {i} \times \pi_ {- i}} (s _ {1})\right) \\ = ① + ② \\ \geq \mathbb {E} _ {\pi , \mathbb {A}, P} \left[ \sum_ {i = 1} ^ {m} \frac {1 - \pi_ {i , h} \left(s _ {h} , \pi_ {i , h} ^ {\dagger} \left(s _ {h}\right)\right)}{2 m ^ {2}} \Delta_ {i, h} ^ {\dagger -} \left(s _ {h}\right) \right] \tag {38} \\ = \mathbb {E} _ {\pi , \mathbb {A}, P} \left[ \sum_ {i = 1} ^ {m} \mathbb {1} \left(a _ {i, h} \neq \pi_ {i, h} ^ {\dagger} \left(s _ {h}\right)\right) \frac {\Delta_ {i , h} ^ {\dagger -} \left(s _ {h}\right)}{2 m ^ {2}} \right] \\ \geq \mathbb {E} _ {\pi , \mathbb {A}, P} \left[ \sum_ {i = 1} ^ {m} \mathbb {1} \left(a _ {i, h} \neq \pi_ {i, h} ^ {\dagger} \left(s _ {h}\right)\right) \right] \frac {\Delta_ {m i n}}{2 m ^ {2}}. \\ \end{array}
$$
$$
\operatorname {R e g} _ {i} (K, H) \geq \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi , \mathbb {A}, P} \left[ \sum_ {i = 1} ^ {m} \mathbb {1} \left(a _ {i, h} \neq \pi_ {i, h} ^ {\dagger} \left(s _ {h}\right)\right) \right] \frac {\Delta_ {m i n}}{2 m ^ {2}} = \mathbb {E} [ \text {L o s s} 1 (K, H) ] \frac {\Delta_ {m i n}}{2 m ^ {2}}. \tag {39}
$$
If the best-in-hindsight regret  $\operatorname{Reg}(K, H)$  of each agent's algorithm is bounded by a sub-linear bound  $\mathcal{R}(T)$ , then the attack loss is bounded by  $\mathbb{E}[\operatorname{Loss1}(K, H)] \leq 2m^2 \mathcal{R}(T) / \Delta_{min}$ .
The  $d$ -portion attack strategy attacks all agents when any agent  $i$  chooses an non-target action. We have
$$
\begin{array}{l} \operatorname {C o s t} (K, H) = \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \sum_ {i = 1} ^ {m} \left(\mathbb {1} \left(\widetilde {a} _ {i, h} ^ {k} \neq a _ {i, h} ^ {k}\right) + | \widetilde {r} _ {i, h} ^ {k} - r _ {i, h} ^ {k} |\right) \\ \leq \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \sum_ {i = 1} ^ {m} \mathbb {1} \left[ \widetilde {a} _ {i, h} ^ {k} \neq a _ {i, h} ^ {k} \right] m. \\ \end{array}
$$
Then, the attack cost is bounded by  $m\mathbb{E}[\mathrm{Loss1}(K,H)] \leq 2m^3\mathcal{R}(T) / \Delta_{min}$ .
H Analysis of the  $\eta$ -gap attack
H.1 Proof of Theorem 5