SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} _ {s _ {1} \sim P _ {0} (\cdot)} \left[ V _ {\dagger , 1} ^ {\pi^ {*}} (s _ {1}) - V _ {\dagger , 1} ^ {\pi^ {\dagger}} (s _ {1}) \right] \\ \leq \mathbb {E} _ {s _ {1} \sim \hat {\mathbb {P}} _ {0} ^ {k} (\cdot)} \left[ V _ {\dagger , 1} ^ {\pi^ {*}} (s _ {1}) - V _ {\dagger , 1} ^ {\pi^ {\dagger}} (s _ {1}) \right] + H \sqrt {S \log (2 \tau / p) / (2 k)} \tag {70} \\ \leq H ^ {2} \sqrt {\frac {S \log (2 H \tau / p)}{2 \tau^ {3}}} + 3 H \sqrt {2 S \log (2 \tau / p) / \tau} + H (H \sqrt {S} + 1) \sqrt {2 S A \log (2 S A H \tau / p) / \tau} \\ \leq 2 H ^ {2} S \sqrt {2 A \log (2 S A H \tau / p) / \tau}, \\ \end{array}
$$
where the last inequality holds when  $S, H, A \geq 2$ . Similarly,
$$
\sum_ {k = 1} ^ {K} \left[ V _ {\dagger , 1} ^ {\pi^ {*}} \left(s _ {1} ^ {k}\right) - V _ {\dagger , 1} ^ {\pi^ {\dagger}} \left(s _ {1} ^ {k}\right) \right] \leq 2 H ^ {2} S \sqrt {2 A \log (2 S A H \tau / p) / \tau}. \tag {71}
$$
J.2 Proof of Theorem 9
For completeness, we describe the main steps of V-learning algorithm [Jin et al., 2021] in Algorithm 2 and the adversarial bandit algorithm in Algorithm 3.
Algorithm 2: V-learning [Jin et al., 2021]
1: For any  $(s,a,h),V_h(s)\gets H + 1 - h,N_h(s)\gets 0,\pi_h(a|s)\gets 1 / A.$    
2: for episodes  $k = 1,\ldots ,K$  do   
3: receive  $s_1$    
4: for episodes  $h = 1,\dots ,H$  do   
5: take action  $a_{h}\sim \pi_{h}(\cdot |s_{h})$  , observe reward  $r_h$  and next state  $s_{h + 1}$    
6:  $t = N_{h}(s_{h})\leftarrow N_{h}(s_{h}) + 1$    
7:  $\overline{V}_h(s_h)\gets (1 - \alpha_t)\overline{V}_h(s_h) + \alpha_t(rh + V_{h + 1}(s_{h + 1}) + \beta_t).$    
8:  $V_{h}(s_{h})\gets \min \{H + 1 - h,\overline{V}_{h}(s_{h})\}$    
9:  $\pi_h(\cdot |s_h)\gets \mathrm{ADV\_BANDIT\_UPDATE}(a_h,\frac{H - r_h - V_{h + 1}(s_{h + 1})}{H})$  on  $(s_h,h)^{\text{th}}$  adversarial bandit.
11: end for
Algorithm 3: FTRL for Weighted External Regret [Jin et al., 2021]
1: For any  $b \in \mathcal{B}$ ,  $\theta_{1}(b) \gets 1 / B$ .  
2: for episode  $t = 1, \ldots, K$  do  
3: Take action  $b_{t} \sim \theta_{t}(\cdot)$ , and observe loss  $\widetilde{l}_{t}(b_{t})$ .  
4:  $\hat{l}_{t}(b) \gets \widetilde{l}_{t}(b_{t}) \mathbb{1}[b_{t} = b] / (\theta_{t}(b) + \gamma_{t})$  for all  $b \in \mathcal{B}$ .  
5:  $\theta_{t+1}(b) \propto \exp[-(\gamma_{t} / w_{t}) \sum_{i=1}^{t} w_{i} \hat{l}_{i}(b)]$   
6: end for
We use the same learning rate  $\alpha_{t}$  in [Jin et al., 2021]. We also use an auxiliary sequence  $\{\alpha_t^i\}_{i = 1}^t$  defined in [Jin et al., 2021] based on the learning rate, which will be frequently used in the proof:
$$
\alpha_ {t} = \frac {H + 1}{H + t}, \alpha_ {t} ^ {0} = \prod_ {j = 1} ^ {t} (1 - \alpha_ {j}), \alpha_ {t} ^ {i} = \alpha_ {i} \prod_ {j = i + 1} ^ {t} (1 - \alpha_ {j}). \tag {72}
$$
We follow the requirement for the adversarial bandit algorithm used in V-learning, which is to have a high probability weighted external regret guarantee as follows.
Assumption 1 For any  $t \in \mathbb{N}$  and any  $\delta \in (0,1)$ , with probability at least  $1 - \delta$ , we have
$$
\max  _ {\theta} \sum_ {j = 1} ^ {t} \alpha_ {t} ^ {j} [ \langle \theta_ {j}, l _ {j} \rangle - \langle \theta , l _ {j} \rangle ] \leq \xi (B, t, \log (1 / \delta)). \tag {73}
$$
In addition, there exists an upper bound  $\Xi(B,t,\log(1/\delta)) \geq \sum_{t'=1}^{t} \xi(B,t,\log(1/\delta))$  where (i)  $\xi(B,t,\log(1/\delta))$  is non-decreasing in  $B$  for any  $t$ ,  $\delta$ ; (ii)  $\Xi(B,t,\log(1/\delta))$  is concave in  $t$  for any  $B$ ,  $\delta$ .
In particular, it was proved in [Jin et al., 2021] that the Follow-the-Regularized-Leader (FTRL) algorithm (Algorithm 3) satisfies Assumption 1 with bounds  $\xi(B,t,\log(1/\delta)) \leq \mathcal{O}(\sqrt{HB\log(B/\delta)/t})$  and  $\Xi(B,t,\log(1/\delta)) \leq \mathcal{O}(\sqrt{HBt\log(B/\delta)})$ . By choosing hyper-parameter  $w_t = \alpha_t\left(\prod_{i=2}^t (1 - \alpha_i)\right)^{-1}$  and  $\gamma_t = \sqrt{\frac{H\log B}{Bt}}$ ,  $\xi(B,t,\log(1/\delta)) = 10\sqrt{HB\log(B/\delta)/t}$  and  $\Xi(B,t,\log(1/\delta)) = 20\sqrt{HBt\log(B/\delta)}$ .
We use  $V^k, N^k, \pi^k$  to denote the value, counter and policy maintained by V-learning algorithm at the beginning of the episode  $k$ . Suppose  $s$  was previously visited at episodes  $k^1, \dots, k^t < k$  at the step  $h$ . Set  $t'$  such that  $k^{t'} \leq \tau$  and  $k^{t' + 1} > \tau$ .
In the exploration phase of the proposed approximate mixed attack strategy, the rewards are equal to 1 for any state  $s$ , any action  $\pmb{a}$ , any agent  $i$  and any step  $h$ . The loss updated to the adversarial bandit update step in Algorithm 2 is equal to  $\frac{h - 1}{H}$ .
In the attack phase, the expected loss updated to the adversarial bandit update step in Algorithm 2 is equal to
$$
\sum_ {j = 1} ^ {t ^ {\prime}} \alpha_ {t} ^ {j} \frac {h - 1}{H} + \sum_ {j = t ^ {\prime} + 1} ^ {t} \alpha_ {t} ^ {j} \mathbb {D} _ {\pi^ {\dagger}} \left(\frac {H - \mathbb {P} _ {h} V _ {i , h + 1} ^ {k ^ {j}}}{H}\right) (s) + \sum_ {j = t ^ {\prime} + 1} ^ {t} \alpha_ {t} ^ {j} \mathbb {D} _ {\pi_ {h} ^ {k ^ {j}}} \left(\frac {- \widetilde {r} _ {i , h}}{H}\right) (s). \tag {74}
$$
Thus, in both of the exploration phase and the attack phase,  $\pi^{\dagger}$  is the best policy for the adversarial bandit algorithm.
By Assumption 1 and the adversarial bandit update step in Algorithm 2, with probability at least  $1 - \delta$ , for any  $(s,h) \in S \times [H]$  and any  $k > \tau$ , we have
$$
\begin{array}{l} \xi (A, t, \iota) \geq \sum_ {j = 1} ^ {t ^ {\prime}} \alpha_ {t} ^ {j} \frac {h - 1}{H} + \sum_ {j = t ^ {\prime} + 1} ^ {t} \alpha_ {t} ^ {j} \mathbb {D} _ {\pi^ {\dagger}} \left(\frac {H - \mathbb {P} _ {h} V _ {i , h + 1} ^ {k ^ {j}}}{H}\right) (s) + \sum_ {j = t ^ {\prime} + 1} ^ {t} \alpha_ {t} ^ {j} \mathbb {D} _ {\pi_ {h} ^ {k j}} \left(\frac {- \widetilde {r} _ {i , h}}{H}\right) (s) \\ - \sum_ {j = 1} ^ {t ^ {\prime}} \alpha_ {t} ^ {j} \frac {h - 1}{H} + \sum_ {j = t ^ {\prime} + 1} ^ {t} \alpha_ {t} ^ {j} \mathbb {D} _ {\pi^ {\dagger}} \left(\frac {H - \mathbb {P} _ {h} V _ {i , h + 1} ^ {k ^ {j}}}{H}\right) (s) + \sum_ {j = t ^ {\prime} + 1} ^ {t} \alpha_ {t} ^ {j} \mathbb {D} _ {\pi^ {\dagger}} \left(\frac {- \widetilde {r} _ {i , h}}{H}\right) (s) \\ = \sum_ {j = t ^ {\prime} + 1} ^ {t} \alpha_ {t} ^ {j} \left(1 - \pi_ {i, h} ^ {k ^ {j}} \left(\pi_ {i, h} ^ {\dagger} (s) | s\right)\right) \frac {r _ {i , h} \left(s , \pi_ {h} ^ {\dagger} (s)\right)}{H}, \tag {75} \\ \end{array}
$$
where  $\iota = \log (mHSAK / \delta)$
Note that  $R_{min} = \min_{h\in [H]}\min_{s\in S}\min_{i\in [m]}R_{i,h}(s,\pi_h^\dagger (s))$ . We have
$$
\frac {H}{R _ {\min }} \xi (A, t, \iota) \geq \sum_ {j = t ^ {\prime} + 1} ^ {t} \alpha_ {t} ^ {j} \left(1 - \pi_ {i, h} ^ {k ^ {j}} \left(\pi_ {i, h} ^ {\dagger} (s) | s\right)\right). \tag {76}
$$
Let  $n_h^k = N_h^k(s_h^k)$  and suppose  $s_h^k$  was previously visited at episodes  $k^1, \dots, k^{n_h^k} < k$  at the step  $h$ . Let  $k^j(s)$  denote the episode that  $s$  was visited in  $j$ -th time.
$$
\frac {H}{R _ {\text {m i n}}} \xi (A, n _ {h} ^ {k}, \iota) \geq \sum_ {j = N _ {h} ^ {\tau} \left(s _ {h} ^ {k}\right) + 1} ^ {n _ {h} ^ {k}} \alpha_ {n _ {h} ^ {k}} ^ {j} \left(1 - \pi_ {i, h} ^ {k ^ {j}} \left(\pi_ {i, h} ^ {\dagger} \left(s _ {h} ^ {k}\right) \mid s _ {h} ^ {k}\right)\right). \tag {77}
$$
According to the property of the learning rate  $\alpha_{t}$ , we have
$$
\frac {H}{R _ {\operatorname* {m i n}} \alpha_ {t} ^ {t}} \xi (A, n _ {h} ^ {k}, \iota) + \sum_ {j = N _ {h} ^ {\tau} \left(s _ {h} ^ {k}\right) + 1} ^ {n _ {h} ^ {k} - 1} \frac {H}{R _ {\operatorname* {m i n}}} \xi (A, j, \iota) \leq \sum_ {j = N _ {h} ^ {\tau} \left(s _ {h} ^ {k}\right) + 1} ^ {n _ {h} ^ {k}} \left(1 - \pi_ {i, h} ^ {k ^ {j}} \left(\pi_ {i, h} ^ {\dagger} \left(s _ {h} ^ {k}\right) | s _ {h} ^ {k}\right)\right). \tag {78}
$$
$$
\frac {4 0 H}{R _ {\text {m i n}}} \sqrt {H A n _ {h} ^ {k} \iota} \leq \sum_ {j = N _ {h} ^ {\tau} \left(s _ {h} ^ {k}\right) + 1} ^ {n _ {h} ^ {k}} \left(1 - \pi_ {i, h} ^ {k ^ {j}} \left(\pi_ {i, h} ^ {\dagger} \left(s _ {h} ^ {k}\right) \mid s _ {h} ^ {k}\right)\right). \tag {79}
$$
Computing the summation of the above inequality over  $h$  and  $s$ , we have
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {h = 1} ^ {H} \sum_ {k = \tau + 1} ^ {K} \mathbb {1} \left[ a _ {i, h} ^ {k} \neq \pi_ {i, h} ^ {\dagger} \left(s _ {h} ^ {k}\right) \right] \right] \\ = \sum_ {h = 1} ^ {H} \sum_ {s \in S} \sum_ {j = N _ {h} ^ {\tau} (s) + 1} ^ {N _ {h} ^ {K} (s)} \left(1 - \pi_ {i, h} ^ {k ^ {j} (s)} \left(\pi_ {i, h} ^ {\dagger} (s) | s\right)\right) \tag {80} \\ \leq \sum_ {s \in \mathcal {S}} \frac {4 0}{R _ {\min}} \sqrt {H ^ {5} A N _ {h} ^ {K} (s) \iota} \\ \leq \frac {4 0}{R _ {\min}} \sqrt {H ^ {5} A S K \iota}. \\ \end{array}
$$
In the exploration phase, the loss at each episode is up to  $H$ . In the attack phase, the expected number of episodes that the agents do not follow  $\pi^{\dagger}$  is up to  $\frac{40}{R_{min}} m\sqrt{H^{7}ASK\iota}$ .
According to Lemma 1, the attack loss is bounded by
$$
\mathbb {E} [ \operatorname {l o s s} (K, H) ] \leq H \tau + \frac {4 0}{R _ {\min }} m \sqrt {H ^ {9} A S K \iota} + 2 H ^ {2} S K \sqrt {2 A \iota / \tau}. \tag {81}
$$
In the exploration phase, the approximate mixed attack strategy attacks at any step and any episode. In the attack phase, the approximate mixed attack strategy only attacks agent  $i$  when agent  $i$  chooses a non-target action. We have
$$
\begin{array}{l} \operatorname {C o s t} (K, H) = \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \sum_ {i = 1} ^ {m} \left(\mathbb {1} \left(\widetilde {a} _ {i, h} ^ {k} \neq a _ {i, h} ^ {k}\right) + \left| \tilde {r} _ {i, h} ^ {k} - r _ {i, h} ^ {k} \right|\right) \tag {82} \\ \leq \sum_ {k = 1} ^ {\tau} \sum_ {h = 1} ^ {H} \sum_ {i = 1} ^ {m} (1 + 1) + \sum_ {k = \tau = 1} ^ {K} \sum_ {h = 1} ^ {H} \sum_ {i = 1} ^ {m} \mathbb {1} \left[ \tilde {a} _ {i, h} ^ {k} \neq a _ {i, h} ^ {k} \right] (1 + 1). \\ \end{array}
$$
Then, the attack cost is bounded by
$$
\mathbb {E} \left[ \operatorname {C o s t} (K, H) \right] \leq 2 m H \tau + \frac {8 0}{R _ {\min }} \sqrt {H ^ {5} A S K \iota}. \tag {83}
$$
For the executing output policy  $\hat{\pi}$  of V-learning, we have
$$
\begin{array}{l} 1 - \hat {\pi} _ {i, h} \left(\pi_ {i, h} ^ {\dagger} (s) | s\right) \\ = \frac {1}{K} \sum_ {k = 1} ^ {K} \sum_ {j = 1} ^ {N _ {h} ^ {k} (s)} \alpha_ {N _ {h} ^ {k} (s)} ^ {j} \left(1 - \pi_ {i, h} ^ {k ^ {j} (s)} \left(\pi_ {i, h} ^ {\dagger} (s) | s\right)\right) \\ = \frac {1}{K} \sum_ {k = \tau + 1} ^ {K} \sum_ {j = N _ {h} ^ {\tau} (s) + 1} ^ {N _ {h} ^ {k} (s)} \alpha_ {N _ {h} ^ {k} (s)} ^ {j} \left(1 - \pi_ {i, h} ^ {k ^ {j} (s)} \left(\pi_ {i, h} ^ {\dagger} (s) | s\right)\right) + \frac {1}{K} \sum_ {k = 1} ^ {\tau} \sum_ {j = 1} ^ {N _ {h} ^ {k} (s)} \alpha_ {N _ {h} ^ {k} (s)} ^ {j} \left(1 - \pi_ {i, h} ^ {k ^ {j} (s)} \left(\pi_ {i, h} ^ {\dagger} (s) | s\right)\right) \\ + \frac {1}{K} \sum_ {k = \tau + 1} ^ {K} \sum_ {j = 1} ^ {N _ {h} ^ {\tau} (s)} \alpha_ {N _ {h} ^ {k} (s)} ^ {j} \left(1 - \pi_ {i, h} ^ {k ^ {j} (s)} \left(\pi_ {i, h} ^ {\dagger} (s) | s\right)\right) \\ \leq \frac {2 0}{R _ {\text {m i n}}} \sqrt {\frac {H ^ {3} A \iota}{K}} + \frac {2 \tau}{K}. \tag {84} \\ \end{array}
$$
The probability that the agents with  $\hat{\pi}$  do not follow the target policy is bounded by  $\frac{20mS}{R_{min}}\sqrt{\frac{H^5A\iota}{K}} + \frac{2\tau mSH}{K}$ .
According to Lemma 1, the attack loss of the executing output policy  $\hat{\pi}$  is upper bounded by
$$
V _ {\dagger , 1} ^ {\pi^ {*}} \left(s _ {1}\right) - V _ {\dagger , 1} ^ {\hat {\pi}} \left(s _ {1}\right) \leq H \left(\frac {2 0 m S}{R _ {\min }} \sqrt {\frac {H ^ {5} A \iota}{K}} + \frac {2 \tau m S H}{K}\right) + 2 H ^ {2} S \sqrt {2 A \iota / \tau}. \tag {85}
$$