SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\max _ {i \in [ m ]} (\widetilde {V} _ {i, 1} ^ {\dagger , \pi_ {- i} ^ {\dagger}} (s) - \widetilde {V} _ {i, 1} ^ {\pi^ {\dagger}} (s)) = 0;
$$
if  $\pi$  is a product policy and  $\pi \neq \pi^{\dagger}$ , then  $\max_{i\in [m]}(\widetilde{V}_{i,1}^{\dagger,\pi-i}(s) - \widetilde{V}_{i,1}^{\pi}(s)) > 0$ ; (1)
if  $\pi (\pi^{\dagger}(s^{\prime})|s^{\prime}) = 0$  for all  $s^\prime$ , then  $\max_{i\in [m]}(\widetilde{V}_{i,1}^{\dagger,\pi_{-i}}(s) - \widetilde{V}_{i,1}^{\pi}(s)) > \epsilon$ ,
where  $\widetilde{V}$  is the expected reward based on the post-attack environments.
We now investigate whether there exist efficient and successful attack strategies that use action poisoning alone or reward poisoning alone. We first show that the power of action poisoning attack alone is limited.
Theorem 1 There exists a target policy  $\pi^{\dagger}$  and a  $MG\left(S, \{\mathcal{A}_i\}_{i=1}^m, H, P, \{R_i\}_{i=1}^m\right)$  such that no action poisoning Markov attack strategy alone can efficiently and successfully attack MARL agents by achieving the objective in (1).
We now focus on strategies that use only reward poisoning. If the post-attack mean reward  $\widetilde{R}$  is unbounded and the attacker can arbitrarily manipulate the rewards, there always exists an efficient and successful poisoning attack strategy. For example, the attacker can change the rewards of non-target actions to  $-H$ . However, such attacks can be easily detected, as the boundary of post-attack mean reward is distinct from the boundary of pre-attack mean reward. The following theorem shows that if the post-attack mean reward has the same boundary conditions as the pre-attack mean reward, the power of reward poisoning only attack is limited.
Theorem 2 If we limit the post-attack mean reward  $\tilde{R}$  to have the same boundary condition as that of the pre-attack mean reward  $R$ , i.e.  $\widetilde{R} \in [0,1]$ , there exists a MG  $(S, \{\mathcal{A}_i\}_{i=1}^m, H, P, \{R_i\}_{i=1}^m)$  and a target policy  $\pi^\dagger$  such that no reward poisoning Markov attack strategy alone can efficiently and successfully attack MARL agents by achieving the objective in (1).
The proofs of Theorem 1 and Theorem 2 are provided in Appendix F. The main idea of the proofs is as follows. In successful poisoning attacks, the attack loss scales as  $o(T)$  so that the agents will follow the target policy  $\pi^{\dagger}$  in at least  $T - o(T)$  times. To efficiently attack the MARL agents, the attacker should avoid to attack when the agents follow the target policy. Otherwise, the poisoning attack cost will grow linearly with  $T$ . The proofs of Theorem 1 and Theorem 2 proceed by constructing an MG and a target policy  $\pi^{\dagger}$  where the expected rewards under  $\pi^{\dagger}$  is always the worst for some agents if the attacker avoids to attack when the agents follow the target policy.
3.2 White-box action poisoning attacks
Even though Section 3.1 shows that there exists MG and target policy such that the action poisoning only attacks cannot be efficiently successful, here we show that it can be efficient and successful for a class of target policies. The following condition characterizes such class of target policies.
Condition 1: For the underlying environment  $MG\left(S, \{\mathcal{A}_i\}_{i=1}^m, H, P, \{R_i\}_{i=1}^m\right)$ , the attacker's target policy  $\pi^\dagger$  satisfies that for any state  $s$  and any step  $h$ , there exists an action  $\mathbf{a}$  such that  $V_{i,h}^{\pi^\dagger}(s) > Q_{i,h}^{\pi^\dagger}(s, \mathbf{a})$ , for any agent  $i$ .
Under Condition 1, we can find a worse policy  $\pi^{-}$  by
$$
\pi_ {h} ^ {-} (s) = \underset {\boldsymbol {a} \in \mathcal {A}} {\arg \max } \underset {i \in [ m ]} {\min } \left(V _ {i, h} ^ {\pi^ {\dagger}} (s) - Q _ {i, h} ^ {\pi^ {\dagger}} (s, \boldsymbol {a})\right) s. t. \forall i \in [ m ], V _ {i, h} ^ {\pi^ {\dagger}} (s) > Q _ {i, h} ^ {\pi^ {\dagger}} (s, \boldsymbol {a}). \tag {2}
$$
Under this condition, we now introduce an effective white-box action attack strategies:  $d$ -portion attack. Specifically, at the step  $h$  and state  $s$ , if all agents pick the target action, i.e.,  $\pmb{a} = \pi_h^\dagger(s)$ , the attacker does not attack, i.e.,  $\widetilde{\pmb{a}} = \pmb{a} = \pi_h^\dagger(s)$ . If some agents pick a non-target action, i.e.,  $\pmb{a} \neq \pi_h^\dagger(s)$ , the  $d$ -portion attack sets  $\widetilde{\pmb{a}}$  as
$$
\widetilde {\boldsymbol {a}} = \left\{ \begin{array}{l} \pi_ {h} ^ {\dagger} (s), \text {w i t h p r o b a b i l i t y} d _ {h} (s, \boldsymbol {a}) / m \\ \pi_ {h} ^ {-} (s), \text {w i t h p r o b a b i l i t y} 1 - d _ {h} (s, \boldsymbol {a}) / m, \end{array} \right. \tag {3}
$$
where  $d_h(s, \mathbf{a}) = m / 2 + \sum_{i=1}^{m} \mathbb{1}(a_i = \pi_{i,h}^\dagger(s)) / 2$ .
Theorem 3 If the attacker follows the  $d$ -portion attack strategy on the MG agents, the best response of each agent  $i$  towards the target policy  $\pi_{-i}^{\dagger}$  is  $\pi_i^\dagger$ . The target policy  $\pi^{\dagger}$  is an  $\{NE, CE, CCE\}$  from any agent's point of view. If every state  $s \in S$  is reachable at every step  $h \in [H]$  under the target policy,  $\pi^{\dagger}$  is the unique  $\{NE, CE, CCE\}$ .
The detailed proof can be found in Appendix G.1. Theorem 3 shows that the target policy  $\pi^{\dagger}$  is the unique  $\{\mathrm{NE},\mathrm{CE},\mathrm{CCE}\}$  under the  $d$  -portion attack. Thus, if the agents follow an MARL algorithm that is able to learn an  $\epsilon$  -approximate  $\{\mathrm{NE},\mathrm{CE},\mathrm{CCE}\}$ , the agents will learn a policy approximate to the target policy. We now discuss the high-level idea why the  $d$  -portion attack works. Under Condition 1,  $\pi^{-}$  is worse than the target policy  $\pi^{\dagger}$  at the step  $H$  from every agent's point of view. Thus, under the  $d$  portion attack, the target action strictly dominates any other action at the step  $H$ , and  $\pi^{\dagger}$  is the unique  $\{\mathrm{NE},\mathrm{CE},\mathrm{CCE}\}$  at the step  $H$ . From induction on  $h = H,H - 1,\dots ,1$ , we can further prove that the  $\pi^{\dagger}$  is the unique  $\{\mathrm{NE},\mathrm{CE},\mathrm{CCE}\}$  at any step  $h$ . We define  $\Delta_{i,h}^{+ - }(s) = Q_{i,h}^{\pi^{\dagger}}(s,\pi_h^{\dagger}(s)) - Q_{i,h}^{\pi^{\dagger}}(s,\pi_h^{-}(s))$  and the minimum gap  $\Delta_{min} = \min_{h\in [H],s\in S,i\in [m]} = \Delta_{i,h}^{+ - }(s)$ . In addition, any other distinct policy is not an  $\epsilon$  -approximate CCE with different gap  $\epsilon < \Delta_{min} / 2$ . We can derive upper bounds of the attack loss and the attack cost when attacking some special MARL algorithms.
Theorem 4 If the best-in-hindsight regret  $\operatorname{Reg}(K, H)$  of each agent's algorithm is bounded by a sub-linear bound  $\mathcal{R}(T)$  for any  $MG$  in the absence of attack, and  $\min_{s \in S, i \in [m]} \Delta_{i,h}^{\dagger -}(s) \geq \sum_{h' = h + 1}^{H} \max_{s \in S, i \in [m]} \Delta_{i,h'}^{\dagger -}(s)$  holds for any  $h \in [H]$ , then  $d$ -portion attack will force the agents to follow the target policy with the attack loss and the attack cost bounded by
$$
\mathbb {E} [ \operatorname {L o s s 1} (K, H) ] \leq 2 m ^ {2} \mathcal {R} (T) / \Delta_ {\min }, \mathbb {E} [ \operatorname {C o s t} (K, H) ] \leq 2 m ^ {3} \mathcal {R} (T) / \Delta_ {\min }. \tag {4}
$$
3.3 White-box reward poisoning attacks
As stated in Theorem 2, the reward poisoning only attacks may fail, if we limit the post-attack mean reward  $\widetilde{R}$  to satisfy the same boundary conditions as those of the pre-attack mean reward  $R$ , i.e.  $\widetilde{R} \in [0,1]$ . However, similar to the case with action poisoning only attacks, the reward poisoning only attacks can be efficiently successful for a class of target policies. The following condition specifies such class of target policies.
Condition 2: For the underlying environment  $MG(\mathcal{S},\{\mathcal{A}_i\}_{i = 1}^m,H,P,\{R_i\}_{i = 1}^m)$ , there exists constant  $\eta >0$  such that for any state  $s$ , any step  $h$ , and any agent  $i$ ,  $(R_{i,h}(s,\pi^{\dagger}(s)) - \eta) / (H - h)\geq \Delta_R > 0$  where  $\Delta_R = [\max_{s\times a\times h'}R_{i,h'}(s,a) - \min_{s\times a\times h'}R_{i,h'}(s,a)]$ .
We now introduce an effective white-box reward attack strategies:  $\eta$ -gap attack. Specifically, at the step  $h$  and state  $s$ , if agents all pick the target action, i.e.,  $\pmb{a} = \pi_h^\dagger(s)$ , the attacker does not attack, i.e.,  $\widetilde{r}_{i,h} = r_{i,h}$  for each agent  $i$ . If agent  $i$  picks a non-target action, i.e.,  $\pmb{a} \neq \pi_h^\dagger(s)$ , the  $\eta$ -gap attack sets  $\widetilde{r}_{i,h} = R_{i,h}(s, \pi^\dagger(s)) - (\eta + (H - h)\Delta_R)\mathbb{1}(a_i \neq \pi_{i,h}^\dagger(s))$  for each agent  $i$ . From Condition 2, we have  $\widetilde{r}_{i,h} \in [0,1]$ .
Theorem 5 If the attacker follows the  $\eta$ -gap attack strategy on the MG agents, the best response of each agent  $i$  towards any policy  $\pi_{-i}$  is  $\pi_i^\dagger$ . The target policy  $\pi^\dagger$  is the  $\{NE, CE, CCE\}$  from any agent's point of view. If every state  $s \in S$  is reachable at every step  $h \in [H]$  under the target policy,  $\pi^\dagger$  is the unique  $\{NE, CE, CCE\}$ .
The detailed proof can be found in Appendix H.1. Theorem 5 shows that the target policy  $\pi^{\dagger}$  is the unique  $\{\mathrm{NE},\mathrm{CE},\mathrm{CCE}\}$  under the  $\eta$ -gap attack. Thus, if the agents follow an MARL algorithm that is able to learn an  $\epsilon$ -approximate  $\{\mathrm{NE},\mathrm{CE},\mathrm{CCE}\}$ , the agents will learn a policy approximate to the target policy. Here, we discuss the high-level idea why the  $\eta$ -gap attack works.  $\Delta_R$  is the difference between the upper bound and the lower bound of the mean rewards. Condition 2 implies that each action is close to other actions from every agent's point of view. Although we limit the post-attack mean reward  $\widetilde{R}$  in [0, 1], the target policy can still appear to be optimal by making small changes to the rewards. Under Condition 2 and the  $\eta$ -gap attacks, the target actions strictly dominate any other non-target actions by at least  $\eta$  and any other distinct policy is not an  $\epsilon$ -approximate CCE with different gap  $\epsilon < \eta$ . Thus,  $\pi^{\dagger}$  becomes the unique  $\{\mathrm{NE},\mathrm{CE},\mathrm{CCE}\}$ . In addition, we can derive upper bounds of the attack loss and the attack cost when attacking MARL algorithms with sub-linear best-in-hindsight regret.
Theorem 6 If the best-in-hindsight regret  $\operatorname{Reg}(K, H)$  of each agent's algorithm is bounded by a sub-linear bound  $\mathcal{R}(T)$  for any  $MG$  in the absence of attack, then  $\eta$ -gap attack will force the agents to follow the target policy with the attack loss and the attack cost bounded by
$$
\mathbb {E} [ \operatorname {L o s s 1} (k, H) ] \leq m \mathcal {R} (T) / \eta , \mathbb {E} [ \operatorname {C o s t} (K, H) ] \leq m ^ {2} \mathcal {R} (T) / \eta . \tag {5}
$$
We note that proposed sufficient conditions (namely Condition 1 and Condition 2), under which the action poisoning only attacks or the reward poisoning only attacks can be efficient and successful, may be strict. They may not always hold in practice. This motivates us to investigate mixed attack strategy to be discussed in the sequel.
4 Gray-box attack strategy and analysis
In the gray-box attack setting, the attacker has no prior information about the underlying environment and the agents' algorithm, and it only observes samples generated when the agents interact with the environment. However, the attacker is given the target policy  $\pi^{\dagger}$ . Since the  $\eta$ -gap reward attack strategy and  $d$ -portion action attack strategy described in Section 3 for the white-box setting rely on the information of the underlying environment, these two attack strategies are not applicable in the gray-box setting. In addition, without the information of the underlying environment, the attacker cannot check whether the action poisoning attack alone or the reward poisoning attack alone can be efficiently successful. Building on insights obtained from the white-box attack strategies, we develop a mixed attack strategy for MG in the gray-box attack setting.
In the proposed mixed attack strategy, at the step  $h$  and state  $s$ , if agent  $i$  picks the target action, i.e.,  $a_{i,h} = \pi_{i,h}^{\dagger}(s)$ , the attacker does not override the action and the reward, i.e.  $\widetilde{a}_{i,h} = a_{i,h}$  and  $\widetilde{r}_{i,h} = r_{i,h}$ . If agent  $i$  picks a non-target action, i.e.,  $a_{i,h} \neq \pi_{i,h}^{\dagger}(s)$ , the attacker overrides its action  $\widetilde{a}_{i,h} = \pi_{i,h}^{\dagger}(s)$  and then overrides the reward  $\widetilde{r}_{i,h} = 0$ .
Theorem 7 If the attacker follows the mixed attack strategy the best response of each agent  $i$  towards any product policy  $\pi_{-i}$  is  $\pi_i^\dagger$ . The optimal policy  $\pi^\dagger$  is the unique  $\{NE, CE, CCE\}$ .
The detailed proof can be found in Appendix I.1. Here, we discuss the high-level idea why the mixed attack works. Under the mixed attacks, the state transitions are the same over the different actions and the reward of the non-target actions is worse than the target action. Thus, in the postattack environment, the target policy is better than any other policy from any agent's point of view, and any other distinct policy is not an  $\epsilon$ -approximate CCE with different gap  $\epsilon < R_{min}$ , where  $R_{min} = \min_{h \in [H]} \min_{s \in S} \min_{i \in [m]} R_{i,h}(s, \pi_h^\dagger(s))$ . Thus,  $\pi^\dagger$  is the unique {NE, CE, CCE}. In addition, we can derive upper bounds of the attack loss and the attack cost when attacking some special MARL algorithms.
Theorem 8 If the best-in-hindsight regret  $\operatorname{Reg}(K, H)$  of each agent's algorithm is bounded by a sub-linear bound  $\mathcal{R}(T)$  for any  $MG$  in the absence of attacks, then the mixed attacks will force the agents to follow the target policy  $\pi^{\dagger}$  with the attack loss and the attack cost bounded by
$$
\mathbb {E} [ \operatorname {L o s s 1} (K, H) ] \leq m \mathcal {R} (T) / R _ {\min }, \mathbb {E} [ \operatorname {C o s t} (K, H) ] \leq 2 m \mathcal {R} (T) / R _ {\min }. \tag {6}
$$
5 Black-box attack strategy and analysis
In the black-box attack setting, the attacker has no prior information about the underlying environment and the agents' algorithm, and it only observes the samples generated when the agents interact with the environment. The attacker aims to maximize the cumulative rewards under some specific reward functions  $R_{\dagger}$  chosen by the attacker. But unlike in the gray-box case, the corresponding target policy  $\pi^{\dagger}$  is also unknown for the attacker. After each time step, the attacker will receive the attacker reward  $r_{\dagger}$ . Since the optimal (target) policy that maximizes the attacker's reward is unknown, the attacker needs to explore the environment to obtain the optimal policy. As the mixed attack strategy described in Section 4 for the gray-box setting relies on the knowledge of the target policy, it is not applicable in the black-box setting.
However, by collecting observations and evaluating the attacker's reward function and transition probabilities of the underlying environment, the attacker can perform an approximate mixed attack strategy. In particular, we propose an approximate mixed attack strategy that has two phases: the exploration phase and the attack phase. In the exploration phase, the attacker explores the environment to identify an approximate optimal policy, while in the attack phase, the attacker performs the mixed attack strategy and forces the agents to learn the approximate optimal policy. The total attack cost (loss) will be the sum of attack cost (loss) of these two phases.
In the exploration phase, the approximate mixed attack strategy uses an optimal-policy identification algorithm, which is summarized in Algorithm 1, listed in Appendix A. It will return an approximate
optimal policy  $\pi^{\dagger}$ . Note that  $\pi^k$  denotes the product policy deployed by the agents for each episode  $k$ .  $\overline{V}$  is the upper bound of  $V^{\pi^{*}}$  and  $\underline{V}$  is the lower bound of  $V^{\pi^k}$ . By minimizing  $\overline{V} -\underline{V}$ , Algorithm 1 finds an approximate optimal policy  $\pi^{\dagger}$ . Here, we assume that the reward on the approximate optimal policy  $\pi^{\dagger}$  is positive, i.e.  $R_{min} = \min_{h\in [H]}\min_{s\in S}\min_{i\in [m]}R_{i,h}(s,\pi_h^\dagger (s)) > 0$ . In the exploration phase, the attacker will override both the agents' actions and rewards.
After the exploration phase, the approximate mixed attack strategy performs the attack phase. The attacker will override both the agents' actions and rewards in this phase. At the step  $h$  and state  $s$ , if agent  $i$  picks the action  $\pi_{i,h}^{\dagger}(s)$ , the attacker does not override actions and rewards, i.e.  $\widetilde{a}_{i,h} = a_{i,h}$  and  $\widetilde{r}_{i,h} = r_{i,h}$ . If agent  $i$  picks action  $a_{i,h} \neq \pi_{i,h}^{\dagger}(s)$ , the attacker overrides the action  $\widetilde{a}_{i,h} = a_{i,h}$  and then overrides the reward  $\widetilde{r}_{i,h} = 0$ . The attack strategy in the attack phase is same with the mixed attack strategy. From Theorem 7, in the attack phase, the best response of each agent  $i$  towards product policy  $\pi_{-i}^{\dagger}$  is  $\pi_i^\dagger$  and  $\pi^{\dagger}$  is the unique NE. Here, we discuss the high-level idea why the approximate mixed attack works. The attacker finds an approximate optimal policy  $\pi^{\dagger}$  by Algorithm 1. If  $\pi^{*}$  is close to  $\pi^{\dagger}$  and the exploration phase is sub-linear time dependent, the performance of the approximate mixed attack strategy will be close to the mixed attack strategy. We build a confidence bound to show the value function difference between  $\pi^{*}$  and  $\pi^{\dagger}$  in the following lemma.
Lemma 1 If the attacker follows the Algorithm 1 in Appendix A on the agents, for any  $\delta \in (0,1)$  with probability at least  $1 - 5\delta$ , the following bound holds:
$$
\mathbb {E} _ {s _ {1} \sim P _ {0} (\cdot)} \left[ V _ {\dagger , 1} ^ {\pi^ {*}} (s _ {1}) - V _ {\dagger , 1} ^ {\pi^ {\dagger}} (s _ {1}) \right] \leq 2 H ^ {2} S \sqrt {2 A \log (2 S A H \tau / \delta) / \tau}. \tag {7}
$$
We now investigate the impact of the approximate mixed attack strategy attack on V-learning [Jin et al., 2021], a simple, efficient, decentralized algorithm for MARL. The reader's convenience, we list V-learning in Appendix J.2.
Theorem 9 Suppose ADV_BANDIT_UPDATE of  $V$ -learning follows Algorithm 3 in Appendix J.2 and it chooses hyper-parameter  $w_{t} = \alpha_{t}\left(\prod_{i=2}^{t}(1 - \alpha_{i})\right)^{-1}$ ,  $\gamma_{t} = \sqrt{\frac{H \log B}{B t}}$  and  $\alpha_{t} = \frac{H + 1}{H + t}$ . For given  $K$  and any  $\delta \in (0,1)$ , let  $\iota = \log(m H S A K / \delta)$ . The attack loss and the attack cost of the approximate mixed attack strategy during these  $K$  episodes are bounded by
$$
\mathbb {E} [ \text {L o s s 2} (K, H) ] \leq H \tau + \frac {4 0}{R _ {\min }} m \sqrt {H ^ {9} A S K \iota} + 2 H ^ {2} S K \sqrt {2 A \iota / \tau}, \tag {8}
$$
$$
\mathbb {E} \left[ \operatorname {C o s t} (K, H) \right] \leq 2 m H \tau + \frac {8 0}{R _ {\min}} \sqrt {H ^ {5} A S K \iota}.
$$
Let  $\hat{\pi}$  be the executing output policy of  $V$ -learning, the attack loss of the executing output policy  $\hat{\pi}$  is upper bounded by
$$
V _ {\dagger , 1} ^ {\pi^ {*}} (s _ {1}) - V _ {\dagger , 1} ^ {\hat {\pi}} (s _ {1}) \leq \frac {2 0 m S}{R _ {m i n}} \sqrt {\frac {H ^ {7} A \iota}{K}} + \frac {2 \tau m H ^ {2} S}{K} + 2 H ^ {2} S \sqrt {2 A \iota / \tau}. \tag {9}
$$
If we choose the stopping time of the exploration phase  $\tau = K^{2/3}$ , the attack loss and the attack cost of the approximate mixed attack strategy during these  $K$  episodes are bounded by  $\mathcal{O}(K^{2/3})$  and  $V_{\dagger,1}^{\pi^*}(s_1) - V_{\dagger,1}^{\hat{\pi}}(s_1) \leq \mathcal{O}(K^{-1/3})$ .
In this section, we empirically compare the performance of the action poisoning only attack strategy ( $d$ -portion attack), the reward poisoning only attack strategy ( $\eta$ -gap attack) and the mixed attack strategy.
We consider a simple case of Markov game where  $m = 2$ ,  $H = 2$  and  $|\mathcal{S}| = 3$ . This Markov game is the example in Appendix F.2. The initial state is  $s_1$  at  $h = 1$  and the transition probabilities are:
$$
\begin{array}{l}P \left(s _ {2} \mid s _ {1}, a\right) = 0. 9, P \left(s _ {3} \mid s _ {1}, a\right) = 0. 1, \text {i f} a = (\text {D e f e c t , D e f e c t}),\\P \left( \right.- \mid - \mid - \mid - \mid - \mid - \mid - \mid - \mid - \mid - \mid - \mid - \mid - \mid - \mid - \mid - \mid - \mid - \mid - \mid - \mid - \mid - \mid - \mid - \mid - \mid - \mid - \mid - \mid - \mid - \mid - \mid - \mid - \mid - \mid - 0. 0, i f a <   / (\text {D e f e c t , D e f e c t})\end{array}\tag {10}
$$
$$
P \left(s _ {2} \mid s _ {1}, a\right) = 0. 1, P \left(s _ {3} \mid s _ {1}, a\right) = 0. 9, \text {i f} a \neq (\text {D e f e c t , D e f e c t}).
$$
Table 2: Reward matrices
state s1	Cooperate	Defect	state s2	Cooperate	Defect	state s3	Cooperate	Defect
Cooperate	(1,1)	(0.5,0.5)	Cooperate	(1,1)	(0.5,0.5)	Cooperate	(1,1)	(0.5,0.5)
Defect	(0.5,0.5)	(0.2,0.2)	Defect	(0.5,0.5)	(0.1,0.1)	Defect	(0.5,0.5)	(0.9,0.9)
The reward functions are expressed in the following Table 2.
We set the total number of episodes  $K = 10^{7}$ . We set two different target policies. For the first target policy, no action/reward poisoning Markov attack strategy alone can efficiently and successfully attack MARL agents. For the second target policy, the  $d$ -portion attack and the  $\eta$ -gap attack can efficiently and successfully attack MARL agents.
Case 1. The target policy is that the two agents both choose to defect at any state. As stated in Section 3 and Appendix 3.1, the Condition 1 and Condition 2 do not hold for this Markov game and target policy, and no action/reward poisoning Markov attack strategy alone can efficiently and successfully attack MARL agents.
In Figure 1, we illustrate the mixed attack strategy, the  $d$ -portion attack strategy and the  $\eta$ -gap attack strategy on V-learning agents for the proposed MG. The  $x$ -axis represents the episode  $k$  in the MG. The  $y$ -axis represents the cumulative attack cost and attack loss that change over time steps. The results show that, the attack cost and attack loss of the mixed attack strategy sublinearly scale as  $T$ , but the attack cost and attack loss of the  $d$ -portion attack strategy and the  $\eta$ -gap attack strategy linearly scale as  $T$ , which is consistent with our analysis.
Figure 1: The attack loss (cost) on case 1.
Figure 2: The attack loss (cost) on case 2.
Case 2. The target policy is that the two agents choose to cooperate at state  $s_1$  and  $s_2$  but to defect at state  $s_3$ . As stated in Section 3 and Appendix 3.1, the Condition 1 and Condition 2 hold for this Markov game and target policy. Thus, the  $d$ -portion attack strategy and the  $\eta$ -gap attack strategy can efficiently and successfully attack MARL agents.
In Figure 2, we illustrate the mixed attack strategy, the  $d$ -portion attack strategy and the  $\eta$ -gap attack strategy on V-learning agents for the proposed MG. The results show that, the attack cost and attack loss of all three strategies sublinearly scale as  $T$ , which is consistent with our analysis. Additional numerical results that compare the performance of the mixed attack strategy and the approximate mixed attack strategy are provided in Appendix B.