SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this paper, we have introduced an adversarial attack model on MARL. We have discussed the attack problem in three different settings: the white-box, the gray-box and the black-box settings. We have shown that the power of action poisoning only attacks and reward poisoning only attacks is limited. Even in the white-box setting, there exist some MGs, under which no action poisoning only attack strategy or reward poisoning only attack strategy can be efficient and successful. We have then characterized conditions when action poisoning only attacks or only reward poisoning only attacks can efficiently work. We have further introduced the mixed attack strategy in the gray-box setting that can efficiently attack any sub-linear-regret MARL agents. Finally, we have proposed the approximate mixed attack strategy in the black-box setting and shown its effectiveness on V-learning. This paper raises awareness of the trustworthiness of online multi-agent reinforcement learning. In the future, we will investigate the defense strategy to mitigate the effects of this attack.
This work was supported by the National Science Foundation under Grant CCF-22-32907.
Xuezhou Zhang, Yuzhe Ma, Adish Singla, and Xiaojin Zhu. Adaptive reward-poisoning attacks against reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pages 11225-11234, 2020b.
Xuezhou Zhang, Yiding Chen, Xiaojin Zhu, and Wen Sun. Robust policy gradient against strong data corruption. In International Conference on Machine Learning, pages 12391-12401. PMLR, 2021b.
A The exploration phase of the approximate mixed attack strategy
The exploration phase of the approximate mixed attack strategy uses an optimal-policy identification algorithm, which is summarized in Algorithm 1. It will return an approximate optimal policy  $\pi^{\dagger}$ , which is an approximate optimal policy.
Algorithm 1: Exploration phase for Markov games
Input: Stopping time  $\tau$ . Set  $B(N) = (H\sqrt{S} + 1)\sqrt{\log(2AH\tau/\delta)/(2N)}$ .  
1: Initialize  $\overline{Q}_{\dagger,h}(s,a) = \overline{V}_{\dagger,h}(s,a) = H, Q_{\dagger,h}(s,a) = \underline{V}_{\dagger,h}(s,a) = 0$ $\overline{V}_{\dagger,H+1} = \underline{V}_{\dagger,H+1} = 0, \Delta = \infty, N_0(s) = N_h(s,a) = N_h(s,a,s') = 0$  and  $\hat{R}_{\dagger,h}(s,a) = 0$  for any  $(s,s',a,i,h)$ .  
2: for episode  $k = 1,\ldots,\tau$  do  
3: for step  $h = H,\ldots,1$  do  
4: for each  $(s,a) \in S \times A$  with  $N_h(s,a) > 0$  do  
5: Update  $\overline{Q}_{\dagger,h}(s,a) = \min\{\hat{R}_{\dagger,h} + \hat{\mathbb{P}}_h\overline{V}_{\dagger,h+1}(s,a) + B(N_h(s,a)), H\}$  and  $Q_{\dagger,h}(s,a) = \max\{\hat{R}_{\dagger,h} + \hat{\mathbb{P}}_h\overline{V}_{\dagger,h+1}(s,a) - B(N_h(s,a)), 0\}$ .  
6: end for  
7: for each  $s \in S$  with  $N_h(s,a) > 0$  do  
8: Update  $\pi_h(s) = \max_{a \in A} \overline{Q}_{\dagger,h}(s,a)$ .  
9: Update  $\overline{V}_{\dagger,h}(s,a) = \overline{Q}_{\dagger,h}(s,\pi_h(s))$  and  $\underline{V}_{\dagger,h}(s,a) = Q_{\dagger,h}(s,\pi_h(s))$ .  
10: end for  
11: end for  
12: if  $\mathbb{E}_{s \sim \hat{\mathbb{P}}_0(\cdot)}(\overline{V}_{\dagger,1}(s) - \underline{V}_{\dagger,1}(s)) + H\sqrt{\frac{S\log(2\tau/\delta)}{2k}} \leq \Delta$  then  
13:  $\Delta = \mathbb{E}_{s \sim \hat{\mathbb{P}}_0(\cdot)}(\overline{V}_{\dagger,1}(s) - \underline{V}_{\dagger,1}(s)) + H\sqrt{\frac{S\log(2\tau/\delta)}{2k}}$  and  $\pi^{\dagger} = \pi$ .  
14: end if  
15: for step  $h = 1,\ldots,H$  do  
16: Attacker overrides each agent's action by changing  $a_i,h$  to  $\widetilde{a}_i,h$ , where  $\widetilde{a}_h = \pi_h(sh)$ .  
17: The environment returns the reward  $r_{i,h}$  and the next state  $s_{h+1}$  according to action  $\widetilde{a}_h$ . The attacker receives its reward  $r_{i,h}$ .  
18: Attacker overrides each agent's reward by changing  $r_i,h$  to  $\widetilde{r}_i,h$   
19: Add 1 to  $N_h(sh,\widetilde{a}_h)$  and  $N_h(sh,\widetilde{a}_h,s_{h+1})$ .  $\hat{\mathbb{P}}_h(|sh,\widetilde{a}_h) = N_h(sh,\widetilde{a}_h,\cdot)/N_h(sh,\widetilde{a}_h)$   
20: Update  $\hat{R}_{\dagger,h}(sh,\widetilde{a}_h) = \hat{R}_{\dagger,h}(sh,\widetilde{a}_h) + (r_{i,t} - \hat{R}_{\dagger,h}(sh,\widetilde{a}_h)/N_h(sh,\widetilde{a}_h)$ .  
21: end for  
22: Update  $N_0(s_1) = N_0(s_1) + 1$  and  $\hat{\mathbb{P}}_0(\cdot) = N_0(\cdot)/k$ .  
23: end for  
24: Return  $\pi^{\dagger}$ .
B Additional numerical results
In this section, we empirically compare the performance of the mixed attack strategy and the approximate mixed attack strategy. We consider a multi-agent system with three recycling robots. In this scenario, a mobile robot with a rechargeable battery and a solar battery collects empty soda cans in a city. The number of agents is 3, i.e.  $m = 3$ . Each robot has two different energy levels, high energy level and low energy level, resulting in 8 states in total, i.e.  $S = 8$ .
Each robot can choose a conservative action or an aggressive action, so  $A_{i} = 2$  and  $A = 8$ . At the high energy level, the conservative action is to wait in some place to save energy and then the mean reward is 0.4. At the high energy level, the aggressive action is to search for cans. All the robots that choose to search will get a basic reward 0.2 and equally share an additionally mean reward 0.9. For example, if all robots choose to search at a step, the mean reward of each robot is 0.5. At the low energy level, the conservative action is to return to change the battery and find the cans on the way. In this state and action, the robot only gets a low mean reward 0.2. At the low energy level, the
conservative action is to wait in some place to save energy and then the mean reward is 0.3. We use Gaussian distribution to randomize the reward signal.
We set the total number of steps  $H = 6$ . At the step  $h \leq 3$ , it is the daytime and the robot who chooses to search will change to the low energy level with low probability 0.3. At the step  $h \geq 4$ , it is the night and the robot who chooses to search will change to the low energy level with high probability 0.7. The energy level transition probabilities are stated in Figure 3 and Figure 4. 'H' represents the high energy level. 'L' represents the low energy level. 'C' represents the conservative action. 'A' represents the aggressive action.
Figure 3: Energy level transitions at  $h \leq 3$ .
Figure 4: Energy level transitions at  $h \geq 4$ .
We consider two different attack goals: (1) maximize the first robot's rewards; (2) minimize the second robot's and the third robot's rewards. For the gray box case, we provide the target policy that maximizes the first robot's rewards or minimizes the second robot's and the third robot's rewards. For the black box case, we set  $R_{\dagger,h} = R_{1,h}$  to maximize the first robot's rewards and set  $R_{\dagger,h} = 1 - R_{2,h} / 2 - R_{3,h} / 2$  to minimize the second robot's and the third robot's rewards.
Figure 5: The cumulative attack loss and cost of the mixed attack and the approximate mixed attack.
We set the total number of episodes  $K = 10^{7}$ . In Figure 5, we illustrate the mixed attack strategy and approximate-mixed attack strategy on V-learning agents for the proposed MG. The  $x$ -axis represents the episode  $k$  in the MG. The  $y$ -axis represents the cumulative attack cost and attack loss that change over time steps. The results show that, the attack cost and attack loss of the mixed attack strategy and approximate-mixed attack strategy sublinearly scale as  $T$ , which is consistent with our analysis. Furthermore, Figure 5 shows that the performance of the approximate-mixed attack strategy nearly match that of the mixed attack strategy. This illustrates that the proposed approximate-mixed attack strategy is very effective in the black-box scenario.
C Related works
Due to the page limit of the main paper, we do not provide a comprehensive comparison with prior research on adversarial attacks. We add the following discussion to Appendix.
Among the existing works on attacks in single-agent RL, the most related paper is [Rangi et al., 2022], which studies the limitations of reward only manipulation or action only manipulation in single-agent RL and proposed an attack strategy combining reward and action manipulation.
There are multiple differences between our work and [Rangi et al., 2022]. First, the MARL is modeled as a Markov game, but the single-agent RL is modeled as a MDP. In Markov game, each agent's
action will impact other agents' rewards. Second, the learning object of single-agent RL and MARL is different. The single-agent RL algorithms learn the optimal policy, but MARL algorithms learn the equilibrium. Since the attacks on one agent will impact all other agents and the equilibrium is considered as the agents' learning object, we have to develop techniques to carefully analyze the impact of attacks and the bound of the attack cost.
Here, we discuss the related work on the adversarial attacks on single MARL. [Ma et al., 2019] studies reward poisoning attack against batch RL in which the attacker is able to gather and modify the collected batch data. [Rakhsha et al., 2020] proposes a white-box environment poisoning model in which the attacker could manipulate the original MDP to a poisoned MDP. [Behzadan and Munir, 2017, Zhang et al., 2020b, Rangi et al., 2022] study online white-box reward poisoning attacks in which the attacker could manipulate the reward signal before the agent receives it. [Sun et al., 2021] proposes a practical black-box poisoning algorithm called VA2C-P. Their empirical results show that VA2C-P works for deep policy gradient RL agents without any prior knowledge of the environment. [Rakhsha et al., 2021] develops a black-box reward poisoning attack strategy called U2, that can provably attack any efficient RL algorithms. [Xu et al., 2021] investigates training-time attacks on RL agents and the introduced attacker can manipulate the environment.
Here, we discuss the related work on the adversarial attacks on MARL. [Ma et al., 2022] considers a game redesign problem where the designer knows the full information of the game and can redesign the reward functions. The proposed redesign methods can incentivize players to take a specific target action profile frequently with a small cumulative design cost. Ma's work considered the norm-form game but we considered the Markov game. The norm-form game is a simple case of the Markov game with horizon  $H = 1$ . [Gleave et al., 2020, Guo et al., 2021] study the poisoning attack on multi agent reinforcement learners, assuming that the attacker controls one of the learners. In our work, the attacker is not one of the learners, but an external unit out of the original Markov game. The attacker can poisoning the reward/action of all learners at the same time so that can fool the learners to learn a specific policy. [Wu et al., 2022] studies the reward poisoning attack on offline multi-agent reinforcement learners. The attacker can poisoning the reward of the agents. We considered the online MARL. In offline MARL, the attacker can estimate the underline Markov game from the offline datasets. In online MARL, the attacker may not have the knowledge (reward/transition functions) of the Markov game. [Wang et al., 2021] studies the backdoor attack in two-player competitive RL systems. The trigger is the action of another agent in the environment. They propose a unified method to design fast-failing agents which will fast fail when trigger occurred. [Liu et al., 2022] studies the controllable attack by constraining the state distribution shift caused by the adversarial policy and offering a more controllable attack scheme. [Chen et al., 2023] considers a situation that fraction of agents are adversarial and can report arbitrary fake information. They design two Byzantine-robust distributed value iteration algorithms that can identify a near-optimal policy with near-optimal sample complexity. [Mohammadi et al., 2023] studies targeted poisoning attacks in a two-agent setting where an attacker implicitly poisons the effective environment of one of the agents by modifying the policy of its peer.
Due to the page limit of the main paper, we put the discussions regard the attack detection, the computational cost, the scalability of the attack strategies in Appendix.
Attack detection We did not consider the attack detection in our problem, but the attack detection problem is also important. In this paper, we assumed that the agents do not know the existence of the attacker. Under this assumption, if the agents have no prior information of the MG, the proposed white and gray box attack is hard to be detected. As we consider the Markov attack strategy in this paper, the post-attack environment under the Markov attack strategy is still a Markov game. Without reference or prior information of the MG, the agents can not figure out whether the environment they observe is a post-attack environment or an attack-free environment. The proposed black attack may be detected, as the transition probabilities of the post-attack environment change over time. The goal of our paper is to understand and identify the impacts of different adversarial attacks. We hope our work can inspire follow-up work that can detect and mitigate such attacks so that RL models can be used in safety-critical applications. It is an important future direction for us to pursue.
Computational cost For the proposed black-box attack strategy (the approximation mixed attack), the computational cost is  $O(S^2 AH\tau + mKH)$ . The proposed algorithm will compute the  $Q$ -values for each visited action-state pair at every steps and every episodes in the exploration phase. The computation of  $Q$ -value costs  $O(S)$ . Thus, the total computational cost in the exploration phase is  $O(S^2 AH\tau)$ . In the attack phase, the attacker only need to change the action and the reward for each agent so that the computational cost in the attack phase is  $O(mKH)$ .
**Scalability** The gray-box attack strategies can be directly used in large-scale environments, even in some high-dimensional continuous environment. However, in the continuous space, the attacker does not change the non-target action to 0 but to  $r_{i,h} * e^{c\| a_{i,h} - a_{i,h}^{\dagger}\|}$ , in order to avoiding sparse reward. The ideas of the black-box attack strategies still work. However, the exploration phase should resort to some function approximation methods to efficiently explore an approximate target policy. Then the attack phase keeps the same as the gray-box attack.
E Notations
In this section, we introduce some notations that will be frequently used in appendixes.
The attack strategies in this paper are all Markov and only depend on the current state and actions. The post-attack reward function has the same form as the original reward function which is Markov and bounded in  $[0,1]$ . Thus, the combination of the attacker and the environment  $MG(\mathcal{S},\{\mathcal{A}_i\}_{i = 1}^m,H,P,\{R_i\}_{i = 1}^m)$  can also be considered as a new environment  $\widetilde{MG} (\mathcal{S},\{\mathcal{A}_i\}_{i = 1}^m,H,\widetilde{P},\{\widetilde{R}_i\}_{i = 1}^m)$ , and the agents interact with the new environment.  $\widetilde{R}_{i,h}:\mathcal{S}\times \mathcal{A}\to [0,1]$  represents the post-attack reward function for the  $i^{th}$  agent in the step  $h$ . The post-attack transition probabilities satisfy  $\widetilde{P}_h(s'|s,\pmb {a}) = \sum_{\pmb {a}'}\mathbb{A}_h(\pmb {a}'|s,\pmb {a})P_h(s'|s,\pmb {a}')$
We use  $\widetilde{R}_i$ ,  $\widetilde{N}_i$ ,  $\widetilde{Q}_i$  and  $\widetilde{V}_i$  to denote the mean rewards, counter,  $Q$ -values and value functions of the new post-attack environment that each agent  $i$  observes. We use  $N^k$ ,  $V^k$  and  $\pi^k$  to denote the counter, value functions, and policy maintained by the agents' algorithm at the beginning of the episode  $k$ .
For notation simplicity, we define two operators  $\mathbb{P}$  and  $\mathbb{D}$  as follows:
$$
\begin{array}{l} \mathbb {P} _ {h} [ V ] (s, \boldsymbol {a}) = \mathbb {E} _ {s ^ {\prime} \sim P _ {h} (\cdot | s, \boldsymbol {a})} [ V (s ^ {\prime}) ], \\ \mathbb {P} _ {h} [ C ^ {\prime} (s) - \mathbb {E} _ {s ^ {\prime}} [ C (s ^ {\prime}) ] \end{array} \tag {11}
$$
$$
\mathbb {D} _ {\pi} [ Q ] (s) = \mathbb {E} _ {\boldsymbol {a} \sim \pi (\cdot | s)} \left[ Q (s, \boldsymbol {a}) \right].
$$
Furthermore, we let  $\mathbb{A}$  denote the action manipulation.  $\mathbb{A} = \{\mathbb{A}_h\}_{h\in [H]}$  is a collection of actionmanipulation matrices, so that  $\mathbb{A}_h(\cdot |s,\pmb {a})$  gives the probability distribution of the post-attack action if actions  $\pmb{a}$  are taken at state  $s$  and step  $h$ . Using this notation, in the  $d$ -portion attack strategy, we have  $\mathbb{A}_h(\pi_h^\dagger (s)|s,\pmb {a}) = d_h(s,\pmb {a}) / m$ , and  $\mathbb{A}_h(\pi_h^- (s)|s,\pmb {a}) = 1 - d_h(s,\pmb {a}) / m$ .
F Proof of the insufficiency of action poisoning only attacks and reward poisoning only attacks
F.1 Proof of Theorem 1
We consider a simple case of Markov game where  $m = 2$ ,  $H = 1$  and  $|\mathcal{S}| = 1$ . The reward function can be expressed in the matrix form in Table 3.
Table 3: Reward matrix
Cooperate	Defect
Cooperate	(1, 1)	(0.5, 0.5)
Defect	(0.5, 0.5)	(0.1, 0.1)
The target policy is that the two agents both choose to defect. In this MG, the two agents' rewards are the same under any action. As the action attacks only change the agent's action, the post-attack
Table 4: Post-attack reward matrix
Cooperate	Defect
Cooperate	(r1, r1)	(r2, r2)
Defect	(r3, r3)	(r4, r4)
rewards have the same property. The post-attack reward function can be expressed in the matrix form in Table 4.
To achieve the objective in (1), we first have  $r_2 \leq r_4$  and  $r_3 \leq r_4$ , as the target policy should be an NE. Since the other distinct policy should not be an  $\epsilon$ -approximate CCE, we consider the other three pure-strategy policies and have
$$
\left\{ \begin{array}{l} r _ {1} > r _ {2} + \epsilon , \text {o r} r _ {4} > r _ {2} + \epsilon \\ r _ {1} > r _ {3} + \epsilon , \text {o r} r _ {4} > r _ {3} + \epsilon . \\ r _ {3} > r _ {1} + \epsilon , \text {o r} r _ {2} > r _ {1} + \epsilon \end{array} \right. \tag {12}
$$
Note that  $r_3 > r_1 + \epsilon$  and  $r_1 > r_3 + \epsilon$  are contradictory and  $r_2 > r_1 + \epsilon$  and  $r_1 > r_2 + \epsilon$  are contradictory. We must have  $r_4 > r_3 + \epsilon$  or  $r_4 > r_2 + \epsilon$ . As the action attacks will keep the same boundary of the rewards,  $r_3 \geq 0.1$  and  $r_2 \geq 0.1$ . Then,  $r_4 > 0.1 + \epsilon$ .
Suppose there exists an action poisoning attack strategy that can successfully attack MARL agents. We have  $\sum_{k=1}^{K}\sum_{h=1}^{H}\sum_{i=1}^{m}\mathbb{1}\left(a_{i,h}^{k}=\pi^{\dagger}(s_{i,h}^{k})\right)=T-o(T)=\Omega(T)$ , i.e. the attack loss scales on  $o(T)$ . To achieve the post-attack reward satisfy  $r_4>0.1+\epsilon$ , the attacker needs to change the target action (Defect, Defect) to other actions with probability at least  $\epsilon$ , when the agents choose the target action. Then, we have  $\sum_{k=1}^{K}\sum_{h=1}^{H}\sum_{i=1}^{m}\mathbb{E}(\mathbb{1}(\widetilde{a}_{i,h}^{k}\neq a_{i,h}^{k}))=\Omega(\epsilon T)$ . The expected attack cost is linearly dependent on  $T$ . Hence, there does not exist an action poisoning attack strategy that is both efficient and successful for this case.
F.2 Proof of Theorem 2
We consider a simple case of Markov game where  $m = 2$ ,  $H = 2$  and  $|\mathcal{S}| = 3$ . The reward functions are expressed in the following Table 5.
Table 5: Reward matrix
state s1	Cooperate	Defect
Cooperate	(1, 1)	(0.5, 0.5)
Defect	(0.5, 0.5)	(0.2, 0.2)
state s2	Cooperate	Defect
Cooperate	(1, 1)	(0.5, 0.5)
Defect	(0.5, 0.5)	(0.1, 0.1)
state s3	Cooperate	Defect
Cooperate	(1, 1)	(0.5, 0.5)
Defect	(0.5, 0.5)	(0.9, 0.9)
The initial state is  $s_1$  at  $h = 1$  and the transition probabilities are:
$$
\begin{array}{l} P \left(s _ {2} \mid s _ {1}, a\right) = 0. 9, P \left(s _ {3} \mid s _ {1}, a\right) = 0. 1, \text {i f} a = (\text {D e f e c t , D e f e c t}), \\ P \left(s _ {2} \mid s _ {1}, a\right) = 0. 1, P \left(s _ {3} \mid s _ {1}, a\right) = 0. 9, \text {i f} a \neq (\text {D e f e c t , D e f e c t}). \end{array} \tag {13}
$$
The target policy is that the two agents both choose to defect at any state. The post-attack reward function of the three states can be expressed in the matrix form in Table 6.
Table 6: Post-attack reward matrix
state s1	Cooperate	Defect
Cooperate	(r1, r2)	(r3, r4)
Defect	(r5, r6)	(r7, r8)
state s2	Cooperate	Defect
Cooperate	(-,-)	(-,-)
Defect	(-,-)	(r9, r10)
state s3	Cooperate	Defect
Cooperate	(-,-)	(-,-)
Defect	(-,-)	(r11, r12)
We limit that the post-attack mean reward  $\widetilde{R}$  has the same boundary condition with that of the pre-attack mean reward  $R$ , i.e.  $\widetilde{R} \in [0,1]$ . Then,  $0 \leq r_1, \ldots, r_{12} \leq 1$ .
Suppose there exists a reward poisoning attack strategy that can successfully attack MARL agents, we have  $\sum_{k=1}^{K} \sum_{h=1}^{H} \sum_{i=1}^{m} \mathbb{1}\left(a_{i,h}^{k} = \pi^{\dagger}(s_{i,h}^{k})\right) = T - o(T) = \Omega(T)$ , i.e. the attack loss scales on  $o(T)$ .
If  $|r_9 - 0.1| > 0.1$ ,  $|r_{10} - 0.1| > 0.1$ ,  $|r_{11} - 0.9| > 0.1$ , or  $|r_{12} - 0.9| > 0.1$ , we have the attack cost  $\sum_{k=1}^{K} \sum_{h=1}^{H} \sum_{i=1}^{m} \mathbb{E}(|\tilde{r}_{i,h}^k - r_{i,h}^k|) = \Omega(0.1 * K) = \Omega(T)$ . Thus,  $|r_9 - 0.1| \leq 0.1$ ,  $|r_{10} - 0.1| \leq 0.1$ ,  $|r_{11} - 0.9| \leq 0.1$  and  $|r_{12} - 0.9| \leq 0.1$ .
For the target policy, we have  $\widetilde{V}_{i,1}^{\pi^{\dagger}}(s_1) = r_7 + 0.9*r_9 + 0.1*r_{11}$ . For the policy  $\pi'$  with  $\pi_1'(s_1) = (\text{Cooperate}, \text{Defect}), \pi_2'(s_2) = (\text{Defect}, \text{Defect}), \pi_2'(s_3) = (\text{Defect}, \text{Defect})$ , we have  $\widetilde{V}_{i,1}^{\pi'}(s_1) = r_3 + 0.1*r_9 + 0.9*r_{11}$ .
To achieve the objective in (1), the attacker should let the target policy to be an NE. Thus, we have  $\widetilde{V}_{i,1}^{\pi^{\dagger}}(s_1) \geq \widetilde{V}_{i,1}^{\pi'}(s_1)$  and then  $r_7 + 0.9 * r_9 + 0.1r_{11} \geq r_3 + 0.1 * r_9 + 0.9 * r_{11}$ . As  $|r_9 - 0.1| \leq 0.1$  and  $|r_{11} - 0.9| \leq 0.1$ , we have  $r_7 \geq r_3 + 0.48$ . From the boundary condition, we have  $r_3 \geq 0$  and then  $r_7 \geq 0.48$ . The attack cost scales at least on  $\Omega(0.28 * T)$  for a successful reward attack strategy.
In summary, there does not exist an reward poisoning attack strategy that is both efficient and successful for this case.
G Analysis of the  $d$ -portion Attack
G.1 Proof of Theorem 3
We assume that the minimum gap exists and is positive, i.e.  $\Delta_{min} > 0$ . This positive gap provides an opportunity for efficient action poisoning attacks.
We assume that the agent does not know the attacker's manipulations and the presence of the attacker. The attacker's manipulations on actions are stationary. We can consider the combination of the attacker and the environment  $\mathrm{MG}(\mathcal{S},\{\mathcal{A}_i\}_{i = 1}^m,H,P,\{R_i\}_{i = 1}^m)$  as a new environment  $\widetilde{\mathrm{MG}} (\mathcal{S},\{\mathcal{A}_i\}_{i = 1}^m,H,\widetilde{P},\{\widetilde{R}_i\}_{i = 1}^m)$ , and the agents interact with the new environment. We define  $\widetilde{Q}_i$  and  $\widetilde{V}_i$  as the  $Q$ -values and value functions of the new environment  $\widetilde{\mathrm{MG}}$  that each agent  $i$  observes.
We first prove that  $\pi^{\dagger}$  is an NE from every agent's point of view.
Condition 1 implies that  $\pi^{\dagger}$  is not the worst policy from every agent's point of view, and there exists a policy  $\pi^{-}$  that is worse than the target policy from every agent's point of view. Denote  $\Delta_{i,h}^{\dagger -}(s) = Q_{i,h}^{\pi^{\dagger}}(s,\pi_{h}^{\dagger}(s)) - Q_{i,h}^{\pi^{\dagger}}(s,\pi_{h}^{-}(s))$ . We define the minimum gap  $\Delta_{min} = \min_{h\in [H],s\in S,i\in [m]} = \Delta_{i,h}^{\dagger -}(s)$ .
We set  $\mathbb{P}_hV_{i,h + 1}^\pi (s,\pmb {a}) = \mathbb{E}_{s'\sim P_h(\cdot |s,\pmb {a})}\left[V_{i,h + 1}^\pi (s')\right]$ . From  $d$ -portion attack strategy, we have