SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Guanlin Liu Lifeng Lai
One Shields Avenue, Davis, CA 95616
{glmnliu, lflai}@ucdavis.edu
Due to the broad range of applications of multi-agent reinforcement learning (MARL), understanding the effects of adversarial attacks against MARL model is essential for the safe applications of this model. Motivated by this, we investigate the impact of adversarial attacks on MARL. In the considered setup, there is an exogenous attacker who is able to modify the rewards before the agents receive them or manipulate the actions before the environment receives them. The attacker aims to guide each agent into a target policy or maximize the cumulative rewards under some specific reward function chosen by the attacker, while minimizing the amount of manipulation on feedback and action. We first show the limitations of the action poisoning only attacks and the reward poisoning only attacks. We then introduce a mixed attack strategy with both the action poisoning and the reward poisoning. We show that the mixed attack strategy can efficiently attack MARL agents even if the attacker has no prior information about the underlying environment and the agents' algorithms.
Recently reinforcement learning (RL), including single agent RL and multi-agent RL (MARL), has received significant research interests, partly due to its many applications in a variety of scenarios such as the autonomous driving, traffic signal control, cooperative robotics, economic policy-making, and video games [Silver et al., 2016, Brown and Sandholm, 2019, Vinyals et al., 2019, Berner et al., 2019, Shalev-Shwartz et al., 2016, OroojlooyJadid and Hajinezhad, 2019, Baker et al., 2020, Zhang et al., 2021a]. In MARL, at each state, each agent takes its own action, and these actions jointly determine the next state of the environment and the reward of each agent. The rewards may vary for different agents. In this paper, we focus on the model of Markov Games (MG) [Shapley, 1953]. In this class of problems, researchers typically consider learning objectives such as Nash equilibrium (NE), correlated equilibrium (CE) and coarse correlated equilibrium (CCE) etc. A recent line of works provide non-asymptotic guarantees for learning NE, CCE or CE under different assumptions [Sidford et al., 2020, Zhang et al., 2020a, Bai and Jin, 2020, Xie et al., 2020, Liu et al., 2021, Jin et al., 2021, Mao and Bāşar, 2022].
As RL models, including single agent RL and MARL, are being increasingly used in safety critical and security related applications, it is critical to developing trustworthy RL systems. As a first step towards this important goal, it is essential to understand the effects of adversarial attacks on RL systems. Motivated by this, there have been many recent works that investigate adversarial attacks on single agent RL under various settings [Behzadan and Munir, 2017, Huang and Zhu, 2019, Ma et al., 2019, Zhang et al., 2020b, Sun et al., 2021, Rakhsha et al., 2020, 2021].
On the other hand, except the ones that will be reviewed below, existing work on adversarial attacks on MARL is limited. In this paper, we aim to fill in this gap and systematically investigate the impact of adversarial attacks on online MARL. We consider a setting in which there is an attacker sits between the agents and the environment, and can monitor the states, the actions of the agents and the reward signals from the environment. The attacker is able to manipulate the feedback or action of the agents. The objective of the MARL learner is to learn an equilibrium. The attacker's goal is to force the agents to learn a target policy or to maximize the cumulative rewards under some specific reward function chosen by the attacker, while minimizing the amount of the manipulation on feedback and action. Our contributions are follows.
1) We propose an adversarial attack model in which the attacker aims to force the agent to learn a policy selected by the attacker (will be called target policy in the sequel) or to maximize the cumulative rewards under some specific reward function chosen by the attacker. We use loss and cost functions to evaluate the effectiveness of the adversarial attack on MARL agents. The cost is the cumulative sum of the action manipulations and the reward manipulations. If the attacker aims to force the agents to learn a target policy, the loss is the cumulative number of times when the agent does not follow the target policy. Otherwise, the loss is the regret to the policy that maximizes the attacker's rewards. It is clearly of interest to minimize both the loss and cost.
2) We study the attack problem in three different settings: the white-box, the gray-box and the black-box settings. In the white-box setting, the attacker has full information of the underlying environment. In the gray-box setting, the attacker has no prior information about the underlying environment and the agents' algorithm, but knows the target policy that maximizes its cumulative rewards. In the black-box setting, the target policy is also unknown for the attacker.
3) We show that the effectiveness of action poisoning only attacks and reward poisoning only attacks is limited. Even in the white-box setting, we show that there exist some MGs under which no action poisoning only Markov attack strategy or reward poisoning only Markov attack strategy can be efficient and successful. At the same time, we provide some sufficient conditions under which the action poisoning only attacks or the reward poisoning only attacks can efficiently attack MARL algorithms. Under such conditions, we introduce an efficient action poisoning attack strategy and an efficient reward poisoning attack strategy, and analyze their cost and loss.
4) We introduce a mixed attack strategy in the gray-box setting and an approximate mixed attack strategy in the black-box setting. We show that the mixed attack strategy can force any sub-linear regret MARL agents to choose actions according to the target policy specified by the attacker with sub-linear cost and sub-linear loss. We further investigate the impact of the approximate mixed attack strategy attack on V-learning [Jin et al., 2021], a simple, efficient, decentralized algorithm for MARL.
Attacks on Single Agent RL: Adversarial attacks on single agent RL have been studied in various settings [Behzadan and Munir, 2017, Huang and Zhu, 2019, Ma et al., 2019, Zhang et al., 2020b, Sun et al., 2021, Rakhsha et al., 2020, 2021]. For example, [Behzadan and Munir, 2017, Zhang et al., 2020b, Rangi et al., 2022] study online reward poisoning attacks in which the attacker could manipulate the reward signal before the agent receives it. [Liu and Lai, 2021] studies online action poisoning attacks in which the attacker could manipulate the action signal before the environment receives it. [Rangi et al., 2022] studies the limitations of reward only manipulation or action only manipulation in single-agent RL.
Attacks on MARL: [Ma et al., 2022] considers a game redesign problem where the designer knows the full information of the game and can redesign the reward functions. The proposed redesign methods can incentivize players to take a specific target action profile frequently with a small cumulative design cost. [Gleave et al., 2020, Guo et al., 2021] study the poisoning attack on multi-agent reinforcement learners, assuming that the attacker controls one of the learners. [Wu et al., 2022] studies the reward poisoning attack on offline multi-agent reinforcement learners.
Defense Against Attacks on RL: There is also recent work on defending against adversarial attacks on RL [Banihashem et al., 2021, Zhang et al., 2021b, Lykouris et al., 2021, Chen et al., 2021, Wei et al., 2022, Wu et al., 2021]. These work focuses on the single-agent RL setting where an adversary can corrupt the reward and state transition.
2.1 Definitions
To increase the readability of the paper, we first introduce some standard definitions related to MARL that will be used throughout of the paper. These definitions mostly follow those defined in [Jin et al., 2021]. We denote a tabular episodic MG with  $m$  agents by a tuple  $\mathrm{MG}(S, \{\mathcal{A}_i\}_{i=1}^m, H, P, \{R_i\}_{i=1}^m)$ , where  $S$  is the state space with  $|S| = S$ ,  $\mathcal{A}_i$  is the action space for the  $i^{\text{th}}$  agent with  $|\mathcal{A}_i| = A_i$ ,  $H \in \mathbb{Z}^+$  is the number of steps in each episode. We let  $\pmb{a} := (a_1, \dots, a_m)$  denote the joint action of all the  $m$  agents and  $\mathcal{A} := \mathcal{A}_i \times \dots \times \mathcal{A}_m$  denote the joint action space.  $P = \{P_h\}_{h \in [H]}$  is a collection of transition matrices.  $P_h: S \times \mathcal{A} \times S \to [0,1]$  is the probability transition function that maps state-action-state pair to a probability,  $R_{i,h}: S \times \mathcal{A} \to [0,1]$  represents the reward function for the  $i^{\text{th}}$  agent in the step  $h$ . In this paper, the probability transition functions and the reward functions can be different at different steps. We note that this MG model incorporates both cooperation and competition because the reward functions of different agents can be arbitrary.
Interaction protocol: The agents interact with the environment in a sequence of episodes. The total number of episodes is  $K$ . In each episode  $k \in [K]$  of MG, the initial states  $s_1$  is generated randomly by a distribution  $P_0(\cdot)$ . Initial states may be different between episodes. At each step  $h \in [H]$  of an episode, each agent  $i$  observes the state  $s_h$  and chooses an action  $a_{i,h}$  simultaneously. After receiving the action, the environment generates a random reward  $r_{i,h} \in [0,1]$  for each agent  $i$  derived from a distribution with mean  $R_{i,h}(s_h, a_h)$ , and transits to the next state  $s_{h+1}$  drawn from the distribution  $P_h(\cdot | s_h, a_h)$ .  $P_h(\cdot | s, a)$  represents the probability distribution over states if joint action  $a$  is taken for state  $s$ . The agent stops interacting with environment after  $H$  steps and starts another episode. At each time step, the agents may observe the actions played by other agents.
Policy and value function: A Markov policy takes actions only based on the current state. The policy  $\pi_{i,h}$  of agent  $i$  at step  $h$  is expressed as a mappings  $\pi_{i,h}:S\to \Delta_{\mathcal{A}_i}$ .  $\pi_{i,h}(a_i|s)$  represents the probability of agent  $i$  taking action  $a_{i}$  in state  $s$  under policy  $\pi_{i}$  at step  $h$ . A deterministic policy is a policy that maps each state to a particular action. For notation convenience, for a deterministic policy  $\pi_{i}$ , we use  $\pi_{i,h}(s)$  to denote the action  $a_{i}$  which satisfies  $\pi_{i,h}(a_i|s) = 1$ . We denote the product policy of all the agents as  $\pi \coloneqq \pi_1\times \dots \times \pi_m$ . We also denote  $\pi_{-i} \coloneqq \pi_1\times \dots \times \pi_{i - 1}\times \pi_{i + 1}\times \dots \times \pi_m$  to be the product policy excluding agent  $i$ . If every agent follows a deterministic policy, the product policy of all the agents is also deterministic. We use  $V_{i,h}^{\pi}:S\rightarrow \mathbb{R}$  to denote the value function of agent  $i$  at step  $h$  under policy  $\pi$  and define  $V_{i,h}^{\pi}(s)\coloneqq \mathbb{E}\left[\sum_{h^{\prime} = h}^{H}r_{i,h^{\prime}}|s_{h} = s,\pi \right]$ . Given a policy  $\pi$  and step  $h$ , the  $i^{\mathrm{th}}$  agent's  $Q$ -function  $Q_{i,h}^{\pi}:S\times \mathcal{A}\to \mathbb{R}$  of a state-action pair  $(s,\pmb {a})$  is defined as:  $Q_{i,h}^{\pi}(s,\pmb {a}) = \mathbb{E}\left[\sum_{h^{\prime} = h}^{H}r_{i,h^{\prime}}|s_{h} = s,\pmb {a}_{h} = \pmb {a},\pi \right]$ .
Best response: For any policy  $\pi_{-i}$ , there exists a best response of agent  $i$ , which is a policy that achieves the highest cumulative reward for itself if all other agents follow policy  $\pi_{-i}$ . We define the best response of agent  $i$  towards policy  $\pi_{-i}$  as  $\mu^{\dagger}(\pi_{-i})$ , which satisfies  $\mu^{\dagger}(\pi_{-i}) := \arg \max_{\pi_i} V_{i,h}^{\pi_i \times \pi_{-i}}(s)$  for any state  $s$  and any step  $h$ . We denote  $\max_{\pi_i} V_{i,h}^{\pi_i \times \pi_{-i}}(s)$  as  $V_{i,h}^{\dagger, \pi_{-i}}(s)$  for notation simplicity. By its definition, we know that the best response can always be achieved by a deterministic policy.
Nash Equilibrium (NE) is defined as a product policy where no agent can improve his own cumulative reward by unilaterally changing his strategy.
Nash Equilibrium (NE) [Jin et al., 2021]: A product policy  $\pi$  is a NE if for all initial state  $s$ ,  $\max_{i\in [m]}(V_{i,1}^{\dagger,\pi-i}(s)-V_{i,1}^{\pi}(s))=0$  holds. A product policy  $\pi$  is an  $\epsilon$ -approximate Nash Equilibrium if for all initial state  $s$ ,  $\max_{i\in [m]}(V_{i,1}^{\dagger,\pi-i}(s)-V_{i,1}^{\pi}(s))\leq\epsilon$  holds.
General correlated policy: A general Markov correlated policy  $\pi$  is a set of  $H$  mappings  $\pi := \{\pi_h : \Omega \times S \to \Delta_{\mathcal{A}}\}_{h \in [H]}$ . The first argument of  $\pi_h$  is a random variable  $\omega \in \Omega$  sampled from some underlying distributions. For any correlated policy  $\pi = \{\pi_h\}_{h \in [H]}$  and any agent  $i$ , we can define a marginal policy  $\pi_{-i}$  as a set of  $H$  maps  $\pi_i = \{\pi_{h,-i} : \Omega \times S \to \Delta_{\mathcal{A}_{-i}}\}_{h \in [H]}$ , where  $\mathcal{A}_{-i} = \mathcal{A}_1 \times \dots \times \mathcal{A}_{i-1} \times \mathcal{A}_{i+1} \times \dots \times \mathcal{A}_m$ . It is easy to verify that a deterministic joint policy is a product policy. The best response value of agent  $i$  towards policy  $\pi_{-i}$  as  $\mu^{\dagger}(\pi_{-i})$ , which satisfies  $\mu^{\dagger}(\pi_{-i}) := \arg \max_{\pi_i} V_{i,h}^{\pi_i \times \pi_{-i}}(s)$  for any state  $s$  and any step  $h$ .
Coarse Correlated Equilibrium (CCE)[Jin et al., 2021]: A correlated policy  $\pi$  is an CCE if for all initial state  $s$ ,  $\max_{i\in [m]}(V_{i,1}^{\dagger,\pi_{-i}}(s) - V_{i,1}^{\pi}(s)) = 0$  holds. A correlated policy  $\pi$  is an  $\epsilon$ -approximate CCE if for all initial state  $s$ ,  $\max_{i\in [m]}(V_{i,1}^{\dagger,\pi_{-i}}(s) - V_{i,1}^{\pi}(s)) \leq \epsilon$  holds.
Strategy modification: A strategy modification  $\phi_{i}$  for agent  $i$  is a set of mappings  $\phi_{i} := \{(\mathcal{S} \times \mathcal{A})^{h-1} \times \mathcal{S} \times \mathcal{A}_{i} \to \mathcal{A}_{i}\}_{h \in [H]}$ . For any policy  $\pi_{i}$ , the modified policy (denoted as  $\phi_{i} \diamond \pi_{i}$ ) changes the action  $\pi_{i,h}(\omega, s)$  under random sample  $\omega$  and state  $s$  to  $\phi_{i}((s_{1}, a_{1}, \ldots, s_{h}, a_{i,h}), \pi_{i,h}(\omega, s))$ . For any joint policy  $\pi$ , we define the best strategy modification of agent  $i$  as the maximizer of  $\max_{\phi_{i}} V_{i,1}^{(\phi_{i} \diamond \pi_{i}) \odot \pi_{-i}}(s)$  for any initial state  $s$ .
Correlated Equilibrium (CE)[Jin et al., 2021]: A correlated policy  $\pi$  is an CE if for all initial state  $s$ ,  $\max_{i\in [m]}\max_{\phi_i}(V_{i,1}^{(\phi_i\diamond \pi_i)\odot \pi_{-i}}(s) - V_{i,1}^{\pi}(s)) = 0$ . A correlated policy  $\pi$  is an  $\epsilon$ -approximate CE if for all initial state  $s$ ,  $\max_{i\in [m]}\max_{\phi_i}(V_{i,1}^{(\phi_i\diamond \pi_i)\odot \pi_{-i}}(s) - V_{i,1}^{\pi}(s))\leq \epsilon$  holds.
In Markov games, it is known that an NE is an CE, and an CE is an CCE.
Best-in-hindsight Regret: Let  $\pi^k$  denote the product policy deployed by the agents for each episode  $k$ . After  $K$  episodes, the best-in-hindsight regret of agent  $i$  is defined as  $\mathrm{Reg}_i(K,H) = \max_{\pi_i'}\sum_{k = 1}^{K}[V_{i,1}^{\pi_i',\pi_{-i}^k}(s_1^k) - V_{i,1}^{\pi_i^k}(s_1^k)]$ .
2.2 Poisoning attack setting
We are now ready to introduce the considered poisoning attack setting, in which there is an attacker sits between the agents and the environment. The attacker can monitor the states, the actions of the agents and the reward signals from the environment. Furthermore, the attacker can override actions and observations of agents. In particular, at each episode  $k$  and step  $h$ , after each agent  $i$  chooses an action  $a_{i,h}^k$ , the attacker may change it to another action  $\widetilde{a}_{i,h}^k \in \mathcal{A}_i$ . If the attacker does not override the actions, then  $\widetilde{a}_{i,h}^k = a_i$ . When the environment receives  $\widetilde{\boldsymbol{a}}_h^k$ , it generates random rewards  $r_{i,h}^k$  with mean  $R_{i,h}(s_h^k,\widetilde{\boldsymbol{a}}_h^k)$  for each agent  $i$  and the next state  $s_{h + 1}^k$  is drawn from the distribution  $P_{h}(\cdot |s_{h}^{k},\widetilde{\boldsymbol{a}}_{h}^{k})$ . Before each agent  $i$  receives the reward  $r_{i,h}^k$ , the attacker may change it to another reward  $\widetilde{r}_{i,h}^k$ . Agent  $i$  receives the reward  $\widetilde{r}_{i,h}^k$  and the next state  $s_{h + 1}^k$  from the environment. Note that agent  $i$  does not know the attacker's manipulations and the presence of the attacker and hence will still view  $\widetilde{r}_{i,h}^k$  as the reward and  $s_{h + 1}^k$  as the next state generated from state-action pair  $(s_h^k,\boldsymbol {a}_h^k)$ .
In this paper, we call an attack as action poisoning only attack, if the attacker only overrides the action but not the rewards. We call an attack as reward poisoning only attack if the attacker only overrides the rewards but not the actions. In addition, we call an attack as mixed attack if the attack can carry out both action poisoning and reward poisoning attacks simultaneously.
The goal of the MARL learners is to learn an equilibrium. On the other hand, the attacker's goal is to either force the agents to learn a target policy  $\pi^{\dagger}$  of the attacker's choice or to force the agents to learn a policy that maximizes the cumulative rewards under a specific reward function  $R_{\dagger,h}: S \times \mathcal{A} \to (0,1]$  chosen by the attacker. We note that this setup is very general. Different choices of  $\pi^{\dagger}$  or  $R_{\dagger,h}$  could lead to different objectives. For example, if the attacker aims to reduce the benefit of the agent  $i$ , the attacker's reward function  $R_{\dagger,h}$  can be set to  $1 - R_{i,h}$ , or choose a target policy  $\pi^{\dagger}$  that is detrimental to the agent  $i$ 's reward. If the attacker aims to maximize the total rewards of a subset of agents  $\mathcal{C}$ , the attacker's reward function  $R_{\dagger,h}$  can be set to  $\sum_{i \in \mathcal{C}} R_{i,h}$ , or choose a target policy  $\pi^{\dagger} = \arg \max \sum_{i \in \mathcal{C}} V_{i,1}^{\pi}(s_1)$  that maximizes the total rewards of agents in  $\mathcal{C}$ . We assume that the target policy  $\pi^{\dagger}$  is deterministic and  $R_{i,h}(s, \pi^{\dagger}(s)) > 0$ . We measure the performance of the attack over  $K$  episodes by the total attack cost and the attack loss. Set  $\mathbb{1}(\cdot)$  as the indicator function. The attack cost over  $K$  episodes is defined as  $\mathrm{Cost}(K,H) = \sum_{k=1}^{K} \sum_{h=1}^{H} \sum_{i=1}^{m} \left( \mathbb{1}(\widetilde{a}_{i,h}^k \neq a_{i,h}^k) + |\widetilde{r}_{i,h}^k - r_{i,h}^k| \right)$ .
There are two different forms of attack loss based on the different goals of the attacker.
If the attacker's goal is to force the agents to learn a target policy  $\pi^{\dagger}$ , the attack loss over  $K$  episodes is defined as  $\mathrm{Loss1}(K,H) = \sum_{k=1}^{K}\sum_{h=1}^{H}\sum_{i=1}^{m}\mathbb{1}\left(a_{i,h}^{k}\neq \pi_{i,h}^{\dagger}(s_{i,h}^{k})\right)$ .
If the attacker's goal is to force the agents to maximize the cumulative rewards under some specific reward function  $R_{\dagger}$  chosen by the attacker, the attack loss over  $K$  episodes is defined as  $\mathrm{Loss2}(K,H) =$
$\sum_{k=1}^{K}[V_{\dagger,1}^{\pi^{*}}(s_{1}^{k}) - V_{\dagger,1}^{\pi^{k}}(s_{1}^{k})]$ . Here,  $V_{\dagger,1}^{\pi}(s)$  is the expected cumulative rewards in state  $s$  based on the attacker's reward function  $R_{\dagger}$  under product policy  $\pi$  and  $V_{\dagger,1}^{\pi^{*}}(s) = \max_{\pi} V_{\dagger,1}^{\pi}(s)$ .  $\pi^{k}$  denotes the product policy deployed by the agents for each episode  $k$ .  $\pi^{*}$  is the optimal policy that maximizes the attacker's cumulative rewards. We have  $\mathrm{Loss2}(K,H) \leq H * \mathrm{Loss1}(K,H)$ .
Denote the total number of steps as  $T = KH$ . In the proposed poisoning attack problem, we call an attack strategy successful if the attack loss of the strategy scales as  $o(T)$ . Furthermore, we call an attack strategy efficient and successful if both the attack cost and attack loss scale as  $o(T)$ .
The attacker aims to minimize both the attack cost and the attack loss, or minimize one of them subject to a constraint on the other. However, obtaining optimal solutions to these optimization problems is challenging. As the first step towards understanding the attack problem, we show the limitations of the action poisoning only or the reward poisoning only attacks and then propose a simple mixed attack strategy that is efficient and successful.
Depending on the capability of the attacker, we consider three settings: the white-box, the gray-box and the black-box settings. The table below summarizes the differences among these settings.
Table 1: Differences of the white/gray/black-box attackers
white-box attacker	gray-box attacker	black-box attacker
MG	Has full information	No information	No information
π†	Can be calculated if R† given	Required and given	Not given
R†	Not required if π† given	Not required if π† given	Required and given
Loss1	Suitable by specify π†	Suitable	Not suitable
Loss2	Suitable if R† given	Suitable if R† given	Suitable
3 White-box attack strategy and analysis
In this section, to obtain insights to the problem, we consider the white-box model, in which the attacker has full information of the underlying MG  $(\mathcal{S},\{\mathcal{A}_i\}_{i = 1}^m,H,P,\{R_i\}_{i = 1}^m)$ . Even in the white-box attack model, we show that there exist some environments where the attacker's goal cannot be achieved by reward poisoning only attacks or action poisoning only attacks in Section 3.1. Then, in Section 3.2 and Section 3.3, we provide some sufficient conditions under which the action poisoning attacks alone or the reward poisoning attacks alone can efficiently attack MARL algorithms. Under such conditions, we then introduce an efficient action poisoning attack strategy and an efficient reward poisoning attack strategy.
3.1 The limitations of the action poisoning attacks and the reward poisoning attacks
As discussed in Section 2, the attacker aims to force the agents to either follow the target policy  $\pi^{\dagger}$  or to maximize the cumulative rewards under attacker's reward function  $R_{\dagger}$ . In the white-box poisoning attack model, these two goals are equivalent as the optimal policy  $\pi^{*}$  on the attacker's reward function  $R_{\dagger}$  can be calculated by the Bellman optimality equations. To maximize the cumulative rewards under attacker's reward function  $R_{\dagger}$  is equivalent to force the agents to follow the policy  $\pi^{\dagger} = \pi^{*}$ .
Existing MARL algorithms [Liu et al., 2021, Jin et al., 2021] can learn an  $\epsilon$ -approximate  $\{\mathrm{NE}, \mathrm{CE}, \mathrm{CCE}\}$  with  $\widetilde{\mathcal{O}}(1/\epsilon^2)$  sample complexities. To force the MARL agents to follow the policy  $\pi^\dagger$ , the attacker first needs to attack the agents such that the target policy  $\pi^\dagger$  is the unique NE in the observation of the agents. However, this alone is not enough to force the MARL agents to follow the policy  $\pi^\dagger$ . Any other distinct policy should not be an  $\epsilon$ -approximate CCE. The reason is that, if there exists an  $\epsilon$ -approximate CCE  $\pi$  such that  $\pi(\pi^\dagger(s)|s) = 0$  for any state  $s$ , the agents, using existing MARL algorithms, may learn and then follow  $\pi$ , which will lead the attack loss to be  $\mathcal{O}(T) = \mathcal{O}(KH)$ . Hence, we need to ensure that any  $\epsilon$ -approximate CCE stays in the neighborhood of the target policy. This requirement is equivalent to achieve the following objective: for all  $s \in S$ , and policy  $\pi$ ,