SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yunkai Zhang
yunkai_zhang@berkeley.edu
Yuhao Ding
yuhao_ding@berkeley.edu
Alec Koppel
Artificial Intelligence Research
alec.koppel@jpmchase.com
We investigate safe multi-agent reinforcement learning, where agents seek to collectively maximize an aggregate sum of local objectives while satisfying their own safety constraints. The objective and constraints are described by general utilities, i.e., nonlinear functions of the long-term state-action occupancy measure, which encompass broader decision-making goals such as risk, exploration, or imitations. The exponential growth of the state-action space size with the number of agents presents challenges for global observability, further exacerbated by the global coupling arising from agents' safety constraints. To tackle this issue, we propose a primal-dual method utilizing shadow reward and  $\kappa$ -hop neighbor truncation under a form of correlation decay property, where  $\kappa$  is the communication radius. In the exact setting, our algorithm converges to a first-order stationary point (FOSP) at the rate of  $\mathcal{O}\left(T^{-2/3}\right)$ . In the sample-based setting, we demonstrate that, with high probability, our algorithm requires  $\widetilde{\mathcal{O}}\left(\epsilon^{-3.5}\right)$  samples to achieve an  $\epsilon$ -FOSP with an approximation error of  $\mathcal{O}(\phi_0^{2\kappa})$ , where  $\phi_0 \in (0,1)$ . Finally, we demonstrate the effectiveness of our model through extensive numerical experiments.
Cooperative multi-agent reinforcement learning (MARL) involves agents operating within a shared environment, where each agent's decisions influence not only their objectives, but also those of others and the state trajectories [1]. In seeking to bring conceptually sound MARL techniques out of simulation [2, 3] and into real-world environments [4, 5], some key issues emerge: safety and communications overhead implied by a training mechanism. Although experimentally, the centralized training decentralized execution (CTDE) framework has gained traction recently [6, 7], its requirement for centralized data collection can pose issues for large-scale [8] or privacy-sensitive applications [9]. Therefore, we prioritize decentralized training, where to date most MARL techniques impose global state observability for performance certification [1]. In this work, we extend recent efforts to alleviate this bottleneck [10] especially in the case of safety critical settings, in a flexible manner that allows agents to incorporate risk, exploration, or prior information.
More specifically, we hypothesize that the multi-agent system consists of a network of agents that interact with each other locally according to an underlying dependence graph [10]. Second, to model safety constraints in reinforcement learning (RL), we adopt a standard approach based on constrained
Markov Decision Processes (CMDPs) [11], where one maximizes the expected total reward subject to a safety-related constraint on the expected total utility. Third, since many decision-making problems take a form beyond the classic cumulative reward, such as apprenticeship learning [12], diverse skill discovery [13], pure exploration [14], and state marginal matching [15], we focus on utility functions defined as nonlinear functions of the induced state-action occupancy measure, which can be abstracted as RL with general utilities [16, 17].
Towards formalizing the approach, we consider an MARL model consisting of  $n$  agents, each with its own local state  $s_i$  and action  $a_i$ , where the multi-agent system is associated with an underlying dependence graph  $\mathcal{G}$ . Each agent is privately associated with two local general utilities  $f_i(\cdot)$  and  $g_i(\cdot)$ , where  $f_i(\cdot)$  and  $g_i(\cdot)$  are functions of the local occupancy measure. The objective is to find a safe policy for each agent that maximizes the average of the local objective utilities, namely,  $1/n \cdot \sum_{i=1}^{n} f_i(\cdot)$ , and satisfies each agent's individual safety constraint described by its local utility  $g_i(\cdot)$ . This setting captures a wide range of safety-critical applications, for example, resource allocation for the control of networked epidemic models [18], influence maximization in social networks [19], portfolio optimization in interbank network structures [20], intersection management for connected vehicles [21], and energy constraints of wireless communication networks [22].
Despite the significance of safe MARL with general utilities, prior works have either ignored the necessity of safety [23] or the computational bottleneck associated with global information exchange regarding the state and action per step [24]. In fact, the interaction of these two aspects requires addressing the fact that each agent's own safety constraint requires information from all others. In particular, the existing works in safe MARL allow full access to the global state or unlimited communications among all agents for policy implementation, value estimation, and constraint satisfaction [25, 26, 27]. However, this assumption is impractical due to the "curse of dimensionality" [28], as well as the limited information exchanges and communications among agents [29].
Therefore, to our knowledge, there is no methodology to both guarantee safety and incur manageable communications overhead for each agent. Compounding these issues is the fact that standard RL training schemes based on the policy gradient theorem [30] are not applicable in the context of general utilities. This deviation from the cumulative rewards adds to the difficulty of estimating the gradient, since there does not exist a policy-independent reward function. We refer the reader to Appendix A for an extended discussion of related works.
To address these challenges, we focus on the setting of distributed training without global observability and aim to develop a scalable algorithm with theoretical guarantees. Our main contributions are summarized below:
- Compared with existing theoretical works on safe MARL [25, 26, 31], we present the first safe MARL formulation that extends beyond cumulative forms in both the objective and constraints. We develop a truncated policy gradient estimator utilizing shadow reward and  $\kappa$ -hop policies under a form of correlation decay property, where  $\kappa$  represents the communication radius. The approximation errors arising from both policy implementation and value estimation are quantified.
- Despite of the global coupling of agents' local utility functions, we propose a scalable Primal-Dual Actor-Critic method, which allows each agent to update its policy based only on the states and actions of its close neighbors and under limited communications. The effectiveness of the proposed algorithm is verified through numerical experiments.
- From the perspective of optimization, we devise new tools to analyze the convergence of the algorithm. In the exact setting, we establish an  $\mathcal{O}\left(T^{-2/3}\right)$  convergence rate for finding an FOSP, matching the standard convergence rate for solving nonconcave-convex saddle point problems. In the sample-based setting, we prove that, with high probability, the algorithm requires  $\widetilde{\mathcal{O}}\left(\epsilon^{-3.5}\right)$  samples to obtain an  $\epsilon$ -FOSP with an approximation error of  $\mathcal{O}(\phi_0^{2\kappa})$ , where  $\phi_0 \in (0,1)$ .
Consider a Constrained Markov Decision Process (CMDP) over a finite state space  $\mathcal{S}$  and a finite action space  $\mathcal{A}$  with a discount factor  $\gamma \in [0,1)$ . A policy  $\pi$  is a function that specifies the decision rule of the agent, i.e., the agent takes action  $a \in \mathcal{A}$  with probability  $\pi(a|s)$  in state  $s \in \mathcal{S}$ . When action  $a$  is taken, the transition to the next state  $s'$  from state  $s$  follows the probability distribution
$s' \sim \mathbb{P}(\cdot | s, a)$ . Let  $\rho$  be the initial distribution. For each policy  $\pi$  and state-action pair  $(s, a) \in S \times \mathcal{A}$ , the discounted state-action occupancy measure is defined as
$$
\lambda^ {\pi} (s, a) = \sum_ {k = 0} ^ {\infty} \gamma^ {k} \mathbb {P} \left(s ^ {k} = s, a ^ {k} = a \mid \pi , s ^ {0} \sim \rho\right). \tag {1}
$$
The goal of the agent is to find a policy  $\pi$  that maximizes a general objective described by a (possibly) nonlinear function  $f(\cdot)$  of  $\lambda^{\pi}$ , known as the general utility, subject to a constraint in the form of another general utility  $g(\cdot)$ , namely
$$
\max  _ {\pi} f \left(\lambda^ {\pi}\right) \quad \text {s . t .} \quad g \left(\lambda^ {\pi}\right) \geq 0. \tag {2}
$$
When  $f(\cdot) = \langle r, \cdot \rangle$  and  $g(\cdot) = \langle u, \cdot \rangle$  are linear functions, (2) recovers the standard CMDP problem:
$$
\max  _ {\pi} V ^ {\pi} (r) = \mathbb {E} \left[ \sum_ {k = 0} ^ {\infty} \gamma^ {k} r \left(s ^ {k}, a ^ {k}\right) \mid \pi , s ^ {0} \sim \rho \right], \text {s . t .} V ^ {\pi} (u) = \mathbb {E} \left[ \sum_ {k = 0} ^ {\infty} \gamma^ {k} u \left(s ^ {k}, a ^ {k}\right) \mid \pi , s ^ {0} \sim \rho \right] \geq 0, \tag {3}
$$
where  $V^{\pi}(\cdot)$  is usually referred to as the value function. In contrast, it has been shown that for some MDPs, there is no standard value function that can be equivalent to the general utility [16, Lemma 1]. In Appendix C, we provide more examples of formulation (2) beyond standard value functions.
In this work, we study the decentralized version of problem (2). Consider the system is composed of a network of agents associated with a graph  $\mathcal{G} = (\mathcal{N},\mathcal{E}_{\mathcal{G}})$  (not densely connected in general), where the vertex set  $\mathcal{N} = \{1,2,\ldots ,n\}$  denotes the set of  $n$  agents and the edge set  $\mathcal{E}_{\mathcal{G}}$  prescribes the communication links among the agents. Let  $d(i,j)$  be the length of the shortest path between agents  $i$  and  $j$  on  $\mathcal{G}$ . For  $\kappa \geq 0$ , let  $\mathcal{N}_i^\kappa = \{j\in \mathcal{N}|d(i,j)\leq \kappa \}$  denote the set of agents in the  $\kappa$ -hop neighborhood of agent  $i$ , with the shorthand notation  $\mathcal{N}_{-i}^{\kappa}\coloneqq \mathcal{N}\backslash \mathcal{N}_{i}^{\kappa}$  and  $-i\coloneqq \mathcal{N}\backslash \{i\}$ . The details of the decentralized nature of the system are summarized below:
Space decomposition The global state and action spaces are the product of local spaces, i.e.,  $\mathcal{S} = \mathcal{S}_1\times \mathcal{S}_2\times \dots \times \mathcal{S}_n$ $\mathcal{A} = \mathcal{A}_1\times \mathcal{A}_2\times \dots \times \mathcal{A}_n$  , meaning that for every  $s\in S$  and  $a\in \mathcal{A}$  , we can write  $s = (s_{1},s_{2},\ldots ,s_{n})$  and  $a = (a_{1},a_{2},\ldots ,a_{n})$  . For each subset  $\mathcal{N}'\subset \mathcal{N}$  , we use  $(s_{\mathcal{N}'}$ $a_{\mathcal{N}'}$  ) to denote the state-action pair for the agents in  $\mathcal{N}'$
Observation and communication Each agent  $i$  only has direct access to its own state  $s_i$  and action  $a_i$ , while being allowed to communicate with its  $\kappa$ -hop neighborhood  $\mathcal{N}_i^\kappa$  for information exchanges. The communication radius  $\kappa$  is a given but tunable parameter.
Transition decomposition Given the current global state  $s$  and action  $a$ , the local states in the next period are independently generated, i.e.,  $\mathbb{P}(s'|s,a) = \prod_{i\in \mathcal{N}}\mathbb{P}_i(s_i'|s,a)$ ,  $\forall s^{\prime}\in S$ , where we use  $\mathbb{P}_i$  to denote the local transition probability for agent  $i$ .
Policy factorization The global policy can be expressed as the product of local policies, such that  $\pi(a|s) = \prod_{i \in \mathcal{N}} \pi^i(a_i|s), \forall (s,a)$ , i.e., given the global state  $s$ , each agent  $i$  acts independently based on its local policy  $\pi^i$ . We assume that each local policy  $\pi^i$  is parameterized by a parameter  $\theta_i$  within a convex set  $\Theta_i$ . Thus, we can write  $\pi(a|s) = \pi_\theta(a|s) = \prod_{i \in \mathcal{N}} \pi_{\theta_i}^i(a_i|s)$ , where  $\theta \in \Theta = \Theta_1 \times \Theta_2 \times \dots \times \Theta_n$  is the concatenation of local parameters.
Localized objective and constraint For each agent  $i$  and its local state-action pair  $(s_i, a_i)$ , the local state-action occupancy measure under policy  $\pi$  is defined as
$$
\lambda_ {i} ^ {\pi} \left(s _ {i}, a _ {i}\right) = \sum_ {k = 0} ^ {\infty} \gamma^ {k} \mathbb {P} \left(s _ {i} ^ {k} = s _ {i}, a _ {i} ^ {k} = a _ {i} \mid \pi , s ^ {0} \sim \rho\right), \tag {4}
$$
which can be viewed as the marginalization of the global occupancy measure, i.e.,  $\lambda_i^\pi (s_i,a_i) = \sum_{s_{-i},a_{-i}}\lambda^{\pi}(s,a)$ . Each agent  $i$  is privately associated with two local (general) utilities  $f_{i}(\cdot)$  and  $g_{i}(\cdot)$ , which are functions of the local occupancy measure  $\lambda_i^\pi$ . Agents cooperate with each other aiming at maximizing the global objective  $f(\cdot)$ , defined as the average of local utilities  $\{f_i(\cdot)\}_{i\in \mathcal{N}}$ , while each agent  $i$  needs to satisfy its own safety constraint described by the local utility  $g_{i}(\cdot)$ . Then, under the parameterization  $\pi_{\theta}$ , (2) can be rewritten as
$$
\max  _ {\theta \in \Theta} F (\theta) := \frac {1}{n} \sum_ {i \in \mathcal {N}} f _ {i} \left(\lambda_ {i} ^ {\pi_ {\theta}}\right), \text {s . t .} G _ {i} (\theta) := g _ {i} \left(\lambda_ {i} ^ {\pi_ {\theta}}\right) \geq 0, \forall i \in \mathcal {N}. \tag {5}
$$
Note that problem (5) is not separable among agents due to the coupling of occupancy measures. Compared to the formulation where the constraint is modeled as the average of local constraints, e.g.,
[27], (5) is stricter and more interpretable. We emphasize that the method proposed in this paper does not require the relaxation of local constraints in (5) to a joint constraint and it directly generalizes to the case of multiple constraints per agent.
Consider the Lagrangian function associated with (5):
$$
\mathcal {L} (\theta , \mu) := F (\theta) + \frac {1}{n} \sum_ {i \in \mathcal {N}} \mu_ {i} G _ {i} (\theta) = \frac {1}{n} \sum_ {i \in \mathcal {N}} \left[ f _ {i} \left(\lambda_ {i} ^ {\pi_ {\theta}}\right) + \mu_ {i} g _ {i} \left(\lambda_ {i} ^ {\pi_ {\theta}}\right) \right], \tag {6}
$$
where  $\mu \in \mathbb{R}_+^n$  is the Lagrangian multiplier. The Lagrangian formulation [32] of (5) can be written as
$$
\max  _ {\theta \in \Theta} \min  _ {\mu \geq 0} \mathcal {L} (\theta , \mu). \tag {7}
$$
Since the general utilities  $f_{i}(\lambda_{i}^{\pi_{\theta}})$  and  $g_{i}(\lambda_{i}^{\pi_{\theta}})$  may not be non-concave w.r.t.  $\theta$  even in the form of cumulative rewards, finding the global optimum to (5) is NP-hard in general [33]. Our goal in this work is to develop a scalable and provably efficient gradient-based primal-dual algorithm that can find the first-order stationary points of (5).
3 Scalable primal-dual actor-critic method
For a standard value function with the reward  $r \in \mathbb{R}^{|S| \times |\mathcal{A}|}$ , denoted as  $V^{\pi_{\theta}}(r) = \langle r, \lambda^{\pi_{\theta}} \rangle$ , the policy gradient theorem (see Lemma D.1) yields that
$$
\nabla_ {\theta} V ^ {\pi_ {\theta}} (r) = r ^ {\top} \cdot \nabla_ {\theta} \lambda^ {\pi_ {\theta}} = \frac {1}{1 - \gamma} \mathbb {E} _ {s \sim d ^ {\pi_ {\theta}}, a \sim \pi_ {\theta} (\cdot | s)} \left[ \nabla_ {\theta} \log \pi_ {\theta} (a | s) \cdot Q ^ {\pi_ {\theta}} (r; s, a) \right],
$$
where  $d^{\pi_{\theta}}(s) \coloneqq (1 - \gamma)\sum_{a\in A}\lambda^{\pi_{\theta}}(s,a)$  is the discounted state occupancy measure,  $\nabla_{\theta}\log \pi_{\theta}(\cdot |\cdot)$  is the score function, and  $Q^{\pi_{\theta}}(r;\cdot ,\cdot)$  is the Q-function with the reward  $r$ , defined as
$$
Q ^ {\pi_ {\theta}} (r; s, a) = \mathbb {E} \left[ \sum_ {k = 0} ^ {\infty} \gamma^ {k} r \left(s ^ {k}, a ^ {k}\right) \mid \pi_ {\theta}, s ^ {0} = s, a ^ {0} = a \right]. \tag {8}
$$
Although this elegant result no longer holds for general utilities, we can apply the chain rule:
$$
\nabla_ {\theta} f \left(\lambda^ {\pi_ {\theta}}\right) = \left[ \nabla_ {\lambda} f \left(\lambda^ {\pi_ {\theta}}\right) \right] ^ {\top} \cdot \nabla_ {\theta} \lambda^ {\pi_ {\theta}} = \nabla_ {\theta} V ^ {\pi_ {\theta}} \left(\nabla_ {\lambda} f \left(\lambda^ {\pi_ {\theta}}\right)\right), \tag {9}
$$
i.e., the gradient  $\nabla_{\theta}f(\lambda^{\pi_{\theta}})$  is equal to the policy gradient of a standard value function with the reward  $\nabla_{\lambda}f(\lambda^{\pi_{\theta}})$ . We introduce the following definitions [23] for the distributed problem (5).
Definition 3.1 (Shadow reward and shadow Q-function). For each agent  $i$ , define  $r_{f_i}^{\pi_\theta} \coloneqq \nabla_{\lambda_i} f_i(\lambda_i^{\pi_\theta}) \in \mathbb{R}^{|\mathcal{S}_i| \times |\mathcal{A}_i|}$  as the (local) shadow reward for the utility  $f_i(\cdot)$  under policy  $\pi_\theta$ . Define  $Q_{f_i}^{\pi_\theta}(s,a) \coloneqq Q^{\pi_\theta}(r_{f_i}^{\pi_\theta};s,a)$  as the associated (local) shadow Q-function for  $f_i(\cdot)$ . Similarly, let  $r_{g_i}^{\pi_\theta}$  and  $Q_{g_i}^{\pi_\theta}(s,a)$  be the shadow reward and the Q function for  $g_i(\cdot)$ .
Combining Definition 3.1 with (9), we can write the local gradient for agent  $i$ , i.e.,  $\nabla_{\theta_i}\mathcal{L}(\theta ,\mu)$ , as
$$
\nabla_ {\theta_ {i}} \mathcal {L} (\theta , \mu) = \frac {1}{1 - \gamma} \mathbb {E} _ {s \sim d ^ {\pi_ {\theta}}, a \sim \pi_ {\theta} (\cdot | s)} \left[ \nabla_ {\theta_ {i}} \log \pi_ {\theta_ {i}} ^ {i} (a _ {i} | s) \cdot \frac {1}{n} \sum_ {j \in \mathcal {N}} \left(Q _ {f _ {j}} ^ {\pi_ {\theta}} (s, a) + \mu_ {j} Q _ {g _ {j}} ^ {\pi_ {\theta}} (s, a)\right) \right], \tag {10}
$$
where we apply the policy factorization to arrive at  $\nabla_{\theta_i}\log \pi_\theta (a|s) = \nabla_{\theta_i}\log \pi_{\theta_i}^i (a_i|s)$ . By (10), each agent needs to know the shadow Q functions of all agents, as well as the global state, to evaluate its own gradient. However, especially in large networks, this is both inefficient, due to the communication cost, and impractical because of the limited communication radius. In the remainder of this section, we aim to design a scalable estimator for  $\nabla_{\theta_i}\mathcal{L}(\theta ,\mu)$  that requires only local communications.
3.1 Spatial correlation decay and  $\kappa$ -hop policies
Inspired by [34], we assume that the transition probability satisfies a form of the spatial correlation decay property [35, 36].
Assumption 3.2. For a matrix  $M \in \mathbb{R}^{n \times n}$  whose  $(i,j)$ -th entry is defined as
$$
M _ {i j} = \sup  _ {s _ {j}, a _ {j}, s _ {j} ^ {\prime}, a _ {j} ^ {\prime}, s _ {- j}, a _ {- j}} \left\| \mathbb {P} _ {i} \left(\cdot | s _ {j}, s _ {- j}, a _ {j}, a _ {- j}\right) - \mathbb {P} _ {i} \left(\cdot | s _ {j} ^ {\prime}, s _ {- j}, a _ {j} ^ {\prime}, a _ {- j}\right) \right\| _ {1}, \tag {11}
$$
assume that there exists  $\omega > 0$  such that  $\max_{i \in \mathcal{N}} \sum_{j \in \mathcal{N}} e^{\omega d(i,j)} M_{ij} \leq \chi$  with  $\chi < 2/\gamma$ , where  $\gamma$  is the discount factor.
The value of  $M_{ij}$  reflects the extent to which agent  $j$ 's state and action influence the local transition probability of agent  $i$ . Thus, Assumption 3.2 amounts to requiring this influence to decrease exponentially with the distance between any two agents. Such a decay is often observed in many large-scale real-world systems, e.g., the strength of signals decreases exponentially with distance [37].
Furthermore, as mentioned earlier, the implementation of the local policy  $\pi_{\theta_j}^i (\cdot |s)$  is still impractical, since it requires access to the global state  $s$ , while the allowable communication radius is limited to  $\kappa$ . To alleviate this issue, we focus on a specific class of policies in which the local policy of agent  $i$  only depends on the states of these agents in its  $\kappa$ -hop neighborhood  $\mathcal{N}_i^\kappa$ . This class of policies is also referred to as  $\kappa$ -hop policies in the concurrent work [38].
Assumption 3.3 ( $\kappa$ -hop policies). For each agent  $i \in \mathcal{N}$  and  $\theta \in \Theta$ , the local policy  $\pi_{\theta_i}^i(\cdot|s)$  depends only on the neighbor states  $s_{\mathcal{N}_i^\kappa}$ , i.e.,
$$
\pi_ {\theta_ {i}} ^ {i} \left(\cdot | s _ {\mathcal {N} _ {i} ^ {\kappa}}, s _ {\mathcal {N} _ {- i} ^ {\kappa}}\right) = \pi_ {\theta_ {i}} ^ {i} \left(\cdot | s _ {\mathcal {N} _ {i} ^ {\kappa}}, s _ {\mathcal {N} _ {- i} ^ {\kappa}} ^ {\prime}\right), \forall s \in \mathcal {S} a n d \forall s _ {\mathcal {N} _ {- i} ^ {\kappa}} ^ {\prime} \in \mathcal {S} _ {\mathcal {N} _ {- i} ^ {\kappa}}. \tag {12}
$$
For simplicity, we use the notation  $\pi_{\theta_i}^i (\cdot |s) = \pi_{\theta_i}^i (\cdot |s_{\mathcal{N}_i^\kappa})$  for  $\kappa$ -hop policies when it is clear from context. We note that, for any original policy function  $\pi_{\theta}(\cdot |s)$ , an induced  $\kappa$ -hop policy  $\hat{\pi}_{\theta}(\cdot |s_{\mathcal{N}_i^\kappa})$  can be defined by fixing the states  $s_{\mathcal{N}_{-i}^{\kappa}}$  to some arbitrary values and focusing only on the states of agents in  $\mathcal{N}_i^\kappa$ . When considering only  $\kappa$ -hop policies, it is essential to understand how much information is lost compared to the case where agents have access to the global states. The following proposition quantifies the maximum information loss in terms of the occupancy measure under the assumption that the original policy function also satisfies a spatial correlation decay property.
Proposition 3.4. Suppose that there exist  $c \geq 0$  and  $\phi \in [0,1)$  such that for every  $\theta \in \Theta$ , agent  $i \in \mathcal{N}$ , and states  $s, s' \in S$  such that  $s_{\mathcal{N}_i^\kappa} = s_{\mathcal{N}_i^\kappa}'$ , we have  $\left\| \pi_{\theta_i}^i(\cdot |s) - \pi_{\theta_i}^i(\cdot |s') \right\|_1 \leq c\phi^\kappa$ . Let  $\hat{\pi}_{\theta}$  be an induced  $\kappa$ -hop policy of  $\pi_{\theta}$ . Then, it holds that
$$
\left\| \lambda_ {i} ^ {\hat {\pi} _ {\theta}} - \lambda_ {i} ^ {\pi_ {\theta}} \right\| _ {1} \leq \frac {n c \phi^ {k}}{(1 - \gamma) ^ {2}}, \forall i \in \mathcal {N}. \tag {13}
$$
The condition on the local policy in Proposition 3.4 encodes that every  $\pi_{\theta_i}^i$  is exponentially less sensitive to the states of agents outside  $\mathcal{N}_i^\kappa$ , which is a common assumption in MARL to alleviate computationally burdensome and practically intractable communication requirements imposed by the global observability [34, 39, 38]. By Proposition 3.4, the difference in occupancy measures under  $\pi_{\theta}$  and  $\hat{\pi}_{\theta}$  is controlled by  $\| \pi_{\theta_i}^i - \hat{\pi}_{\theta_i}^i \|_1$ . Therefore, if  $f_i(\lambda^\pi)$  and  $g_i(\lambda^\pi)$  are Lipschitz continuous w.r.t.  $\lambda^\pi$ , Proposition 3.4 implies an  $\mathcal{O}(\phi^\kappa)$  approximation of the Lagrangian function (6) using  $\kappa$ -hop policies. The faster the spatial decay of policy is, the more accurate the approximation of the  $\kappa$ -hop policy is. This justifies our focus on learning a  $\kappa$ -hop policy.
3.2 Truncated policy gradient estimator
In the absence of global observability, it is critical to find a scalable estimator for the local gradient  $\nabla_{\theta_i}\mathcal{L}(\theta ,\mu)$  in (10), so that each agent can update its local policy with limited communications.
By leveraging the similar idea in the definition of  $\kappa$ -hop policies, we define the  $\kappa$ -hop truncated (shadow)  $Q$ -function, denoted as  $\widehat{Q}_{\diamond_i}^{\pi_\theta}: S_{\mathcal{N}_i^\kappa} \times \mathcal{A}_{\mathcal{N}_i^\kappa} \to \mathbb{R}$ , to be