SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\widehat {Q} _ {\diamond_ {i}} ^ {\pi_ {\theta}} \left(s _ {\mathcal {N} _ {i} ^ {\kappa}}, a _ {\mathcal {N} _ {i} ^ {\kappa}}\right) := Q _ {\diamond_ {i}} ^ {\pi_ {\theta}} \left(s _ {\mathcal {N} _ {i} ^ {\kappa}}, \bar {s} _ {\mathcal {N} _ {- i} ^ {\kappa}}, a _ {\mathcal {N} _ {i} ^ {\kappa}}, \bar {a} _ {\mathcal {N} _ {- i} ^ {\kappa}}\right), \forall \left(s _ {\mathcal {N} _ {i} ^ {\kappa}}, a _ {\mathcal {N} _ {i} ^ {\kappa}}\right) \in \mathcal {S} _ {\mathcal {N} _ {i} ^ {\kappa}} \times \mathcal {A} _ {\mathcal {N} _ {i} ^ {\kappa}}, \diamond \in \{f, g \}, \tag {14}
$$
where  $(\bar{s}_{\mathcal{N}_i^\kappa},\bar{a}_{\mathcal{N}_i^\kappa})$  is any fixed state-action pair for the agents in  $\mathcal{N}_{-i}^{\kappa}$ . Now, we introduce the following truncated policy gradient estimator for agent  $i$ :
$$
\widehat {\nabla} _ {\theta_ {i}} \mathcal {L} (\theta , \mu) = \frac {1}{1 - \gamma} \underset {a \sim \pi_ {\theta} (\cdot | s)} {\mathbb {E}} \left[ \nabla_ {\theta_ {i}} \log \pi_ {\theta_ {i}} ^ {i} \left(a _ {i} \mid s _ {\mathcal {N} _ {i} ^ {\kappa}}\right) \cdot \frac {1}{n} \sum_ {j \in \mathcal {N} _ {i} ^ {\kappa}} \left(\widehat {Q} _ {f _ {j}} ^ {\pi_ {\theta}} \left(s _ {\mathcal {N} _ {j} ^ {\kappa}}, a _ {\mathcal {N} _ {j} ^ {\kappa}}\right) + \mu_ {j} \widehat {Q} _ {g _ {j}} ^ {\pi_ {\theta}} \left(s _ {\mathcal {N} _ {j} ^ {\kappa}}, a _ {\mathcal {N} _ {j} ^ {\kappa}}\right)\right) \right]. \tag {15}
$$
In comparison to the true policy gradient (10),  $\widehat{\nabla}_{\theta_i}\mathcal{L}(\theta ,\mu)$  replaces the shadow Q-functions with their truncated versions and only considers the agents in the  $\kappa$  -hop neighborhood  $\mathcal{N}_i^\kappa$  . Surprisingly, the following lemma shows that the approximation error of  $\widehat{\nabla}_{\theta_i}\mathcal{L}(\theta ,\mu)$  decreases exponentially with  $\kappa$  when the shadow rewards and the score functions are bounded.
Lemma 3.5. Suppose that Assumptions 3.2 and 3.3 hold and there exist  $M_r, M_\pi > 0$  such that  $\left\| r_{\diamond_i}^{\pi_\theta} \right\|_\infty \leq M_r$  and  $\left\| \nabla_{\theta_i} \log \pi_{\theta_i}^i \right\|_2 \leq M_\pi$ , for every  $\diamond \in \{f, g\}$ ,  $\theta \in \Theta$ ,  $i \in \mathcal{N}$ . Then, for all  $\theta \in \Theta$ ,  $i \in \mathcal{N}$ , we have that
$$
\left\| \widehat {\nabla} _ {\theta_ {i}} \mathcal {L} (\theta , \mu) - \nabla_ {\theta_ {i}} \mathcal {L} (\theta , \mu) \right\| _ {2} \leq \frac {(1 + \| \mu \| _ {\infty}) M _ {\pi} c _ {0} \phi_ {0} ^ {\kappa}}{1 - \gamma} = \mathcal {O} \left(\phi_ {0} ^ {\kappa}\right), \tag {16}
$$
where  $c_0 = 2\gamma \chi M_r / (2 - \gamma \chi)$  and  $\phi_0 = e^{-\omega}$ .
Recall that the shadow reward is defined as the gradient of  $f_{i}(\cdot)$  or  $g_{i}(\cdot)$  w.r.t. the local occupancy measure. Since the set of all possible occupancy measures is compact (see (43)), the existence of  $M_r > 0$  in Lemma 3.5 is satisfied if  $f_{i}(\cdot)$  and  $g_{i}(\cdot)$  are continuously differentiable. The main advantage of using the estimator  $\widehat{\nabla}_{\theta_i}\mathcal{L}(\theta ,\mu)$  lies in that every agent  $i$  only needs to know the truncated Q-functions of agents in its neighborhood  $\mathcal{N}_i^\kappa$ , which can significantly reduce the communication burden and the storage requirement when graph  $\mathcal{G}$  is not densely connected. The proof of Lemma 3.5 can be found in Appendix E.2.
3.3 Algorithm design
Using the results of the preceding section, we put together all the pieces and propose the Primal-Dual Actor-Critic Method with Shadow Reward and  $\kappa$ -hop Policy, as outlined in Algorithm 1. It includes three stages: policy evaluation by the critic, Lagrangian multiplier update, and policy update by the actor. Below, we provide an overview of Algorithm 1, while referring the reader to Appendix D for a flow diagram (Figure 2) of the algorithm, as well as a more detailed discussion.
Stage 1 (policy evaluation by the critic, lines 3-6) In each iteration  $t$ , the current policy  $\pi_{\theta^t}$  is simulated to generate a batch of trajectories, while each agent  $i$  collects its neighborhood trajectories, i.e., the state-action pairs of the agents in  $\mathcal{N}_i^\kappa$ , as batch  $\mathcal{B}_i^t$ . Then, the batch is used to estimate the local occupancy measures  $\lambda_i^{\pi_\theta t}$  through (17), which are subsequently applied to compute the empirical values for the constraint function  $g_i(\lambda_i^{\pi_\theta t})$  and shadow rewards  $r_{f_i}^{\pi_\theta t}$  and  $r_{g_i}^{\pi_\theta t}$ , denoted as  $\widetilde{g_i^t}$ ,  $\widetilde{r}_{f_i}^t$ , and  $\widetilde{r}_{g_i}^t$ , respectively. It is worth mentioning that, when all utility functions reduce to the form of cumulative rewards, the above operation is unnecessary, since all agents have policy-independent local reward functions.
Next, the agents jointly conduct a distributed evaluation subroutine to estimate their truncated shadow Q-functions  $\{\widehat{Q}_{\diamond_i}^{\pi_\theta t}\}_{i\in \mathcal{N}}$  using empirical shadow rewards  $\{\widetilde{r}_{\diamond i}^{t}\}_{i\in \mathcal{N}}$ , where  $\diamond \in \{f,g\}$ . During the subroutine, each agent  $i$  communicates with its neighbor in  $\mathcal{N}_i^\kappa$  to exchange state-action information, but only needs to access its own empirical shadow reward  $\widetilde{r}_{\diamond i}^{t}$ . In principle, any existing approach that satisfies the observation and communication requirements can be used for the truncated Q-function estimation, such as [40, 41, 42]. As an example subroutine, we introduce the Temporal Difference (TD) learning method [43], which is outlined as Algorithm 2 in Appendix D.
Stage 2 (Lagrangian multiplier update, line 7) Instead of employing the projected gradient descent, we propose to update the dual variables by the following formula:
$$
\mu^ {t + 1} = \underset {\mu \in \mathcal {U}} {\operatorname {a r g m i n}} \mathcal {L} \left(\theta^ {t}, \mu\right) + \frac {1}{2 \eta_ {\mu}} \| \mu \| _ {2} ^ {2} = \mathcal {P} _ {\mathcal {U}} \left(- \eta_ {\mu} \nabla_ {\mu} \mathcal {L} \left(\theta^ {t}, \mu^ {t}\right)\right), \tag {22}
$$
where weight  $\eta_{\mu}$  can be viewed as the dual "step-size". In practice, we replace the true dual gradient  $\nabla_{\mu_i}\mathcal{L}(\theta^t,\mu^t) = g_i(\lambda_i^{\pi_\theta t}) / n$  with its empirical estimator  $\widetilde{\nabla}_{\mu_i}\mathcal{L}(\theta^t,\mu^t)$ . The feasible region for the dual variable is denoted by  $\mathcal{U}\subseteq \mathbb{R}_+^n$  and will be specified later.
Stage 3 (policy update by the actor, lines 8-9) To perform the policy update, each agent  $i$  first shares its updated dual variable  $\mu_i^{t + 1}$  and the values of its estimated truncated Q-functions along the trajectories in batch  $\mathcal{B}_i^t$  with the agents in its  $\kappa$ -hop neighborhood  $\mathcal{N}_i^\kappa$ . Then, the agent estimates its truncated policy gradient  $\widehat{\nabla}_{\theta_i} \mathcal{L}(\theta^t, \mu^{t + 1})$  through a REINFORCE-based mechanism [44] as described in (20). Finally, each agent  $i$  updates its local policy parameter by a projected gradient ascent.
We emphasize that Algorithm 1 is based on the distributed training regime and does not require full observability of global states and actions.
Algorithm 1 Primal-Dual Actor-Critic Method with Shadow Reward and  $\kappa$ -hop Policy
1: Input: Initial policy  $\theta^0$  and dual variable  $\mu^0$ ; initial distribution  $\rho$ ; communication radius  $\kappa$ ; step-sizes  $\eta_{\theta}$  and  $\eta_{\mu}$ ; batch size  $B$ ; episode length  $H$ .
2: for iteration  $t = 0,1,2,\ldots$  do
3: Sample  $B$  trajectories with length  $H$  under the  $\kappa$ -hop policy  $\pi_{\theta^t}$  and initial distribution  $\rho$ . Each agent  $i$  collects its neighborhood trajectories  $\tau = \left\{(s_{\mathcal{N}_i^\kappa}^0, a_{\mathcal{N}_i^\kappa}^0), \dots, (s_{\mathcal{N}_i^\kappa}^{H-1}, a_{\mathcal{N}_i^\kappa}^{H-1})\right\}$  as batch  $\mathcal{B}_i^t$ .
4: Each agent  $i$  estimates its local occupancy measure  $\lambda_i^{\pi_\theta t}$  under  $\pi_{\theta^t}$ :
$$
\widetilde {\lambda} _ {i} ^ {t} = \frac {1}{B} \sum_ {\tau \in \mathcal {B} _ {i} ^ {t}} \sum_ {k = 0} ^ {H - 1} \gamma^ {k} \cdot \mathbb {1} _ {i} \left(s _ {i} ^ {k}, a _ {i} ^ {k}\right) \in \mathbb {R} ^ {\left| \mathcal {S} _ {i} \right| \times \left| \mathcal {A} _ {i} \right|}. \tag {17}
$$
5: Each agent  $i$  computes the empirical constraint function value  $\widetilde{g}_i^t = g_i(\widetilde{\lambda}_i^t)$  and empirical shadow rewards  $\widetilde{r}_{f_i}^t = \nabla_{\lambda_i}f_i(\widetilde{\lambda}_i^t)$  and  $\widetilde{r}_{q_i}^t = \nabla_{\lambda_i}g_i(\widetilde{\lambda}_i^t)$ .
6: Each agent  $i$  communicates with its neighborhood  $\mathcal{N}_i^{\kappa}$  and jointly executes an evaluation subroutine to estimate the truncated shadow Q-functions with the empirical shadow rewards  $\widetilde{r}_{\diamond_i}^t$  for  $\diamond \in \{f,g\}$ :
$$
\left(\widetilde {Q} _ {\diamond_ {1}} ^ {t}, \dots , \widetilde {Q} _ {\diamond_ {n}} ^ {t}\right) \leftarrow \operatorname {E v a l} \left(\pi_ {\theta^ {t}}, \left(\widetilde {r} _ {\diamond_ {1}} ^ {t}, \dots , \widetilde {r} _ {\diamond_ {n}} ^ {t}\right)\right). \tag {18}
$$
7: Each agent  $i$  updates the dual variable with the empirical gradient  $\widetilde{\nabla}_{\mu_i}\mathcal{L}(\theta^t,\mu^t) = \widetilde{g}_i^t /n$ :
$$
\mu_ {i} ^ {t + 1} = \mathcal {P} _ {U} \left(- \eta_ {\mu} \widetilde {\nabla} _ {\mu_ {i}} \mathcal {L} \left(\theta^ {t}, \mu^ {t}\right)\right). \tag {19}
$$
8: Each agent  $i$  shares  $\mu_i^{t+1}$  and values of  $\widetilde{Q}_{f_i}^t$ ,  $\widetilde{Q}_{g_i}^t$  along the trajectories in  $\mathcal{B}_i^t$  with agents in  $\mathcal{N}_i^\kappa$  and estimates the truncated policy gradient at  $(\theta^t, \mu^{t+1})$ :
$$
\begin{array}{l} \widetilde {\nabla} _ {\theta_ {i}} \mathcal {L} \left(\theta^ {t}, \mu^ {t + 1}\right) = \frac {1}{B} \sum_ {\tau \in \mathcal {B} _ {i} ^ {t}} \left[ \sum_ {k = 0} ^ {H - 1} \gamma^ {k} \nabla_ {\theta_ {i}} \log \pi_ {\theta_ {i}} ^ {i} \left(a _ {i} ^ {k} \mid s _ {\mathcal {N} _ {i} ^ {\kappa}} ^ {k}\right) \cdot \right. \tag {20} \\ \frac {1}{n} \sum_ {j \in \mathcal {N} _ {i} ^ {\kappa}} \left[ \widetilde {Q} _ {f _ {j}} ^ {t} \left(s _ {\mathcal {N} _ {j} ^ {\kappa}} ^ {k}, a _ {\mathcal {N} _ {j} ^ {\kappa}} ^ {k}\right) + \mu_ {j} ^ {t + 1} \widetilde {Q} _ {g _ {j}} ^ {t} \left(s _ {\mathcal {N} _ {j} ^ {\kappa}} ^ {k}, a _ {\mathcal {N} _ {j} ^ {\kappa}} ^ {k}\right) \right]. \\ \end{array}
$$
9: Each agent  $i$  updates the local policy parameter:
$$
\theta_ {i} ^ {t + 1} = \mathcal {P} _ {\Theta_ {i}} \left(\theta_ {i} ^ {t} + \eta_ {\theta} \cdot \widetilde {\nabla} _ {\theta_ {i}} \mathcal {L} \left(\theta^ {t}, \mu^ {t + 1}\right)\right). \tag {21}
$$
In this section, we analyze the convergence behavior and the sample complexity of Algorithm 1. We begin by summarizing the technical assumptions, including some mentioned previously in the paper. We direct the reader to Appendices F and G where we provide discussions for each assumption and present proofs for the results in this section.
Assumption 4.1. There exists  $L_{\lambda} > 0$  such that  $\nabla_{\lambda_i}f_i(\cdot)$  and  $\nabla_{\lambda_i}g_i(\cdot)$  are  $L_{\lambda}$ -Lipschitz continuous w.r.t.  $\lambda_i$ , i.e.,  $\| \nabla_{\lambda_i}f_i(\lambda_i) - \nabla_{\lambda_i}f_i(\lambda_i')\|_{\infty}\leq L_{\lambda}\| \lambda_i - \lambda_i'\| _2$  and  $\| \nabla_{\lambda_i}g_i(\lambda_i) - \nabla_{\lambda_i}g_i(\lambda_i')\|_{\infty}\leq L_{\lambda}\| \lambda_i - \lambda_i'\| _2$ ,  $\forall i\in \mathcal{N}$ .
Assumption 4.2. The parameterized policy  $\pi_{\theta}$  is such that  $(I)$  the score function is bounded, i.e.,  $\exists M_{\pi} > 0$  s.t.  $\| \nabla_{\theta_i}\log \pi_{\theta_i}^i (a_i|s_{\mathcal{N}_i^\kappa})\| _2\leq M_\pi$ ,  $\forall (s,a)\in S\times \mathcal{A},\theta \in \Theta ,i\in \mathcal{N}$ .  $(II)\exists L_{\theta} > 0$  s.t. the utility functions  $F(\theta) = f(\lambda^{\pi_{\theta}})$  and  $G_{i}(\theta) = g_{i}(\lambda_{i}^{\pi_{\theta}})$  are  $L_{\theta}$ -smooth w.r.t.  $\theta$ ,  $\forall i\in \mathcal{N}$ .
Assumption 4.3. There exist an  $FOSP(\theta^{\star}, \mu^{\star})$  of (5) and a constant  $\overline{\mu} > 0$  s.t.  $\mu_i^{\star} < \overline{\mu}$ ,  $\forall i \in \mathcal{N}$ . Let  $\mathcal{U} = U^n = [0, \overline{\mu}]^n$ .
In Lemma F.5, we summarize a few properties that are the direct consequence consequence of Assumptions 4.1-4.3. Due to the non-concavity of problem (5), our focus is to find an approximate
first-order stationary point (FOSP). A point  $(\theta, \mu) \in \Theta \times \mathcal{U}$  is said to be an  $\epsilon$ -FOSP if
$$
\mathcal {E} (\theta , \mu) := \left[ \mathcal {X} (\theta , \mu) \right] ^ {2} + \left[ \mathcal {Y} (\theta , \mu) \right] ^ {2} \leq \epsilon , \tag {23}
$$
where the metrics  $\mathcal{X}(\cdot ,\cdot)$  and  $\mathcal{V}(\cdot ,\cdot)$  are defined as
$$
\mathcal {X} (\theta , \mu) := \max  _ {\theta^ {\prime} \in \Theta , \| \theta^ {\prime} - \theta \| _ {2} \leq 1} \left\langle \nabla_ {\theta} \mathcal {L} (\theta , \mu), \theta^ {\prime} - \theta \right\rangle , \quad \mathcal {Y} (\theta , \mu) := - \min  _ {\mu^ {\prime} \in \mathcal {U}, \| \mu^ {\prime} - \mu \| _ {2} \leq 1} \left\langle \nabla_ {\mu} \mathcal {L} (\theta , \mu), \mu^ {\prime} - \mu \right\rangle . \tag {24}
$$
The definitions of  $\mathcal{X}(\cdot, \cdot)$  and  $\mathcal{Y}(\cdot, \cdot)$  are based on the first-order optimality condition [45, 46]. Given  $\theta^{\star} \in \Theta$  and  $\mu^{\star} \in \mathcal{U}$ , it can be shown that  $\mathcal{E}(\theta^{\star}, \mu^{\star}) = 0$  implies that  $(\theta^{\star}, \mu^{\star})$  is an FOSP of (5) (see Lemma F.6). In the following, we first consider the exact setting where the agents can obtain the true values of their local occupancy measures, shadow Q-functions, and truncated policy gradients. Therefore, the only source of approximation error is the truncation of the policy gradient.
Theorem 4.4 (Exact setting). Let Assumptions 3.2, 3.3, 4.1-4.3 hold and suppose that the agents can accurately estimate their local occupancy measures, shadow  $Q$ -functions, and truncated policy gradients. For every  $T > 0$ , let  $\left\{\left(\mu^t,\theta^t\right)\right\}_{t = 0}^T$  be the sequence generated by Algorithm 1 with  $\eta_{\mu} = \mathcal{O}\left(T^{1 / 3}\right)$  and  $\eta_{\theta} = 1 / \left(L_{\theta \theta} + 4L_{\theta \mu}^{2}\eta_{\mu}\right)$ , where  $L_{\theta \theta}, L_{\theta \mu}$  are Lipschitz constants defined in Lemma F.5. Then, there exists  $t^\star \in \{0,1,\dots,T - 1\}$  such that
$$
\mathcal {E} \left(\theta^ {t ^ {\star}}, \mu^ {t ^ {\star} + 1}\right) = \mathcal {O} \left(T ^ {- 2 / 3}\right) + \mathcal {O} \left(\phi_ {0} ^ {2 \kappa}\right). \tag {25}
$$
Next, we delve into the sample complexity of Algorithm 1. For theoretical analysis, we assume that the estimation process for the truncated Q-function offers an approximation to the true function, with the error being associated with the magnitude of the reward function. Let  $\widehat{Q}_i^{\pi_\theta}(r_i;\cdot ,\cdot)\in \mathbb{R}^{|\mathcal{S}_{\mathcal{N}_i^\kappa}| \times |\mathcal{A}_{\mathcal{N}_i^\kappa}|}$  be the truncated Q-function with the reward function  $r_i(\cdot ,\cdot)\in \mathbb{R}^{|\mathcal{S}_i|\times |\mathcal{A}_i|}$  for agent  $i\in \mathcal{N}$ .
Assumption 4.5. For every reward function  $r_i(\cdot, \cdot)$  and  $\epsilon_0 > 0$ , the subroutine computes an approximation  $\widehat{Q}_i^{\pi_\theta}(r_i; \cdot, \cdot)$  to the truncated  $Q$ -function  $\widehat{Q}_i^{\pi_\theta}(r_i; \cdot, \cdot)$  such that
$$
\left\| \widetilde {Q} _ {i} ^ {\pi_ {\theta}} \left(r _ {i}; \cdot , \cdot\right) - \widehat {Q} _ {i} ^ {\pi_ {\theta}} \left(r _ {i}; \cdot , \cdot\right) \right\| _ {\infty} \leq \| r _ {i} \| _ {\infty} \epsilon_ {0} \tag {26}
$$
with  $\mathcal{O}(1 / (\epsilon_0)^2)$  samples, for every  $i\in \mathcal{N},\theta \in \Theta$
We comment that the sample complexity of the truncated Q-function evaluation described in Assumption 4.5 is not restrictive. It can be achieved with high probability by the TD-learning procedure outlined in Algorithm 2 when the agents have enough exploration [10, 43]. For brevity, we assume that (26) holds almost surely. The only difference in the probabilistic version would be the presence of an additional term for the failure probability, which does not affect the order of the sample complexity.
Theorem 4.6 (Sample-based setting). Suppose that Assumptions 3.2, 3.3, 4.1-4.3, and 4.5 hold. For every  $\epsilon > 0$  and  $\delta \in (0,1)$ , let  $\left\{\left(\mu^t,\theta^t\right)\right\}_{t=0}^T$  be the sequence generated by Algorithm 1 with  $T = \mathcal{O}\left(\epsilon^{-1.5}\right)$ ,  $\eta_\mu = \mathcal{O}\left(\epsilon^{-0.5}\right)$ ,  $\eta_\theta = 1 / \left(L_{\theta \theta} + 4L_{\theta \mu}^2\eta_\mu\right)$ ,  $\epsilon_0 = \mathcal{O}\left(\sqrt{\epsilon}\right)$ ,  $\delta_0 = \delta /(2n(T + 1))$ , batch size  $B = \mathcal{O}\left(\log (1 / \delta_0)\epsilon^{-2}\right)$ , episode length  $H = \log (1 / \epsilon)$ , where  $L_{\theta \theta}, L_{\theta \mu}$  are Lipschitz constants defined in Lemma F.5. Then, with probability  $1 - \delta$ , there exists  $t^\star \in \{0,1,\ldots,T - 1\}$  such that
$$
\mathcal {E} \left(\theta^ {t ^ {\star}}, \mu^ {t ^ {\star} + 1}\right) = \mathcal {O} (\epsilon) + \mathcal {O} \left(\phi_ {0} ^ {2 \kappa}\right). \tag {27}
$$
The required number of samples is  $\widetilde{\mathcal{O}}\big(\epsilon^{-3.5}\big)$
4.1 Technical discussions
Theorem 4.4 implies an  $\mathcal{O}\left(T^{-2/3}\right)$  iteration complexity of Algorithm 1, matching the fastest convergence rate for solving nonconcave-convex maximin problems in the literature [47]. The approximation error  $\mathcal{O}\left(\phi_0^{2\kappa}\right)$  decays at a linear rate w.r.t. the radius of communications. Thus, as long as the underlying network is not densely connected, such as those in wireless communication [37] and autonomous driving [48], an approximate FOSP to (5) can be efficiently computed, while each agent  $i$  only needs to communicate with a small number of agents in its neighborhood.
In Theorem 4.4, we have chosen large step-sizes for the dual variable update to achieve the best convergence rate. This aggressive update ensures that the dual metric  $\mathcal{V}(\theta^t,\mu^{t + 1})$  always remains
within a small range and also provides a satisfactory ascent direction for the policy update. Then, the average primal metric  $1 / T\cdot \sum_{t = 0}^{T - 1}\left[\mathcal{X}\left(\theta^t,\mu^{t + 1}\right)\right]^2$  is upper-bounded by exploiting a recursive relation between any two consecutive dual updates. Hence, the existence of a point  $\left(\theta^{t^{\star}},\mu^{t^{\star} + 1}\right)$  that satisfies (25) is guaranteed. It is worth noting that the proof of Theorem 4.4 can be easily generalized to the scenario where  $T$  is unspecified, and the same convergence rate can still be achieved with adaptive step-sizes  $\eta_{\mu}^{t} = \mathcal{O}\left(t^{1 / 3}\right)$  and  $\eta_{\theta}^{t} = 1 / \bigl (L_{\theta \theta} + 4L_{\theta \mu}^{2}\eta_{\mu}^{t}\bigr)$ .
Theorem 4.6 states that, with high probability, Algorithm 1 has an  $\widetilde{\mathcal{O}}\left(\epsilon^{-3.5}\right)$  sample complexity for finding an  $\epsilon$ -FOSP of (5) with an approximation error  $\mathcal{O}(\phi_0^{2\kappa})$ . Note that we absorb the logarithmic terms in the notation  $\widetilde{\mathcal{O}}(\cdot)$ . The proof of Theorem 4.6 can be broken down into two parts. Firstly, we evaluate the approximation errors of the estimators used in Algorithm 1 in relation to the model parameters, as outlined in Proposition G.1. Then, we integrate these errors into the iteration complexity result established in Theorem 4.4 and optimize the selection of parameters.
In this section, we validate Algorithm 1 via numerical experiments, focusing on three key questions<sup>1</sup>:
- How does Algorithm 1 perform with multiple agents, and does the policy gradient truncation effectively alleviate computational load?
- While Algorithm 1 is the first approach that provably solves the safe MARL problem with general utilities, how does it compare with existing methods for standard Safe MARL?
- What benefits does the use of general utilities offer over standard cumulative rewards?
To answer these questions, we performed multiple experiments in three environments<sup>2</sup>. The objective functions are based on cumulative rewards, while constraint functions leverage general utilities to incentivize or dissuade agents from exploring the environments.
Synthetic environment Analogous to [24, Section 5.1], where agents are linearly arranged as  $1 - 2 - \dots - n$ . Each agent  $i$  has binary local state and action spaces, i.e.,  $S_{i} = \mathcal{A}_{i} = \{0,1\}$ , and the local transition matrix  $\mathbb{P}_i$  depends solely on its action  $a_i$  and the state of agent  $i + 1$ . The reward functions are constructed such that the optimal unconstrained policy compels all agents to continuously choose action 1, irrespective of their states.
Pistonball A physics-based game that emphasizes cooperations and high-dimensional states as illustrated in Figure 1a. Each piston represents an agent, where its local neighborhood includes adjacent pistons, and the goal is to collectively move the ball from right to left. The agent can move up, down, or remain still. We modify the original game[49] so that the agent can only observe the ball when it enters the local neighborhood, as well as the height of neighboring pistons.
Wireless communication An access control problem following a similar setup as in [24, 50]. As illustrated in Figure 1b, the agents try to transmit packets to common access points, and the transmission fails if the access point receives more than one packet simultaneously. As there are more agents than access points, some agents need to learn to forego their benefits for the collective good.
In addition to the objective, we incorporate two types of safety constraints characterized by general utilities that cannot be easily encapsulated by standard value functions based on cumulative rewards.
- Entropy constraints that stimulates exploration, formalized as  $\mathrm{Entropy}(\lambda_i^{\pi_\theta})\geq c,\forall i\in \mathcal{N}$ . The function  $\mathrm{Entropy}(\lambda_i^{\pi_\theta})$  represents the local entropy, defined as  $-\sum_{s\in \mathcal{S}}d_i^\pi (s)\cdot \log \left(d_i^\pi (s)\right)$ , where  $d_{i}^{\pi \theta}(s_{i}) = (1 - \gamma)\sum_{a_{i}\in \mathcal{A}_{i}}\lambda_{i}^{\pi \theta}(s_{i},a_{i})$  is the local state occupancy measure.
-  $\ell_{2}$ -constrains that deter agents from learning overly randomized policies, formulated as  $\left\| \sum_{s_i \in S_i} \lambda_i^{\pi_\theta} \right\|_2^2 \geq c$ ,  $\forall i \in \mathcal{N}$ . This constraint is beneficial in applications like autonomous driving and human-AI collaboration, where an agent's policy needs to be predictable for other agents.
$^{1}$ Code is available here: https://github.com/zhykoties/Decentralized-Safe-MARL-with-General-Utilities.
See Appendix H for detailed descriptions and complete experimental results.
(a) Pistonball
(b) Wireless comm.
(c) Episodic return
Figure 1: (a,b) Environment illustration. (c,d) Performance of Algorithm 1 in Pistonball with 20 agents under entropy constraints.
(d) Constraint violation
Table 1: Comparison between Scalable Primal-Dual Actor-Critic method in our work with MAPPO-L by [31] in Pistonball and wireless communication.
Pistonball	Wireless Communication
Algorithm	Episodic return	Const. vio.	Episodic return	Const. vio.
Ours	51.788 ± 1.346	0.04919	3.373 ± 0.112	0.1926
MAPPO-L	50.612 ± 2.118	0.06884	3.347 ± 0.131	0.4000
Decen. Agg. MAPPO-L	48.197 ± 6.188	0.2179	3.106 ± 0.673	1.1890
Decen. MAPPO-L	41.102 ± 18.769	0.09303	3.148 ± 0.614	1.5760