Title: Scalable Primal-Dual Actor-Critic Method for Safe Multi-Agent RL with General Utilities
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement Learning Theory

Safe reinforcement learning

Multi-agent reinforcement learning
Abstract: We investigate safe multi-agent reinforcement learning, where agents seek to collectively maximize an aggregate sum of local objectives while satisfying their own safety constraints. The objective and constraints are described by general utilities, i.e., nonlinear functions of the long-term state-action occupancy measure, which encompass broader decision-making goals such as risk, exploration, or imitations. The exponential growth of the state-action space size with the number of agents presents challenges for global observability, further exacerbated by the global coupling arising from agents' safety constraints. To tackle this issue, we propose a primal-dual method utilizing shadow reward and $\kappa$-hop neighbor truncation under a form of correlation decay property, where $\kappa$ is the communication radius. In the exact setting, our algorithm converges to a first-order stationary point (FOSP) at the rate of $\mathcal{O}\left(T^{-2/3}\right)$. In the sample-based setting, we demonstrate that, with high probability, our algorithm requires $\widetilde{\mathcal{O}}\left(\epsilon^{-3.5}\right)$ samples to achieve an $\epsilon$-FOSP with an approximation error of $\mathcal{O}(\phi_0^{2\kappa})$, where $\phi_0\in (0,1)$. Finally, we demonstrate the effectiveness of our model through extensive numerical experiments.

Scalable Primal-Dual Actor-Critic Method for Safe Multi-Agent RL with General Utilities

Abstract

We investigate safe multi-agent reinforcement learning, where agents seek to collectively maximize an aggregate sum of local objectives while satisfying their own safety constraints. The objective and constraints are described by general utilities , i.e., nonlinear functions of the long-term state-action occupancy measure, which encompass broader decision-making goals such as risk, exploration, or imitations. The exponential growth of the state-action space size with the number of agents presents challenges for global observability, further exacerbated by the global coupling arising from agents' safety constraints. To tackle this issue, we propose a primal-dual method utilizing shadow reward and κ -hop neighbor truncation under a form of correlation decay property, where κ is the communication radius. In the exact setting, our algorithm converges to a first-order stationary point (FOSP) at the rate of O( T -2 / 3 ) . In the sample-based setting, we demonstrate that, with high probability, our algorithm requires ̃ O( ϵ -3 . 5 ) samples to achieve an ϵ -FOSP with an approximation error of O( ϕ 2 κ 0 ) , where ϕ 0 ∈ ( 0 , 1 ) . Finally, we demonstrate the effectiveness of our model through extensive numerical experiments.

1 Introduction

Cooperative multi-agent reinforcement learning (MARL) involves agents operating within a shared environment, where each agent's decisions influence not only their objectives, but also those of others and the state trajectories [1]. In seeking to bring conceptually sound MARL techniques out of simulation [2, 3] and into real-world environments [4, 5], some key issues emerge: safety and communications overhead implied by a training mechanism. Although experimentally, the centralized training decentralized execution (CTDE) framework has gained traction recently [6, 7], its requirement for centralized data collection can pose issues for large-scale [8] or privacy-sensitive applications [9]. Therefore, we prioritize decentralized training, where to date most MARL techniques impose global state observability for performance certification [1]. In this work, we extend recent efforts to alleviate this bottleneck [10] especially in the case of safety critical settings, in a flexible manner that allows agents to incorporate risk, exploration, or prior information.

More specifically, we hypothesize that the multi-agent system consists of a network of agents that interact with each other locally according to an underlying dependence graph [10]. Second, to model safety constraints in reinforcement learning (RL), we adopt a standard approach based on constrained

Markov Decision Processes (CMDPs) [11], where one maximizes the expected total reward subject to a safety-related constraint on the expected total utility. Third, since many decision-making problems take a form beyond the classic cumulative reward, such as apprenticeship learning [12], diverse skill discovery [13], pure exploration [14], and state marginal matching [15], we focus on utility functions defined as nonlinear functions of the induced state-action occupancy measure, which can be abstracted as RL with general utilities [16, 17].

Towards formalizing the approach, we consider an MARL model consisting of n agents, each with its own local state s i and action a i , where the multi-agent system is associated with an underlying dependence graph G . Each agent is privately associated with two local general utilities f i (⋅) and g i (⋅) , where f i (⋅) and g i (⋅) are functions of the local occupancy measure. The objective is to find a safe policy for each agent that maximizes the average of the local objective utilities, namely, 1 / n ⋅ ∑ n i = 1 f i (⋅) , and satisfies each agent's individual safety constraint described by its local utility g i (⋅) . This setting captures a wide range of safety-critical applications, for example, resource allocation for the control of networked epidemic models [18], influence maximization in social networks [19], portfolio optimization in interbank network structures [20], intersection management for connected vehicles [21], and energy constraints of wireless communication networks [22].

Despite the significance of safe MARL with general utilities, prior works have either ignored the necessity of safety [23] or the computational bottleneck associated with global information exchange regarding the state and action per step [24]. In fact, the interaction of these two aspects requires addressing the fact that each agent's own safety constraint requires information from all others. In particular, the existing works in safe MARL allow full access to the global state or unlimited communications among all agents for policy implementation, value estimation, and constraint satisfaction [25, 26, 27]. However, this assumption is impractical due to the 'curse of dimensionality' [28], as well as the limited information exchanges and communications among agents [29].

Therefore, to our knowledge, there is no methodology to both guarantee safety and incur manageable communications overhead for each agent. Compounding these issues is the fact that standard RL training schemes based on the policy gradient theorem [30] are not applicable in the context of general utilities. This deviation from the cumulative rewards adds to the difficulty of estimating the gradient, since there does not exist a policy-independent reward function. We refer the reader to Appendix A for an extended discussion of related works.

To address these challenges, we focus on the setting of distributed training without global observability and aim to develop a scalable algorithm with theoretical guarantees. Our main contributions are summarized below:

Compared with existing theoretical works on safe MARL [25, 26, 31], we present the first safe MARL formulation that extends beyond cumulative forms in both the objective and constraints. We develop a truncated policy gradient estimator utilizing shadow reward and κ -hop policies under a form of correlation decay property, where κ represents the communication radius. The approximation errors arising from both policy implementation and value estimation are quantified.

Despite of the global coupling of agents' local utility functions, we propose a scalable Primal-Dual Actor-Critic method, which allows each agent to update its policy based only on the states and actions of its close neighbors and under limited communications. The effectiveness of the proposed algorithm is verified through numerical experiments.

From the perspective of optimization, we devise new tools to analyze the convergence of the algorithm. In the exact setting, we establish an O( T -2 / 3 ) convergence rate for finding an FOSP, matching the standard convergence rate for solving nonconcave-convex saddle point problems. In the sample-based setting, we prove that, with high probability, the algorithm requires ̃ O( ϵ -3 . 5 ) samples to obtain an ϵ -FOSP with an approximation error of O( ϕ 2 κ 0 ) , where ϕ 0 ∈ ( 0 , 1 ) .

2 Problem formulation

Consider a Constrained Markov Decision Process (CMDP) over a finite state space S and a finite action space A with a discount factor γ ∈ [ 0 , 1 ) . A policy π is a function that specifies the decision rule of the agent, i.e., the agent takes action a ∈ A with probability π ( a ∣ s ) in state s ∈ S . When action a is taken, the transition to the next state s ′ from state s follows the probability distribution

s ′ ∼ P (⋅∣ s, a ) . Let ρ be the initial distribution. For each policy π and state-action pair ( s, a ) ∈ S × A , the discounted state-action occupancy measure is defined as

The goal of the agent is to find a policy π that maximizes a general objective described by a (possibly) nonlinear function f (⋅) of λ π , known as the general utility , subject to a constraint in the form of another general utility g (⋅) , namely

When f (⋅) = ⟨ r, ⋅⟩ and g (⋅) = ⟨ u, ⋅⟩ are linear functions, (2) recovers the standard CMDP problem:

where V π (⋅) is usually referred to as the value function . In contrast, it has been shown that for some MDPs, there is no standard value function that can be equivalent to the general utility [16, Lemma 1]. In Appendix C, we provide more examples of formulation (2) beyond standard value functions.

In this work, we study the decentralized version of problem (2). Consider the system is composed of a network of agents associated with a graph G = (N , E G ) (not densely connected in general), where the vertex set N = { 1 , 2 , . . . , n } denotes the set of n agents and the edge set E G prescribes the communication links among the agents. Let d ( i, j ) be the length of the shortest path between agents i and j on G . For κ ≥ 0 , let N κ i = { j ∈ N∣ d ( i, j ) ≤ κ } denote the set of agents in the κ -hop neighborhood of agent i , with the shorthand notation N κ -i ∶= N /N κ i and -i ∶= N /{ i } . The details of the decentralized nature of the system are summarized below:

Space decomposition The global state and action spaces are the product of local spaces, i.e., S = S 1 ×S 2 ×/uni22EF×S n , A=A 1 ×A 2 ×/uni22EF×A n , meaning that for every s ∈ S and a ∈ A , we can write s = ( s 1 , s 2 , . . . , s n ) and a = ( a 1 , a 2 , . . . , a n ) . For each subset N ′ ⊂ N , we use ( s N ′ , a N ′ ) to denote the state-action pair for the agents in N ′ .

Observation and communication Each agent i only has direct access to its own state s i and action a i , while being allowed to communicate with its κ -hop neighborhood N κ i for information exchanges. The communication radius κ is a given but tunable parameter.

Transition decomposition Given the current global state s and action a , the local states in the next period are independently generated, i.e., P ( s ′ ∣ s, a ) = ∏ i ∈N P i ( s ′ i ∣ s, a ) , ∀ s ′ ∈ S , where we use P i to denote the local transition probability for agent i .

Policy factorization The global policy can be expressed as the product of local policies, such that π ( a ∣ s ) = ∏ i ∈N π i ( a i ∣ s ) , ∀( s, a ) , i.e., given the global state s , each agent i acts independently based on its local policy π i . We assume that each local policy π i is parameterized by a parameter θ i within a convex set Θ i . Thus, we can write π ( a ∣ s ) = π θ ( a ∣ s ) = ∏ i ∈N π i θ i ( a i ∣ s ) , where θ ∈ Θ = Θ 1 × Θ 2 ×/uni22EF× Θ n is the concatenation of local parameters.

Localized objective and constraint For each agent i and its local state-action pair ( s i , a i ) , the local state-action occupancy measure under policy π is defined as

which can be viewed as the marginalization of the global occupancy measure, i.e., λ π i ( s i , a i ) = ∑ s -i ,a -i λ π ( s, a ) . Each agent i is privately associated with two local (general) utilities f i (⋅) and g i (⋅) , which are functions of the local occupancy measure λ π i . Agents cooperate with each other aiming at maximizing the global objective f (⋅) , defined as the average of local utilities { f i (⋅)} i ∈N , while each agent i needs to satisfy its own safety constraint described by the local utility g i (⋅) . Then, under the parameterization π θ , (2) can be rewritten as

Note that problem (5) is not separable among agents due to the coupling of occupancy measures. Compared to the formulation where the constraint is modeled as the average of local constraints, e.g.,

[27], (5) is stricter and more interpretable. We emphasize that the method proposed in this paper does not require the relaxation of local constraints in (5) to a joint constraint and it directly generalizes to the case of multiple constraints per agent.

Consider the Lagrangian function associated with (5):

where µ ∈ R n + is the Lagrangian multiplier. The Lagrangian formulation [32] of (5) can be written as

Since the general utilities f i ( λ π θ i ) and g i ( λ π θ i ) may not be non-concave w.r.t. θ even in the form of cumulative rewards, finding the global optimum to (5) is NP-hard in general [33]. Our goal in this work is to develop a scalable and provably efficient gradient-based primal-dual algorithm that can find the first-order stationary points of (5).

3 Scalable primal-dual actor-critic method

For a standard value function with the reward r ∈ R ∣S∣×∣A∣ , denoted as V π θ ( r ) = ⟨ r, λ π θ ⟩ , the policy gradient theorem (see Lemma D.1) yields that

where d π θ ( s ) ∶= ( 1 -γ ) ∑ a ∈A λ π θ ( s, a ) is the discounted state occupancy measure, ∇ θ log π θ (⋅∣⋅) is the score function, and Q π θ ( r ; ⋅ , ⋅) is the Q-function with the reward r , defined as

Although this elegant result no longer holds for general utilities, we can apply the chain rule:

i.e., the gradient ∇ θ f ( λ π θ ) is equal to the policy gradient of a standard value function with the reward ∇ λ f ( λ π θ ) . We introduce the following definitions [23] for the distributed problem (5).

Definition 3.1 (Shadow reward and shadow Q-function) . For each agent i , define r π θ f i ∶= ∇ λ i f i ( λ π θ i ) ∈ R ∣S i ∣×∣A i ∣ as the (local) shadow reward for the utility f i (⋅) under policy π θ . Define Q π θ f i ( s, a ) ∶= Q π θ ( r π θ f i ; s, a ) as the associated (local) shadow Q-function for f i (⋅) . Similarly, let r π θ g i and Q π θ g i ( s, a ) be the shadow reward and the Q function for g i (⋅) .

Combining Definition 3.1 with (9), we can write the local gradient for agent i , i.e., ∇ θ i L( θ, µ ) , as

where we apply the policy factorization to arrive at ∇ θ i log π θ ( a ∣ s ) = ∇ θ i log π i θ i ( a i ∣ s ) . By (10), each agent needs to know the shadow Q functions of all agents, as well as the global state, to evaluate its own gradient. However, especially in large networks, this is both inefficient, due to the communication cost, and impractical because of the limited communication radius. In the remainder of this section, we aim to design a scalable estimator for ∇ θ i L( θ, µ ) that requires only local communications.

3.1 Spatial correlation decay and κ -hop policies

Inspired by [34], we assume that the transition probability satisfies a form of the spatial correlation decay property [35, 36].

Assumption 3.2. For a matrix M ∈ R n × n whose ( i, j ) -th entry is defined as

assume that there exists ω > 0 such that max i ∈N ∑ j ∈N e ωd ( i,j ) M ij ≤ χ with χ < 2 / γ , where γ is the discount factor.

The value of M ij reflects the extent to which agent j 's state and action influence the local transition probability of agent i . Thus, Assumption 3.2 amounts to requiring this influence to decrease exponentially with the distance between any two agents. Such a decay is often observed in many large-scale real-world systems, e.g., the strength of signals decreases exponentially with distance [37].

Furthermore, as mentioned earlier, the implementation of the local policy π i θ i (⋅∣ s ) is still impractical, since it requires access to the global state s , while the allowable communication radius is limited to κ . To alleviate this issue, we focus on a specific class of policies in which the local policy of agent i only depends on the states of these agents in its κ -hop neighborhood N κ i . This class of policies is also referred to as κ -hop policies in the concurrent work [38].

Assumption 3.3 ( κ -hop policies) . For each agent i ∈ N and θ ∈ Θ , the local policy π i θ i (⋅∣ s ) depends only on the neighbor states s N κ i , i.e.,

For simplicity, we use the notation π i θ i (⋅∣ s ) = π i θ i (⋅∣ s N κ i ) for κ -hop policies when it is clear from context. We note that, for any original policy function π θ (⋅∣ s ) , an induced κ -hop policy ˆ π θ (⋅∣ s N κ i ) can be defined by fixing the states s N κ -i to some arbitrary values and focusing only on the states of agents in N κ i . When considering only κ -hop policies, it is essential to understand how much information is lost compared to the case where agents have access to the global states. The following proposition quantifies the maximum information loss in terms of the occupancy measure under the assumption that the original policy function also satisfies a spatial correlation decay property.

Proposition 3.4. Suppose that there exist c ≥ 0 and ϕ ∈ [ 0 , 1 ) such that for every θ ∈ Θ , agent i ∈ N , and states s, s ′ ∈ S such that s N κ i = s ′ N κ i , we have ∥ π i θ i (⋅∣ s ) -π i θ i (⋅∣ s ′ )∥ 1 ≤ cϕ κ . Let ˆ π θ be an induced κ -hop policy of π θ . Then, it holds that

The condition on the local policy in Proposition 3.4 encodes that every π i θ i is exponentially less sensitive to the states of agents outside N κ i , which is a common assumption in MARL to alleviate computationally burdensome and practically intractable communication requirements imposed by the global observability [34, 39, 38]. By Proposition 3.4, the difference in occupancy measures under π θ and ˆ π θ is controlled by ∥ π i θ i -ˆ π i θ i ∥ 1 . Therefore, if f i ( λ π ) and g i ( λ π ) are Lipschitz continuous w.r.t. λ π , Proposition 3.4 implies an O( ϕ κ ) approximation of the Lagrangian function (6) using κ -hop policies. The faster the spatial decay of policy is, the more accurate the approximation of the κ -hop policy is. This justifies our focus on learning a κ -hop policy.

3.2 Truncated policy gradient estimator

In the absence of global observability, it is critical to find a scalable estimator for the local gradient ∇ θ i L( θ, µ ) in (10), so that each agent can update its local policy with limited communications.

By leveraging the similar idea in the definition of κ -hop policies, we define the κ -hop truncated (shadow) Q-function , denoted as ̂ Q π θ ♢ i ∶ S N κ i ×A N κ i → R , to be

where ( ¯ s N κ -i , ¯ a N κ -i ) is any fixed state-action pair for the agents in N κ -i . Now, we introduce the following truncated policy gradient estimator for agent i :

In comparison to the true policy gradient (10), ̂ ∇ θ i L( θ, µ ) replaces the shadow Q-functions with their truncated versions and only considers the agents in the κ -hop neighborhood N κ i . Surprisingly, the following lemma shows that the approximation error of ̂ ∇ θ i L( θ, µ ) decreases exponentially with κ when the shadow rewards and the score functions are bounded.

Lemma 3.5. Suppose that Assumptions 3.2 and 3.3 hold and there exist M r , M π > 0 such that ∥ r π θ ♢ i ∥ ∞ ≤ M r and ∥∇ θ i log π i θ i ∥ 2 ≤ M π , for every ♢ ∈ { f, g } , θ ∈ Θ , i ∈ N . Then, for all θ ∈ Θ , i ∈ N , we have that

where c 0 = 2 γχM r /slash.left.alt1 ( 2 -γχ ) and ϕ 0 = e -ω .

Recall that the shadow reward is defined as the gradient of f i (⋅) or g i (⋅) w.r.t. the local occupancy measure. Since the set of all possible occupancy measures is compact (see (43)), the existence of M r > 0 in Lemma 3.5 is satisfied if f i (⋅) and g i (⋅) are continuously differentiable. The main advantage of using the estimator ̂ ∇ θ i L( θ, µ ) lies in that every agent i only needs to know the truncated Q-functions of agents in its neighborhood N κ i , which can significantly reduce the communication burden and the storage requirement when graph G is not densely connected. The proof of Lemma 3.5 can be found in Appendix E.2.

3.3 Algorithm design

Using the results of the preceding section, we put together all the pieces and propose the Primal-Dual Actor-Critic Method with Shadow Reward and κ -hop Policy , as outlined in Algorithm 1. It includes three stages: policy evaluation by the critic, Lagrangian multiplier update, and policy update by the actor. Below, we provide an overview of Algorithm 1, while referring the reader to Appendix D for a flow diagram (Figure 2) of the algorithm, as well as a more detailed discussion.

Stage 1 (policy evaluation by the critic, lines 3-6) In each iteration t , the current policy π θ t is simulated to generate a batch of trajectories, while each agent i collects its neighborhood trajectories, i.e., the state-action pairs of the agents in N κ i , as batch B t i . Then, the batch is used to estimate the local occupancy measures λ π θ t i through (17), which are subsequently applied to compute the empirical values for the constraint function g i ( λ π θ t i ) and shadow rewards r π θ t f i and r π θ t g i , denoted as ̃ g t i , ̃ r t f i , and ̃ r t g i , respectively. It is worth mentioning that, when all utility functions reduce to the form of cumulative rewards, the above operation is unnecessary, since all agents have policy-independent local reward functions.

Next, the agents jointly conduct a distributed evaluation subroutine to estimate their truncated shadow Q-functions { ̂ Q π θ t ♢ i } i ∈N using empirical shadow rewards {̃ r t ♢ i } i ∈N , where ♢ ∈ { f, g } . During the subroutine, each agent i communicates with its neighbor in N κ i to exchange state-action information, but only needs to access its own empirical shadow reward ̃ r t ♢ i . In principle, any existing approach that satisfies the observation and communication requirements can be used for the truncated Q-function estimation, such as [40, 41, 42]. As an example subroutine, we introduce the Temporal Difference (TD) learning method [43], which is outlined as Algorithm 2 in Appendix D.

Stage 2 (Lagrangian multiplier update, line 7) Instead of employing the projected gradient descent, we propose to update the dual variables by the following formula:

where weight η µ can be viewed as the dual 'step-size'. In practice, we replace the true dual gradient ∇ µ i L( θ t , µ t ) = g i ( λ π θ t i )/ n with its empirical estimator ̃ ∇ µ i L( θ t , µ t ) . The feasible region for the dual variable is denoted by U ⊆ R n + and will be specified later.

Stage 3 (policy update by the actor, lines 8-9) To perform the policy update, each agent i first shares its updated dual variable µ t + 1 i and the values of its estimated truncated Q-functions along the trajectories in batch B t i with the agents in its κ -hop neighborhood N κ i . Then, the agent estimates its truncated policy gradient ̂ ∇ θ i L( θ t , µ t + 1 ) through a REINFORCE-based mechanism [44] as described in (20). Finally, each agent i updates its local policy parameter by a projected gradient ascent.

We emphasize that Algorithm 1 is based on the distributed training regime and does not require full observability of global states and actions.

Algorithm 1 Primal-Dual Actor-Critic Method with Shadow Reward and κ -hop Policy

1: Input: Initial policy θ 0 and dual variable µ 0 ; initial distribution ρ ; communication radius κ ; step-sizes η θ and η µ ; batch size B ; episode length H .

2: for iteration t = 0 , 1 , 2 , . . . do

3: Sample B trajectories with length H under the κ -hop policy π θ t and initial distribution ρ . Each agent i collects its neighborhood trajectories τ = {( s 0 N κ i , a 0 N κ i ) , /uni22EF , ( s H -1 N κ i , a H -1 N κ i )} as batch B t i .

4: Each agent i estimates its local occupancy measure λ π θ t i under π θ t :

6: Each agent i communicates with its neighborhood N κ i and jointly executes an evaluation subroutine to estimate the truncated shadow Q-functions with the empirical shadow rewards ̃ r t ♢ i for ♢ ∈ { f, g } :

5: Each agent i computes the empirical constraint function value ̃ g t i = g i ( ̃ λ t i ) and empirical shadow rewards ̃ r t f i = ∇ λ i f i ( ̃ λ t i ) and ̃ r t g i = ∇ λ i g i ( ̃ λ t i ) .

7: Each agent i updates the dual variable with the empirical gradient ̃ ∇ µ i L( θ t , µ t ) = ̃ g t i / n :

8: Each agent i shares µ t + 1 i and values of ̃ Q t f i , ̃ Q t g i along the trajectories in B t i with agents in N κ i and estimates the truncated policy gradient at ( θ t , µ t + 1 ) :

9: Each agent i updates the local policy parameter:

10: end for

4 Convergence analysis

In this section, we analyze the convergence behavior and the sample complexity of Algorithm 1. We begin by summarizing the technical assumptions, including some mentioned previously in the paper. We direct the reader to Appendices F and G where we provide discussions for each assumption and present proofs for the results in this section.

Assumption 4.1. There exists L λ > 0 such that ∇ λ i f i (⋅) and ∇ λ i g i (⋅) are L λ -Lipschitz continuous w.r.t. λ i , i.e., ∥∇ λ i f i ( λ i )-∇ λ i f i ( λ ′ i )∥ ∞ ≤ L λ ∥ λ i -λ ′ i ∥ 2 and ∥∇ λ i g i ( λ i )-∇ λ i g i ( λ ′ i )∥ ∞ ≤ L λ ∥ λ i -λ ′ i ∥ 2 , ∀ i ∈ N .

Assumption 4.2. The parameterized policy π θ is such that (I) the score function is bounded, i.e., ∃ M π > 0 s.t. ∥∇ θ i log π i θ i ( a i ∣ s N κ i )∥ 2 ≤ M π , ∀( s, a ) ∈ S × A , θ ∈ Θ , i ∈ N . (II) ∃ L θ > 0 s.t. the utility functions F ( θ ) = f ( λ π θ ) and G i ( θ ) = g i ( λ π θ i ) are L θ -smooth w.r.t. θ , ∀ i ∈ N .

Assumption 4.3. There exist an FOSP ( θ ⋆ , µ ⋆ ) of (5) and a constant µ > 0 s.t. µ ⋆ i < µ , ∀ i ∈ N . Let U = U n = [ 0 , µ ] n .

In Lemma F.5, we summarize a few properties that are the direct consequence consequence of Assumptions 4.1-4.3. Due to the non-concavity of problem (5), our focus is to find an approximate

first-order stationary point (FOSP). A point ( θ, µ ) ∈ Θ ×U is said to be an ϵ -FOSP if

where the metrics X(⋅ , ⋅) and Y(⋅ , ⋅) are defined as

The definitions of X(⋅ , ⋅) and Y(⋅ , ⋅) are based on the first-order optimality condition [45, 46]. Given θ ⋆ ∈ Θ and µ ⋆ ∈ U , it can be shown that E( θ ⋆ , µ ⋆ ) = 0 implies that ( θ ⋆ , µ ⋆ ) is an FOSP of (5) (see Lemma F.6). In the following, we first consider the exact setting where the agents can obtain the true values of their local occupancy measures, shadow Q-functions, and truncated policy gradients. Therefore, the only source of approximation error is the truncation of the policy gradient.

Theorem 4.4 (Exact setting) . Let Assumptions 3.2, 3.3, 4.1-4.3 hold and suppose that the agents can accurately estimate their local occupancy measures, shadow Q-functions, and truncated policy gradients. For every T > 0 , let {( µ t , θ t )} T t = 0 be the sequence generated by Algorithm 1 with η µ = O( T 1 / 3 ) and η θ = 1 /slash.left.alt1( L θθ + 4 L 2 θµ η µ ) , where L θθ , L θµ are Lipschitz constants defined in Lemma F.5. Then, there exists t ⋆ ∈ { 0 , 1 , . . . , T -1 } such that

Next, we delve into the sample complexity of Algorithm 1. For theoretical analysis, we assume that the estimation process for the truncated Q-function offers an approximation to the true function, with the error being associated with the magnitude of the reward function. Let ̂ Q π θ i ( r i ; ⋅ , ⋅) ∈ R ∣S N κ i ∣×∣A N κ i ∣ be the truncated Q-function with the reward function r i (⋅ , ⋅) ∈ R ∣S i ∣×∣A i ∣ for agent i ∈ N .

Assumption 4.5. For every reward function r i (⋅ , ⋅) and ϵ 0 > 0 , the subroutine computes an approximation ̃ Q π θ i ( r i ; ⋅ , ⋅) to the truncated Q-function ̂ Q π θ i ( r i ; ⋅ , ⋅) such that

with O( 1 /( ϵ 0 ) 2 ) samples, for every i ∈ N , θ ∈ Θ .

We comment that the sample complexity of the truncated Q-function evaluation described in Assumption 4.5 is not restrictive. It can be achieved with high probability by the TD-learning procedure outlined in Algorithm 2 when the agents have enough exploration [10, 43]. For brevity, we assume that (26) holds almost surely. The only difference in the probabilistic version would be the presence of an additional term for the failure probability, which does not affect the order of the sample complexity.

Theorem 4.6 (Sample-based setting) . Suppose that Assumptions 3.2, 3.3, 4.1-4.3, and 4.5 hold. For every ϵ > 0 and δ ∈ ( 0 , 1 ) , let {( µ t , θ t )} T t = 0 be the sequence generated by Algorithm 1 with T = O( ϵ -1 . 5 ) , η µ = O( ϵ -0 . 5 ) , η θ = 1 /slash.left.alt1( L θθ + 4 L 2 θµ η µ ) , ϵ 0 = O( √ ϵ ) , δ 0 = δ /( 2 n ( T + 1 )) , batch size B = O( log ( 1 / δ 0 ) ϵ -2 ) , episode length H = log ( 1 / ϵ ) , where L θθ , L θµ are Lipschitz constants defined in Lemma F.5. Then, with probability 1 -δ , there exists t ⋆ ∈ { 0 , 1 , . . . , T -1 } such that

The required number of samples is ̃ O( ϵ -3 . 5 ) .

4.1 Technical discussions

Theorem 4.4 implies an O( T -2 / 3 ) iteration complexity of Algorithm 1, matching the fastest convergence rate for solving nonconcave-convex maximin problems in the literature [47]. The approximation error O( ϕ 2 κ 0 ) decays at a linear rate w.r.t. the radius of communications. Thus, as long as the underlying network is not densely connected, such as those in wireless communication [37] and autonomous driving [48], an approximate FOSP to (5) can be efficiently computed, while each agent i only needs to communicate with a small number of agents in its neighborhood. .

In Theorem 4.4, we have chosen large step-sizes for the dual variable update to achieve the best convergence rate. This aggressive update ensures that the dual metric Y( θ t , µ t + 1 ) always remains

within a small range and also provides a satisfactory ascent direction for the policy update. Then, the average primal metric 1 / T ⋅ ∑ T -1 t = 0 [X ( θ t , µ t + 1 )] 2 is upper-bounded by exploiting a recursive relation between any two consecutive dual updates. Hence, the existence of a point ( θ t ⋆ , µ t ⋆ + 1 ) that satisfies (25) is guaranteed. It is worth noting that the proof of Theorem 4.4 can be easily generalized to the scenario where T is unspecified, and the same convergence rate can still be achieved with adaptive step-sizes η t µ = O ( t 1 / 3 ) and η t θ = 1 /slash.left.alt1( L θθ + 4 L 2 θµ η t µ ) .

Theorem 4.6 states that, with high probability, Algorithm 1 has an ̃ O( ϵ -3 . 5 ) sample complexity for finding an ϵ -FOSP of (5) with an approximation error O( ϕ 2 κ 0 ) . Note that we absorb the logarithmic terms in the notation ̃ O(⋅) . The proof of Theorem 4.6 can be broken down into two parts. Firstly, we evaluate the approximation errors of the estimators used in Algorithm 1 in relation to the model parameters, as outlined in Proposition G.1. Then, we integrate these errors into the iteration complexity result established in Theorem 4.4 and optimize the selection of parameters.

5 Numerical experiment

In this section, we validate Algorithm 1 via numerical experiments, focusing on three key questions 1 :

How does Algorithm 1 perform with multiple agents, and does the policy gradient truncation effectively alleviate computational load?

While Algorithm 1 is the first approach that provably solves the safe MARL problem with general utilities, how does it compare with existing methods for standard Safe MARL?

What benefits does the use of general utilities offer over standard cumulative rewards?

To answer these questions, we performed multiple experiments in three environments 2 . The objective functions are based on cumulative rewards, while constraint functions leverage general utilities to incentivize or dissuade agents from exploring the environments.

Synthetic environment Analogous to [24, Section 5.1], where agents are linearly arranged as 1 -2 -⋅⋅ ⋅-n . Each agent i has binary local state and action spaces, i.e., S i = A i = { 0 , 1 } , and the local transition matrix P i depends solely on its action a i and the state of agent i + 1 . The reward functions are constructed such that the optimal unconstrained policy compels all agents to continuously choose action 1 , irrespective of their states.

Pistonball A physics-based game that emphasizes cooperations and high-dimensional states as illustrated in Figure 1a. Each piston represents an agent, where its local neighborhood includes adjacent pistons, and the goal is to collectively move the ball from right to left. The agent can move up, down, or remain still. We modify the original game[49] so that the agent can only observe the ball when it enters the local neighborhood, as well as the height of neighboring pistons.

Wireless communication An access control problem following a similar setup as in [24, 50]. As illustrated in Figure 1b, the agents try to transmit packets to common access points, and the transmission fails if the access point receives more than one packet simultaneously. As there are more agents than access points, some agents need to learn to forego their benefits for the collective good .

In addition to the objective, we incorporate two types of safety constraints characterized by general utilities that cannot be easily encapsulated by standard value functions based on cumulative rewards.

Entropy constraints that stimulates exploration, formalized as Entropy ( λ π θ i ) ≥ c , ∀ i ∈ N . The function Entropy ( λ π θ i ) represents the local entropy, defined as -∑ s ∈S d π i ( s ) ⋅ log ( d π i ( s )) , where d π θ i ( s i ) = ( 1 -γ ) ∑ a i ∈A i λ π θ i ( s i , a i ) is the local state occupancy measure.

ℓ 2 -constrains that deter agents from learning overly randomized policies, formulated as ∥∑ s i ∈S i λ π θ i ∥ 2 2 ≥ c, ∀ i ∈ N . This constraint is beneficial in applications like autonomous driving and human-AI collaboration, where an agent's policy needs to be predictable for other agents.

(a) Pistonball

(b) Wireless comm.

(c) Episodic return

(d) Constraint violation

kappa=1

kappa =3

60

Episodic Return

Constraint Violation

0.8

40

20

0.6

0

0.4

-20

0.2

user

accesspoint

-40

500

1k

1.5k

2k

Step

2.5k

0

500

1k

1.5k

2k

Step

2.5k

In Figure 1, we demonstrate the performance of Algorithm 1 in the 20-agent Pistonball environment under entropy constraints. We observe that, while the truncation with κ = 3 converges in fewer iterations, truncation with κ = 1 also yields comparable performance. This underscores the efficiency of Algorithm 1 as employing a smaller communication radius can significantly reduce the computation.

Finally, we compare Algorithm 1 with three baselines based on the MAPPO-Lagrangian method [31].

MAPPO-L : the original algorithm introduced in [31]. Note that each agent has access to global information.

Decentralized MAPPO-L : decentralized version of MAPPO-L, where each agent only has access to information in the local neighborhood. However, since each agent is trained to greedily maximize its individual reward, its behaviors might sacrifice the performance of other agents.

Decentralized Aggregate MAPPO-L : decentralized version of MAPPO-L, where we address the aforementioned issue by redefining each agent's reward to be the sum of rewards of all agents in its local neighborhood.

For a fair comparison, we consider two standard safe MARL problems, where both objectives and constraints are shaped by cumulative rewards (see Appendix H.4). The results in Table 1 demonstrate that our method consistently outperforms both the centralized and decentralized variants of MAPPOLagrangian. We refer the readers to Appendix H for the comprehensive experimental results that fully answer the three questions raised at the beginning of this section.

6 Conclusion

In this work, we study the safe MARL with general utilities, with a focus on the setting of distributed training without global observability. To address the challenge of scalability and incorporating general utilities, we propose a primal-dual actor-critic method with shadow reward and κ -hop policy. Taking advantage of the spatial correlation decay property of the transition dynamics, we show that the proposed method achieves an O( T -2 / 3 ) convergence rate to the FOSP of the problem in the exact setting and achieves an ̃ O( ϵ -3 . 5 ) sample complexity, with high probability, in the sample-based setting. Finally, the effectiveness of our model and approach is verified by numerical studies. For future research, it would be interesting to develop scalable safe MARL algorithms with adaptive communication of agents' information [51] and intelligent sampling of agents' trajectories.

References

Kaiqing Zhang, Zhuoran Yang, and Tamer Ba¸ sar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of Reinforcement Learning and Control , pages 321-384, 2021.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484-489, 2016.

Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature , 575(7782):350-354, 2019.

Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:1610.03295 , 2016.

Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research , 17(1):1334-1373, 2016.

Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning. Advances in neural information processing systems , 33:10199-10210, 2020.

Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Facmac: Factored multi-agent centralised policy gradients. Advances in Neural Information Processing Systems , 34:12208-12221, 2021.

Md Shirajum Munir, Nguyen H Tran, Walid Saad, and Choong Seon Hong. Multi-agent meta-reinforcement learning for self-powered and sustainable edge computing systems. IEEE Transactions on Network and Service Management , 18(3):3353-3374, 2021.

Selim Amrouni, Aymeric Moulin, Jared Vann, Svitlana Vyetrenko, Tucker Balch, and Manuela Veloso. Abides-gym: gym environments for multi-agent discrete event simulation and application to financial markets. In Proceedings of the Second ACM International Conference on AI in Finance , pages 1-9, 2021.

Guannan Qu, Adam Wierman, and Na Li. Scalable reinforcement learning of localized policies for multi-agent networked systems. In Learning for Dynamics and Control , pages 256-266. PMLR, 2020.

Eitan Altman. Constrained Markov decision processes , volume 7. CRC Press, 1999.

Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning , page 1, 2004.

Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070 , 2018.

Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning , pages 2681-2691. PMLR, 2019.

Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdinov. Efficient exploration via state marginal matching. arXiv preprint arXiv:1906.05274 , 2019.

Tom Zahavy, Brendan O'Donoghue, Guillaume Desjardins, and Satinder Singh. Reward is enough for convex mdps. Advances in Neural Information Processing Systems , 34, 2021.

Junyu Zhang, Alec Koppel, Amrit Singh Bedi, Csaba Szepesvari, and Mengdi Wang. Variational policy gradient method for reinforcement learning with general utilities. Advances in Neural Information Processing Systems , 33:4572-4583, 2020.

Cameron Nowzari, Victor M Preciado, and George J Pappas. Optimal resource allocation for control of networked epidemic models. IEEE Transactions on Control of Network Systems , 4(2):159-169, 2015.

Wei Chen, Alex Collins, Rachel Cummings, Te Ke, Zhenming Liu, David Rincon, Xiaorui Sun, Yajun Wang, Wei Wei, and Yifei Yuan. Influence maximization in social networks when negative opinions may emerge and propagate. In Proceedings of the 2011 siam international conference on data mining , pages 379-390. SIAM, 2011.

Co-Pierre Georg. The effect of the interbank network structure on contagion and common shocks. Journal of Banking & Finance , 37(7):2216-2228, 2013.

Qiu Jin, Guoyuan Wu, Kanok Boriboonsomsin, and Matthew Barth. Platoon-based multi-agent intersection management for connected vehicle. In 16th international ieee conference on intelligent transportation systems (itsc 2013) , pages 1462-1467. IEEE, 2013.

Andrea J Goldsmith and Stephen B Wicker. Design challenges for energy-constrained ad hoc wireless networks. IEEE wireless communications , 9(4):8-27, 2002.

Junyu Zhang, Amrit Singh Bedi, Mengdi Wang, and Alec Koppel. Marl with general utilities via decentralized shadow reward actor-critic. Proceedings of the AAAI Conference on Artificial Intelligence , 2022.

Guannan Qu, Adam Wierman, and Na Li. Scalable reinforcement learning for multiagent networked systems. Operations Research , 70(6):3601-3628, 2022.

Songtao Lu, Kaiqing Zhang, Tianyi Chen, Tamer Ba¸ sar, and Lior Horesh. Decentralized policy gradient descent ascent for safe multi-agent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 8767-8775, 2021.

Washim Uddin Mondal, Vaneet Aggarwal, and Satish V Ukkusuri. Mean-field approximation of cooperative constrained multi-agent reinforcement learning (cmarl). arXiv preprint arXiv:2209.07437 , 2022.

Dongsheng Ding, Xiaohan Wei, Zhuoran Yang, Zhaoran Wang, and Mihailo R Jovanovic. Provably efficient generalized lagrangian policy optimization for safe multi-agent reinforcement learning. https://dongshed.github.io/papers/22dingprovably.pdf , 2023.

Vincent D Blondel and John N Tsitsiklis. A survey of computational complexity results in systems and control. Automatica , 36(9):1249-1274, 2000.

Michael Rotkowitz and Sanjay Lall. A characterization of convex problems in decentralized control. IEEE transactions on Automatic Control , 50(12):1984-1996, 2005.

Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems , 12, 1999.

Shangding Gu, Jakub Grudzien Kuba, Munning Wen, Ruiqing Chen, Ziyan Wang, Zheng Tian, Jun Wang, Alois Knoll, and Yaodong Yang. Multi-agent constrained policy optimisation. arXiv preprint arXiv:2110.02793 , 2021.

Dimitri P Bertsekas. Nonlinear programming. Journal of the Operational Research Society , 48(3):334-334, 1997.

Katta G Murty and Santosh N Kabadi. Some np-complete problems in quadratic and nonlinear programming. Mathematical Programming: Series A and B , 39(2):117-129, 1987.

Carlo Alfano and Patrick Rebeschini. Dimension-free rates for natural policy gradient in multi-agent reinforcement learning. arXiv preprint arXiv:2109.11692 , 2021.

Hans-Otto Georgii. Gibbs measures and phase transitions. In Gibbs Measures and Phase Transitions . de Gruyter, 2011.

David Gamarnik. Correlation decay method for decision, optimization, and inference in largescale networks. In Theory Driven by Influential Applications , pages 108-121. INFORMS, 2013.

David Tse and Pramod Viswanath. Fundamentals of wireless communication . Cambridge university press, 2005.

Yizhou Zhang, Guannan Qu, Pan Xu, Yiheng Lin, Zaiwei Chen, and Adam Wierman. Global convergence of localized policy iteration in networked multi-agent reinforcement learning. arXiv preprint arXiv:2211.17116 , 2022.

Sungho Shin, Yiheng Lin, Guannan Qu, Adam Wierman, and Mihai Anitescu. Near-optimal distributed linear-quadratic regulator for networked systems. arXiv preprint arXiv:2204.05551 , 2022.

Mohammad Gheshlaghi Azar, Rémi Munos, and Hilbert J Kappen. Minimax pac bounds on the sample complexity of reinforcement learning with a generative model. Machine learning , 91(3):325-349, 2013.

Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. Advances in Neural Information Processing Systems , 32, 2019.

Gen Li, Yuting Wei, Yuejie Chi, Yuantao Gu, and Yuxin Chen. Sample complexity of asynchronous q-learning: Sharper analysis and variance reduction. Advances in neural information processing systems , 33:7031-7043, 2020.

Yiheng Lin, Guannan Qu, Longbo Huang, and Adam Wierman. Multi-agent reinforcement learning in stochastic networked systems. Advances in Neural Information Processing Systems , 34:7825-7837, 2021.

Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8(3):229-256, 1992.

Andrew R Conn, Nicholas IM Gould, and Philippe L Toint. Trust region methods . SIAM, 2000.

Maher Nouiehed, Maziar Sanjabi, Tianjian Huang, Jason D Lee, and Meisam Razaviyayn. Solving a class of non-convex min-max games using iterative first order methods. Advances in Neural Information Processing Systems , 32, 2019.

Quoc Tran Dinh, Deyi Liu, and Lam Nguyen. Hybrid variance-reduced sgd algorithms for minimax problems with nonconvex-linear function. Advances in Neural Information Processing Systems , 33:11096-11107, 2020.

Jiadai Wang, Jiajia Liu, and Nei Kato. Networking and communications in autonomous driving: A survey. IEEE Communications Surveys & Tutorials , 21(2):1243-1274, 2018.

J Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis S Santos, Clemens Dieffendahl, Caroline Horsch, Rodrigo Perez-Vicente, et al. Pettingzoo: Gym for multi-agent reinforcement learning. Advances in Neural Information Processing Systems , 34:15032-15043, 2021.

Xin Liu, Honghao Wei, and Lei Ying. Scalable and sample efficient distributed policy gradient algorithms in multi-agent networked systems. arXiv preprint arXiv:2212.06357 , 2022.

Jiechuan Jiang, Chen Dun, Tiejun Huang, and Zongqing Lu. Graph convolutional reinforcement learning. arXiv preprint arXiv:1810.09202 , 2018.

Santiago Paternain, Miguel Calvo-Fullana, Luiz FO Chamon, and Alejandro Ribeiro. Safe policies for reinforcement learning via primal-dual methods. arXiv preprint arXiv:1911.09101 , 2019.

Ming Yu, Zhuoran Yang, Mladen Kolar, and Zhaoran Wang. Convergent policy optimization for safe reinforcement learning. Advances in Neural Information Processing Systems , 32:31273139, 2019.

Gabriel Dulac-Arnold, Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. arXiv preprint arXiv:1904.12901 , 2019.

Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205.10330 , 2022.

Frits De Nijs, Erwin Walraven, Mathijs De Weerdt, and Matthijs Spaan. Constrained multiagent markov decision processes: A taxonomy of problems and algorithms. Journal of Artificial Intelligence Research , 70:955-1001, 2021.

Yonathan Efroni, Shie Mannor, and Matteo Pirotta. Exploration-exploitation in constrained MDPs. arXiv preprint arXiv:2003.02189 , 2020.

Dongsheng Ding, Xiaohan Wei, Zhuoran Yang, Zhaoran Wang, and Mihailo Jovanovic. Provably efficient safe exploration via primal-dual policy optimization. In International Conference on Artificial Intelligence and Statistics , pages 3304-3312. PMLR, 2021.

Tao Liu, Ruida Zhou, Dileep Kalathil, PR Kumar, and Chao Tian. Learning policies with zero or bounded constraint violation for constrained MDPs. arXiv preprint arXiv:2106.02684 , 2021.

Donghao Ying, Yuhao Ding, and Javad Lavaei. A dual approach to constrained markov decision processes with entropy regularization. In International Conference on Artificial Intelligence and Statistics , pages 1887-1909. PMLR, 2022.

Donghao Ying, Mengzi Guo, Yuhao Ding, Javad Lavaei, et al. Policy-based primal-dual methods for convex constrained markov decision processes. Proceedings of the AAAI Conference on Artificial Intelligence , 2022.

Yuhao Ding and Javad Lavaei. Provably efficient primal-dual reinforcement learning for cmdps with non-stationary objectives and constraints. arXiv preprint arXiv:2201.11965 , 2022.

Qinbo Bai, Amrit Singh Bedi, Mridul Agarwal, Alec Koppel, and Vaneet Aggarwal. Achieving zero constraint violation for constrained reinforcement learning via primal-dual approach. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 3682-3689, 2022.

Eitan Altman and Adam Shwartz. Constrained markov games: Nash equilibria. In Advances in dynamic games and applications , pages 213-221. Springer, 2000.

E Gómez-Ramırez, K Najim, and AS Poznyak. Saddle-point calculation for constrained finite markov chains. Journal of Economic Dynamics and Control , 27(10):1833-1853, 2003.

Eitan Altman, Konstantin Avrachenkov, Nicolas Bonneau, Merouane Debbah, Rachid El-Azouzi, and Daniel Sadoc Menasche. Constrained cost-coupled stochastic games with independent state processes. Operations Research Letters , 36(2):160-164, 2008.

Vikas Vikram Singh and N Hemachandra. A characterization of stationary nash equilibria of constrained stochastic games with independent state processes. Operations Research Letters , 42(1):48-52, 2014.

Vinayaka G Yaji and Shalabh Bhatnagar. Necessary and sufficient conditions for optimality in constrained general sum stochastic games. Systems & Control Letters , 85:8-15, 2015.

Qingda Wei. Constrained expected average stochastic games for continuous-time jump processes. Applied Mathematics & Optimization , 83(3):1277-1309, 2021.

Wenzhao Zhang and Xiaolong Zou. Constrained average stochastic games with continuous-time independent state processes. Optimization , 71(9):2571-2594, 2022.

Junyu Zhang, Chengzhuo Ni, Csaba Szepesvari, Mengdi Wang, et al. On the convergence and sample efficiency of variance-reduced policy gradient method. Advances in Neural Information Processing Systems , 34, 2021.

Matthieu Geist, Julien Pérolat, Mathieu Laurière, Romuald Elie, Sarah Perrin, Olivier Bachem, Rémi Munos, and Olivier Pietquin. Concave utility reinforcement learning: the mean-field game viewpoint. arXiv preprint arXiv:2106.03787 , 2021.

Donghao Ying, Yuhao Ding, Alec Koppel, and Javad Lavaei. Scalable multi-agent reinforcement learning with general utilities. American Control Conference , 2023.

Weichao Zhou and Wenchao Li. Safety-aware apprenticeship learning. In International Conference on Computer Aided Verification , pages 662-680. Springer, 2018.

Sobhan Miryoosefi, Kianté Brantley, Hal Daume III, Miro Dudik, and Robert E Schapire. Reinforcement learning with convex constraints. Advances in Neural Information Processing Systems , 32, 2019.

Qisong Yang and Matthijs TJ Spaan. Cem: Constrained entropy maximization for task-agnostic safe exploration. In The Thirty-Seventh AAAI Conference on Artificial Intelligence , 2023.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons, 2014.

Harshat Kumar, Alec Koppel, and Alejandro Ribeiro. On the sample complexity of actorcritic method for reinforcement learning with function approximation. arXiv preprint arXiv:1910.08412 , 2019.

Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research , 22(98):1-76, 2021.

Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

Dongsheng Ding, Kaiqing Zhang, Tamer Basar, and Mihailo Jovanovic. Natural policy gradient primal-dual method for constrained markov decision processes. Advances in Neural Information Processing Systems , 33:8378-8390, 2020.

Gal Dalal, Balazs Szorenyi, and Gugan Thoppe. A tale of two-timescale reinforcement learning with the tightest finite-time bound. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 3701-3708, 2020.

Maxim Kaledin, Eric Moulines, Alexey Naumov, Vladislav Tadic, and Hoi-To Wai. Finite time analysis of linear two-timescale stochastic approximation with markovian noise. In Conference on Learning Theory , pages 2144-2203. PMLR, 2020.

Tengyu Xu and Yingbin Liang. Sample complexity bounds for two timescale value-based reinforcement learning algorithms. In International Conference on Artificial Intelligence and Statistics , pages 811-819. PMLR, 2021.

Vivek S Borkar and Sarath Pattathil. Concentration bounds for two time scale stochastic approximation. In 2018 56th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 504-511. IEEE, 2018.

Shuang Qiu, Zhuoran Yang, Xiaohan Wei, Jieping Ye, and Zhaoran Wang. Single-timescale stochastic nonconvex-concave optimization for smooth nonlinear td learning. arXiv preprint arXiv:2008.10103 , 2020.

Jonas Moritz Kohler and Aurelien Lucchi. Sub-sampled cubic regularization for non-convex optimization. In International Conference on Machine Learning , pages 1895-1904. PMLR, 2017.

Shicong Cen, Chen Cheng, Yuxin Chen, Yuting Wei, and Yuejie Chi. Fast global convergence of natural policy gradient methods with entropy regularization. Operations Research , 70(4):25632578, 2022.

Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning , pages 214-223. PMLR, 2017.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work extends constrained single-agent general utilities RL (where the objective is a non-convex function of occupancy measure) to a constrained multi-agent setting. Naturally, similar to single-agent, the constraints are satisfied in expectations. They assume the combined (global) objective to be additive in terms of agents. The major novelty part of the paper is the handling of constraints (unconstrained multi-agent is a prior work). In a multi-agent setting, communication is a challenge, and to tackle this, authors utilise a k-hop type method under which agents can communicate with nearby agents.\n\nSTRENGTHS:\n- Extension to multi-agent constrained setting (constrained are of general utility type)\n- k-hop policies (I am not sure if it is a novel contribution or from earlier work)\n- Convergence analysis for both exact and stochastic setting\n\nWEAKNESSES:\nObjective: The considered global objective function is additive in agents. For a framework of general utilities RL, I would expect to consider general dependence between the agent, especially since they motivate the problem for exploration/imitation. If the objective is exploration, then the current multi-agent framework is still performing exploration individually only and it might perform differently on a combined exploration objective. \n\nContributions: The contributions were not so clear from [72]. If a multi-agent extension with truncated PG utilizing shadow rewards was already done and if a constrained part for a single agent existed, then I am not sure if the paper has a novel contribution.\n\nExperiments: Since the proposed algorithm is PG type, I would expect authors to level PG tools and perform experiments with high dim state action space and complicated dynamics, e.g. car or Ant (may be continuous space, not sure if easily can be done). Since the motivation of the problem is done using exploration and imitation tasks, I think an essential component in the paper should be some experiments on these tasks and comparing performance with the case when the additivity of agents is relaxed.\n\nQUESTIONS:\n- I would suggest authors to have a subsection on challenges, wrt to a single agent case. It might be worth mentioning here that are not trivial single-agent to multi-agent extensions. \n- Is K-hop strategy and analysis existing in prior work? \n- Maybe bring related works in the main paper\n- It will be good to have an algorithm block in the main paper as well.\n\nLIMITATIONS:\nno potential negative impact", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the safe MARL with general utilities and focuses on the setting of distributed training without global observability. The authors propose a primal-dual actor-critic method with shadow reward and \\kappa-hop policy to tackle the problems of scalability and incorporating general utilities. They provide the theoretical analysis for the convergence rate and sampling complexity. In addition, they also demonstrate the effectiveness of their method via comprehensive experiment results.\n\nSTRENGTHS:\n(1) Novel formulation: This paper provides a clear mathematical formulation for the MARL with general utility problems. This formulation will help the real-world multi-agent problem modeling in the general case, and encourage more discussion on the intersected domain of Safe RL and MARL.\n\n(2) Solid theoretical analysis: This work provides solid mathematical analysis for the convergence guarantee in general-utilities settings. The analysis is rigorous as well as easy to follow. In addition, they provide a detailed discussion of the properties and show how the convergence can be guaranteed in common multi-agent scenarios. \n\n(3) Comprehensive experimental results: The authors validate their proposed method using three experiment tasks of different categories. It is a pleasure to see the detailed task description, code implementation, and results corresponding to different settings (e.g. number of neighbors, constraint thresholds) in the submission.\n\nWEAKNESSES:\n(1) Lack of baseline comparison in the main context: I understand that the authors want to display more theoretical analysis so they may move the baseline comparison into the appendix. However, I would suggest adding this section back to the main content as it is one of the most convincing parts to show the strengths of this work.\n\nQUESTIONS:\n(1) Experiment settings: I am wondering how you design the experiment settings. Based on your description in section H.4, (1.1) what is the motivation to set an upper bound for the cumulative reward in the Pistonball task? (1.2) Do you mean additional cost u_i in line 1287, or this term is directly added to the reward as the penalization?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the safe multi-agent rl in the fully-decentralized setting. In the system, agents need to satisfy local safety constraints, and they also have to maximize the joint objectives. Moreover, the objectives are general utilities, which have broader applications than common discounted reward. Authors utilize shadow reward and kappa-hop neighbor truncation to limit the communication to be only among neighboring agents, which removes the need of global state and global Q function when updating each local agent. Both convergence and the sample complexity are analyzed. The experiment results also validate the effectiveness of the proposed algorithm.\n\nSTRENGTHS:\nThis paper has the following merits.\n1. The paper is well-written generally and easy to follow.\n2. The kappa-hop truncation is intuitive and interesting for many real-world applications, and it also successfully eliminates the need of global state and global Q of all agents.\n3. Authors establish detailed convergence and sample complexity analysis, with thorough proofs and discussions.\n\nWEAKNESSES:\nWhile promising, I have some concerns on the proposed algorithm. See Questions part for more details.\n\nQUESTIONS:\nI have the following concerns on the paper.\n1.\tThe potential scope and real-world applications for such fully-distributed MARL are not clear. As we all know, rl usually requires heavy exploration of the environment, and it also needs lots of samples to converge. Thus, it becomes adventurous and high-cost to deploy these policies directly to the environment, and let them train in a distributed way.\n2.\tThe communication protocol for two unconnected but neighboring agents is not clear. They can communicate directly through P2P, or they can only transmit the data agent-by-agent if the communication is limited to connected agents. Such difference is essential for the algorithm design, as the former one may generate high-cost for communication, while the latter one may bring data latency to each agent.\n3.\tAssumption 3.2 is too strong, as it involves the summation over all agents. The system that satisfies the assumption is nearly independent, as each agent's action has little influence on other's state transition.\n4.\tThe algorithm has 3 major steps but relies heavily on communication. The communication complexity is not analyzed in the work, which is in fact a major metric for decentralized optimization algorithms [1,2]. Further, the scalability is not experimented/discussed in detail in the paper. In fact, I have doubts on the scalability of the algorithm, especially for dense agents, as the communication overhead becomes extremely heavy. One related work is missing [3].\n\n[1]. Optimal and Practical Algorithms for Smooth and Strongly Convex Decentralized Optimization, NIPS, 2020\n[2]. Lower Bounds and Optimal Algorithms for Smooth and Strongly Convex Decentralized Optimization Over Time-Varying Networks, NIPS, 2021\n[3]. Decom: decomposed policy for constrained cooperative multi-agent reinforcement learning, AAAI, 2023\n\nLIMITATIONS:\nAuthors discussed the future works on the scalable safe MARL algorithms with adaptive communication of agents’ state/action information and intelligent sampling of agents’ trajectories. I personally prefer to focus more on the time-varying networks [2], where the connection is unstable and the communication has some unignored latency.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of safe/constrained multi-agent reinforcement learning (MARL) with general utilities and constraints, which could be generic functions of agents' occupancy measures. A distributed method is proposed for solving the constrained MARL problem with local constraints, where each agent can communicate to its few-hop neighbors in the underlying network graph topology. The proposed algorithm is theoretically analyzed in terms of convergence behavior and sample complexity, and numerical results are provided to demonstrate the efficacy of the proposed method in multiple constrained MARL environments.\n\nSTRENGTHS:\n- The problem of constrained/safe MARL is of great practical interest and importance, so the paper focuses on a timely topic of interest to the NeurIPS audience.\n- Distributed training of safe RL policies in multi-agent settings is of significance, and the paper makes important contributions in this direction.\n- Extensive theoretical and numerical analysis is provided to showcase the benefits of the proposed approach.\n\nWEAKNESSES:\n- The presentation and structure of the paper could be improved, especially since many of the important parts of the paper (especially the algorithm pseudo-code, flowchart, and numerical results) are deferred to the appendix. Mindful of the space limitations, it would be great if the authors could shuffle the contents in their revision to include the pseudo-code/flowchart and more numerical results in the main body of the paper, with at least one result for each environment introduced in Section 5.\n- The authors mention [25] (Lu et al., AAAI'21) in their discussion of related work, but I would have liked a more extensive discussion on the similarities and differences between this work and [25], especially since they both consider an underlying graph topology, as well as numerical comparisons with [25] in addition to MAPPO-Lagrangian.\n- The paper mostly focuses on tabular MARL settings with a finite state space. It is unclear how the proposed method generalizes to continuous state spaces.\n\nQUESTIONS:\n- I believe the Transition decomposition assumption in lines 106-108 is very strong. Could you please comment on how you justify this assumption in practical settings and whether it could be relaxed to a weaker assumption?\n- How does the proposed method extend to the case where the objective in Eq. (5) is not separable?\n- Intuitively, the upper bound in Eq. (13) should depend on $\\omega$ and $\\chi$ in Assumption 3.2, so please discuss why they do not appear here.\n- In Eq. (15), could $1/n$ be replaced with $1/|\\mathcal{N}_i^{\\kappa}|$? I understand that this is simply a scaling factor, but truncating the policies to $\\kappa$-hop neighbors could also justify the aforementioned replacement, where the Lagrangian is always *averaged* over the $\\kappa$-hop neighborhood of each agent.\n- Does Fig. 1-d show that the constraints are always violated? Does this mean that no feasible policy in this environment might exist?\n- In the Appendix, it is shown that MAPPO-L performs worse than the proposed method. Could you provide a more in-depth discussion of why this happens? As the authors allude to, MAPPO-L has access to the global state during the centralized training phase, so it is okay (and probably expected) if the proposed method (which is fully distributed) underperforms MAPPO-L.\n- Since a graph structure and distributed implementation are the underpinnings of the proposed method, how does the proposed method related to graph neural networks (GNNs)? The agents could perform message passing to their neighbors through a GNN architecture, so it would be helpful if the authors could discuss potential connections of their method to MARL methods based on GNNs, such as the following reference [A]:  \n[A] Jiang, Jiechuan, Chen Dun, Tiejun Huang, and Zongqing Lu. \"Graph Convolutional Reinforcement Learning.\" In International Conference on Learning Representations. 2019.\n\nLIMITATIONS:\nAs mentioned under **Weaknesses**, the extension to non-tabular MARL settings is unclear. Furthermore, the robustness of the proposed method to imperfect communication among agents (e.g., noise perturbations, random communication link drops, delays, time/frequency resource constraints, etc.) needs to be studied.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}