DPMAC: Differentially Private Communication for Cooperative Multi-Agent Reinforcement Learning
Authors: Canzhe Zhao
Yanjie Ze
Jing Dong
Baoxiang Wang
Shuai Li
Venue: Submitted to ICLR 2023 (2023)
Keywords: Communication in deep multi-agent reinforcement learning
Deep multi-agent reinforcement learning
Differential privacy
Game theory

Abstract: Communication lays the foundation for cooperation in human society and in multi-agent reinforcement learning (MARL). Humans also desire to maintain their privacy when communicating with others, yet such privacy concern has not been considered in existing works in MARL. We propose the \textit{differentially private multi-agent communication} (DPMAC) algorithm, which protects the sensitive information of individual agents by equipping each agent with a local message sender with rigorous $(\epsilon, \delta)$-differential privacy (DP) guarantee. In contrast to directly perturbing the messages with predefined DP noise as commonly done in privacy-preserving scenarios, we adopt a stochastic message sender for each agent respectively and incorporate the DP requirement into the sender, which automatically adjusts the learned message distribution to alleviate the instability caused by DP noise. Further, we prove the existence of a Nash equilibrium in cooperative MARL with privacy-preserving communication, which suggests that this problem is game-theoretically learnable. Extensive experiments demonstrate a clear advantage of DPMAC over baseline methods in privacy-preserving scenarios.

Under review as a conference paper at ICLR 2023
DPMAC: DIFFERENTIALLY PRIVATE COMMUNICA- TION FOR COOPERATIVE MULTI-AGENT REINFORCE- MENT LEARNING
Anonymous authors Paper under double-bling brind review
ABSTRACT
Communication lays the foundation for cooperation human society and multi-agent reinforcement learning (MARL). Humans also desire to maintain their privacy when communicating with others, yet such privacy concern has not been considered in existing works in MARL. We propose the differentially pri- vate multi-agent communication (DPMAC) algorithm. which protects the sensi- tive information of individual agents by equipping each agent with local mes- sage sender with rigorous (c, didfffrertlia privacy (DP) guarantee. In contrast to directly perturbing the messages with predefined DP noise as commonly done in privacy-preserving scenarios, we adopt stochastic message sender for each agent respectively and incorporate the DP requirement into the sender, which automati- cally adjusts the learned message distribution to alleviate the instability caused by DP noise. Further, we prove the existence of a Nash equilibrium cooperative MARL with privacy-presery lem is game- the--thhoretically learnable. Extensive experiments demonstrate a clear advantage of DPMAC over baseline methods in drivacy-presevvin scenarios.
communication, which suggests that this prob-
INTRODUCTION
Multi-agent reinforcement learning (MARL) has shown remarkable achievements many real- world applications such sensor networks (Zhang & Lesser, 2011), autonomous driving (Shalev- Shwartz. et al., 2016b), and traffic control (Wei al., 2019). To mitigate non-stationarity when training the multi-agent system, centralized training and decentralized execution (CTDE) paradigm is proposed. The CTDE paradigm yet faces the hardness to enable complex cooperation and coordi- nation for agents during execution due to the inherent partial observability in multi-agent scenarios (Wang et al., 2020b). To make agents cooperate more efficiently in complex partial observable environments, communication between agents has been considered. Numerous works proposed dif- ferentiable communication methods between agents, which can be trained in an end-to-end manner, for efficient cooperation among agents (Foerster et al., 2016; Jiang & Lu, 2018; Das et al., 2019; Ding et al., 2020; Kim et al., 2021; Wang et al., 2020b). The communication can be either broadcast (Das et al., 2019; Jiang & Lu, 2018; Wang et al., 2020b), where the connection between agents can be modeled as a complete graph, or one-to-one as a general graph (Ding et al., 2020).
However, the advantages of communication, resulting from full information sharing, come with the possible privacy leakage of individual agents for both broadcasted and one-to-one messages. There- fore, practice, one agent may unwilling to fully share its private information with other agents even though in cooperative scenarios. For instance, if we train and deploy an MARL-based au- tonomous driving system, each autonomous vehicle involved in this system could regarded as an agent and all vehicles work together to improve the safety and efficiency of the system. Hence, this can be regarded as a cooperative MARL scenario (Shalev-Shwarte et al., 2016a; Yang et al., 2020). However, owners of autonomous vehicles may not allow their vehicles send private information to other vehicles without any desensitization since this may divulge their private information such as their personal life routines (Hassan et al., 2020). Hence, natural question arises:
Can the MARL algorithm with communication under the CTDE framework be endowed with both the rigorous privacy guarantee and the empirical efficiency?

---

Under review as a conference paper at ICLR 2023
To answer this question, start with simple motivating example called single round binary sums, where several players attempt to guess the bits possessed by others and they can share their own information by communication. In Section 4, we show that a local message sender using the randomized response mechanism allows an analytical receiver to correctly calculate the binary sum in not supposed to be aggregated likewise in previous communication methods in MARL (Das et al., 2019; Ding et al., 2020), a trusted data curator is not available in general. On the contrary, privacy is supposed to be achieved locally for every agent; 2) Once the agents know a priori, that certain privacy constraint exists, they could adjust their inference on the noised message. These two insights indicate the principles of our privacy-preserving communication structure that we desire privacy-preservin local sender and a privacy-aware analytical receiver.
way. From the example gain two insights: 1) The information is
Our algorithm, differentially private multi-agent communication (DPMAC), instantiates the de- scribed principles. More specifically, for the sender part, each agent equipped with a local sender which ensures differential privacy (DP) (Dwork, 2006) by performing The message sender in DPMAC is local the sense that each agent is equipped with its own message sender, which is only used to send its own messages. Equipped with this local sender. DPMAC is able to not only protect the privacy of communications between agents but also satisfy different pri- vacy levels required from different agents. In addition, the sender adopt the Gaussian distribution to represent the message space and sample the stochastic message from the learned distribution. How- ever, is known that the DP noise may impede the original learning process (Dwork et al., 2014; Alvim et al., 201 I), resulting in unstable or even divergent algorithms, especially for deep-lcarning- based methods (Abadi al., 2016; Chen et al., 2020). To cope with this issue, we incorporate the noise variance into the representation of the message distribution, so that the agents could learn to adjust the message distribution automatically according to varying noise scales. For the receiver part, because of the gradient chain between the sender and the receiver, our receiver naturally uti- lizes the privacy-relevant information hidden in the gradients. This implements the privacy-aware analytical receiver described in the motivating example.
additive Gaussian noise.
When protecting the privacy in communication is required in a cooperative game, the game is not purely cooperative anymore since each player involved will face trade-off between the team utility and its personal privacy. To analyze the convergence of cooperative games with privacy-pree rrvaacy-reseervng communication, first define single-step game, namely the collaborative game with privacy (CGP). We prove that under some mild assumptions of the players' value functions, CGP could be transformed into potential game (Monderer & Shapley, 1996), subsequently leading to the existence of a Nash equilibrium (NE). With this property, NE could also be proved to exist in the single round binary sums game. Furthermore, we extend the single round binary sums into a multi- step game called multiple round sums using the notion of Markov potential game (MPG) (Leonardos et al., 2021). 1). Inspired by Macua al. (2018) and modeling the prreeeriig communication as part of the agent action, we prove the existence of NE, which indicates that the multi-step game with
communication could be learnable.
To validate the effectiveness of DPMAC, extensive experiments are conducted in multi-agent parti- cle environment (MPE) (Lowe et al., 2017), including cooperative navigation, cooperative commu- nication and navigation, and predator-prey. Specifically, in privacy-preservigg scenarios, DPMAC significantly outperforms baselines. Moreover, even without any privacy constraints, DPMAC could gain competitive performance against baselines.
To sum up, the contributions of this work are threefold:
To the best of our knowledge, we make the first attempt to develop a framework for private com- munication in MARL, named DPMAC, with the theoretical guarantee of OD.))-DP
We prove the existence of the Nash equilibrium for the cooperative games with privvcc--reeeevvin communication, which shows that these games are learnable.
Experiments on the MPE show that DPMAC clearly outperforms other algorithms in privacy- preserving scenarios and gains competitive performance in non-private scenarios.
2

---

Under review as a conference paper at ICLR 2023
2 RELATED WORK
Learning to communicate in MARL Learning communication protocols in MARL by backprop-
agation and end-to-ent training has achieved great advances in recent years (Sukhbaatar al., 201 Foerster et al., 2016; 6; Jiang & Lu, 2018; Das et al., 2019; Wang et al., 2020b; Ding et al., 2020; Kim al., 2021; Rangwala & Williams, 2020; Zhang et al., 2019; Singh al., 2019; Zhang al., 2020; 2021; Lin et al., 2021; Peng et al., 2017). Amongst these works, Sukhbaatar et al. (2016) propose CommNet as the first differentiable communication framework for MARL. Further, TarMAC (Das et al., 2019) and ATOC (Jiang & Lu, 2018) utilize the attention mechanism to extract useful informa- tion as messages. 12C (Ding al., 2020) makes the first attempt to enable agents to learn one-to-one communication via causal inference. Wang et al. (2020b) propose NDQ, which learns nearly decom- posable value functions to reduce the communication overhead. Kim al. (2021) consider sharing an imagined trajectory as an intention for effectiveness. Besides, to communicate in the scenarios with limited bandwidth, some works consider learning to send compact and informative messages in MARL via minimizing the entropy of messages between agents using information bottleneck methods (Wang et al., 2020a; Tucker et al., 2022; Tian et al., 2021; Li et al., 2021). While learn- ing effective communication in MARL has been extensively investigated, existing communication algorithms potentially leave the privacy of each agent vulnerable to information attacks.
Privacy preserving in RL With wide attention on reinforcement learning (RL) algorithms and applications recent years, so have concerns about their privacy. Sakuma al. (2008) consider privacy in the distributed RL problem and utilize cryptographic tools to protect the private state- action-state triples. Algorithmically, Balle al. (2016) make the first attempt to establish a policy evaluation algorithm with differential privacy (DP) guarantee, where the Monte-Carlo estimates are perturbed with Gaussian noise. Wang & Hegde (2019) generalize the results to Q-learning. where functional noise is added to protect the reward function. Theoretically, Garcelon et al. (2021) analyze the regret bound of finite-horizon MDPs in the tabular case. In a large or continuous state space where function approximation is required, Zhou (2022) subsequently takes the first step to establish the sublinear regret in linear mixture Markov decision processes (MDPs). Zhao et al. (2022) propose the differentially private version of the temporal difference learning with nonlinear function approximation. Mcanwhile, a large number of works focus on preserving privacy in multi- armed bandits (Tao et al., 2022; Tenenbaum et al., 2021; Dubey, 2021; Zheng et al., 2020; Dubey & Pentland, 2020; Tossou& & Dimitrakakis, 2017).
Privacy is also studied in recent literature on MARL and multi-agent system. Ye et al. (2020) study differential advising for value-based agents, which share action values as the advice, largely differing in both the communication framework and the CTDE framework. Dong et al. (2020) propose an average consensus algorithm with DP guarantee in the multi-agent system.
PRELIMINARIES
We consider a fully cooperative MARL problem where N agents work collaboratively maximize the joint rewards. The underlying environment can be captured by decentralized partially observ- able Markov decision process POMDP,, denoted by the tuple (S,
R) Specifically,
is the global state space, A = -I A; is the joint action space, = I I O, is the joint ob- servation space, P (s' s. a)
S x A x S + [0, 1] determines the state transition dynamics, R(s, a): S x A + R is the reward function, and E [0, 1) is the discount factor. Given a joint policy { 1= the joint action-value function time I is QT (s', at) E [G' |s s a T). where G Di=o RR is the cumulative reward, and a' {a:) is the joint action. The ultimate goal of the agents is to find an optimal policy π which maximizes Q" (s* aa))
Under the aforementioned cooperative setting, we study the case where agents allowed to com- municate with a joint message space M = -I M,. When the communication is unrestricted, the problem is reduced to single-agent RL problem, which effectively solves the challenge posed by partially observable states, but puts the individual agent's privacy at risk. To overcome the challenges of privacy and partial observable states simultaneously, we investigate algorithms that maximize the cumulative rewards while satisfying differential privacy (DP), given in Definition 3.1.
3

---

Under review as a conference paper at ICLR 2023
Definition 3.1 ((c, -D,, Dwork (2006)). A randomized mechanism f D - R satisfies (c, () differential privacy for any neighbouring datasets D, D' E ED and C R, it holds that Pr(f(D) E S) S E Pr (D') S] + 5.
DP offers mathematically rigorous way to quantify the privacy of an algorithm (Dwork, 2006). An algorithm is said to be "privatized" under the DP model if it is statistically hard to infer the presence of individual data point in the dataset by observing the output of the algorithm. More intuitively, an algorithm satisfies DP it provides nearly the same outputs given the neighbouring input datasets (i.e., Pr[f(D) E S] ~ [f (D') E SD, which hence protects the sensitive information from the curious attacker.
With DP, each agent is assigned with a privacy budget e,, which is negatively correlated to the level of privacy protection. Then we have e = {< as the set of all privacy budgets. In addition to maximizing the joint rewards as usually required in cooperative MARL, the messages sent from agent i are also required to satisfy the privacy budget with probability at least 1 1 8.
4 MOTIVATING EXAMPLE
Before introducing our communication framework, we first investigate motivating example, which is a cooperative game and inspires the design principles of private communication mechanisms in MARL. The motivating example is simple yet interesting game, called single round binary sums. The game is extended from the example provided in Cheu (2021) for analyzing the shuffle model, while we illustrate the game from the perspective of multi-agent systems. We note that though this game is one-step, which is different from the sequential decision process like MDP, it is illustrative enough to show how the communication protocol works as a tool to achieve better trade-off between privacy and utility.
Assume that there are N agents involved in this game. Each agent i [N] has a bit b, E [0. 1] and can tell other agents the information about its bit by communication. The objective of the game is for every agent to guess E, b, the sum of the bits of all agents. Namely, each agent i makes guess Ii and the utility of the agent is to maximize r; =- EE bj Blg.]l. The (global) reward of this game is the sum of the utility over all agents, ie.,, E.
Without loss of generality, we write the guess Ji into 9: = Eez Big + b,, where Vij the guessed bit of agent j by agent If all agents share their bits without covering up. the guessed bit yiy will obviously be equal to b, and all agents attain optimal return. Hence this game is fully cooperative under no privacy constraints. However, the optimal strategy under the assumption that everyone is altruistic to share their own bits.
To preserve the privacy in communication, the message (i.e., the sent bit) could be randomized using randomized response, which perturbs the bit b, with probability p, shown below:
Ber((1/2)
with probability otherwise
= RRR (b;) = -1b
where x; is the random message and Ber indicates the Bernoulli distribution. Under our context, RRR is privacy-preserving message sender, whose privacy guarantee is shown in Proposition 4.1. Proposition 4.1 (Beimel al. (2008)). Setting p T Ran suffices for (e. -idfifeerett privacy.
When each agent is equipped with such pnivacy-preserre sender RRR while adhering to the originally optimal strategy (i.e., believing what others tell and doing the guess), all agents would make an inaccurate guess. The bias of the guess denoted as err, caused by RRR is then
p(N 1)
err; = Elgi]
Σ by
Σ A
Elay
b,) D
D
Σ by- J
Without any priori knowledge, the bias could not be reduced for (c, DDD algorithms. However, the probability p of perturbation is set prior common knowledge for all agents before the game
4

---

Under review as a conference paper at ICLR 2023
starts, the story will be different. One could transform the biased guess into
ge = ARR ( = 1
1
Σ jf
وの (N - 11p/22
IND ] denote the messages received by agent i. Then the where .-; = [.1,.... 24-1, estimate will be unbiased as
p(N 1) +b; Σ b
[
E
zj
2
This example inspires that a communication algorithm could be both cient. From the perspective of privacy. by the post-processing lemma of DP, any post-processing does not affect the original privacy level. From the perspective of utility, we could eliminate the bias err if the agent is equipped with the receiver ARR and the prior knowledge p is given.
and effi-
In general, our motivating example gives two principles for designing privacy-preserving communi- cation frameworks. First, to prevent the sensitive information from being inferred by other curious agents, we equip each agent with local message sender with certain privacy constraints. Second, given a priori knowledge about the privacy requirement other agents, the receiver could strategi- cally analyze the received noisy messages to statistically reduce error due to the noisy communica- tion. These two design principles correspond to two parts of our DPMAC framework respectively, i.., privacy--reservee local sender and privacy-aware receiver.
METHODOLOGY
Based design principles, now introduce our DPMAC framework, shown in Figure 1. Our framework is general and flexible, which makes compatible with any CTDE method.
5.1 PRIVACY-PRESERVING LOCAL SENDER WITH STOCHASTIC GAUSSIAN MESSAGES
In this section, we present the sender's perspective on the privacy guarantee. At time for agent i, message function fe is used to generate a message for communication. f. takes a subset of transitions local trajectory T/ input, where the subset sampled uniformly without replacement from T. (denote the sampling rate as Ti). This message is perturbed by the Gaussian mechanism with variance of (Dwork, 2006). Agent then samples subset of other agents to share this message (denote the sampling rate as 72). The following theorem guarantees differential privacy.
Theorem 5.1 (Privacy guarantee for DPMAC), Let 7I, 72 E (0, 1), and C be the l2 norm of the message functions. For any 8 > O and privacy budget c,, the communication of agent satisfies (G,, 5)-D when of 14 ifwe have a with 3 E (0,1) 1) and 02 = 0.7.
log +1 < 20 log (1/7aa (1 o²2)) /3+1 c.11-
Ge,
With Theorem 5. one can directly translate non-private MARL with communication algorithm into a private one. However, as we shall see in our experiment section, directly injecting the pri- vacy noise into existing MARL with communication algorithms may lead serious performance degradation. In fact, the injected noise might jeopardize the useful information incorporated in the messages, or even leads meaningless messages. To alleviate the negative impacts of the injected privacy noise on the cooperation between agents, we adopt stochastic message sender in the sense that the messages sent by our sender are sampled from a learned message distribution. This makes DPMAC different from existing works in MARL that communicate through deterministic messages (Sukhbaatar et al., 2016; Foerster et al., 2016; Jiang & Lu, 2018; Das et al., 2019; Ding et al., 2020; Kim et al., 2021).
In the following, we drop the dependency of parameters on t when it is clear from the context. Without loss of generality, let the message distribution be multivariate Gaussian and let pa be the message sampled from the message distribution N (H4 Σ,), where μ4 = f (o,, a;; fe)) and Σ, S (oi, og; 0+) are the mean vector and covariance matrix learned by the sender, and 0r and 07 are the
5

---

Under review as a conference paper at ICLR 2023
Frhwebut
Receiver
Sender
M4
K
Decoder
Encoder
Vessage
Message
T
Msi
i
01
Priatted
Mii
Eov.
mil
Add
Figure 1: The overall structure of DPMAC. The message receiver of agent integrates other agents' messages {mza myi m with the self-attention mechanism and the integrated message fed into the policy n, together with the observation o;, Agent i interacts with the environment taking action ag- Then o and O₂ are concatenated and encoded by a privacy- peeeaccypeeesrvingg message sender and sent other agents.
parameters of the sender's neural networks. Then 6."T and 6: will be optimized towards making all the agents send more effective messages encourage better team cooperation and gain higher team rewards. For notational convenience, let 0 [ A. TT Then the sent privatized message m, p; + u, where
(0, is additional noise. It is clear that m, (N,, Σ; + of since p: is independent from ug. Counterfactually, let my (M,, Σd), where na = (os, (os,an, 0z; 0.") and Σ = S (o,, 04; 0!')) is the sent message when it was under any privacy constraint.
Let the optimal message distribution be N Σj). We are interested to characterize 8:¹ and 6; By the optimality of pº, Eg,
0 = arg min DEL (NL(4/444 E Ef))
arg min log
+ tr(E:-
EX + 1144 147
Then under the privacy constraints, the stochastic sender will learn 0f such that
0% arg min DKI(N(ML E, + of E2))
EEi E + ooaa
arg min log
+ tr oft,) + ||: pb -
Through Equation (2), it is possible to directly incorporate the distribution of privacy noise into the optimization process of the sender help to learn 0 such that (xI(N + of Σ)) < (NL(,, E() " E) which means that the sender could learn to send private message m, = Ps + u, that is at least as effective as the non-privant con-private message m'. this manner, the performance degradation is expected to be well alleviated.
5.2 PRIVACY-AWARE MESSAGE RECEIVER
As shown our motivating example, the message receiver with knowledge a priori could statisti- cally reduce the communication error in privacy-preserving scenarios. In the practical design, this motivation could be naturally instantiated with the gradient flow between the message sender and the message receiver.
Specifically, agent i first concatenates all the received privatized messages as m- = { 1 and then encodes m(-1j; into an aggregated message the decoding function f parameterized by pr Then similar argument to the policy gradient theo- rem (Sutton al., 1999) states that the gradient of the receiver
f fm (7)) with
or (0 Er.oa [E,, Ver Jo (q: m(-))) Va, log n, (a: lo,9))

---

Under review as a conference paper at ICLR 2023
where [( E Λ| is the cumulative discounted reward from the starting state. In this way, the receiver could utilize the prior knowledge o; of the privacy- preeeviir sender encoded in the gradient during the optimization process. Please refer Appendix D for the detailed optimization process of the message senders and receivers.
PRIVACY-PRESERVING EQUILIBRIUM ANALYSIS
Many cooperative multi-agent games enjoy the existence of unique NE, which ensures the vergence of iterative algorithms. Under the privacy constraints, however, the existence of a unique longer be guaranteed even if the original game admits unique equilib- rium. As the convergence of MARL algorithms could depend on the existence of an equilibrium, we investigate such existence in single-step games and extend the result to multi-step games.
Nash equilibrium
6.1 SINGLE-STER GAMES
We study a class of two-player collaborative games, denoted as collaborative game with privacy (CGP). The game involves two agents, each equipped with privacy parameter Pn. n {1.2). The value of Ps, represents the importance of privacy to agent n, with the larger value referring to greater importance. Let M be some message mechanism. We denote the privacy loss by eM (p.).. which measures the quantity of the potential privacy leakage and is formally defined in Definition B.2. Besides, let (V,, V.M (pI.P2) be the utility gained by measuring the gap between private value function VrM (p1. P2) and non-private value function Vn. Then the trade-off between the utility and the privacy depicted by the total utility function un (21,P)) in Equation (3). The formal definition of CGP is given in Definition 6.1. See more details in Appendix B.I.
Definition 6.1 (Collaborative game with privacy (CGP)). The collaborative game with privacy is denoted by a tuple (N, N,,,,,, where N = {1,2] is the the set of players, Σ, = (pi,p2) is the action set with p1. P2 E [0, representing the privacy level, and U = {u1, is the set of utility functions satisfyyng Vn N,
(p1, (21.P2) = B, b (V,, V.M (pI, P2)) C.M .M..MMMp (M())
(3)
Then the following theorem shows that if changes in the value function of each player can be ex- pressed as a change in their own privacy parameter, then CGP is a potential game and pure NE thereafter exists. The proof is deferred to Appendix B.1.
Theorem 6.1 (CGP's NE guarantee). The collaborative game with privacy has at least one non- trivial strr-strategy Nash equilibrium if 0r, V1 = 0% V2, Vi E (1,2].
Equilibrium in single round binary sums Let revisit our motivating example. Armed with the CGP framework, it is immediate that the single round binary sums game guarantees the existence of a NE. This result is formalized in Theorem B.2 in Appendix B.I.
6.2 MULTI-STEP GAMES
We now consider an extended version of single round binary sums named multiple round sums. Consider an N- N-player game where player i owns a saving Fy,t- Rather than sending a binary bit, the agent can choose to give out bi at round l. Meanwhile, each player i selects privacy level pi and sends messages to each other with sender f encoding the information of bi, with the privacy level Pi,t- The reward of the agent is designed to find good trade-off between privacy and utility. The setting of the game is thus similar to the empirical implementation of DPMAC.
We first transform this game into Markov potential game (MPG), with the reward of each agent transformed into a combination of the team reward and the individual reward. Then with existing theoretical results from Macua et al. (2018), we present the following result while deferring its proof to Appendix B.2.
Theorem 6.2 (NE guarantee in multiple round sums). If Assumptions 1, 2, 3, 4 (see Appendix B.2) are satisfied, our MPG has a NE with potential function J defined as,
J n (xx)) = Σ ((1 - Paa)bba + ax + Bpp.))
(4)
JE[N]
7

---

Under review as a conference paper at ICLR 2023
Cosporatiive Navigation
Cooperative Communicotonn Novigation
Prodator Proy
Avorggs Rovands
-140
Avaragu Edtiooo Rowards
-30
115
-33
-150
-20
-10
-170
-45
-130
DTMAC
-30
DPMAC
-30
DPPACC 5.000
12C
12C
RCC
-190
MADOPG
HA0DPN
MAD3PG
-20
-80
3.0
0.6
0.0
5.2
Steps
366
Steps
Steps
166
(a) Learning curves of DPMAC, TarMAC, I2C, and MADDPG on three MPE tasks. Note that on the PP task DPMAC (e 0.10) is shown.
Cooperative
Cosperat Nevigation
Predston-Prey
-140
-100
Aveerag Spstnnde RRewwards
Aroraga Sptoode Rowards
-50
-40
-200
-200
-0
-240
OPMACIE 0.201
DCtg 0.13)
12C1 0.301
DPMAGU 9.12)
-120
DOL 9.10)
DFMACC 6.30)
MADOPG
MAODHN
MADDFG
-250
Tanze 513)
Tarte 0101
Tonte 0.30)
-220
(b) Learning curves of different algorithms under the privacy budget 0.10. MADDPG (non (non-private) is also displayed for comparison.
Cosperstive Nav gstion
Cooperat we Communicat on Nevigstion
Predstor-Prey
-15
-20
-140
Reworts
-150
Avee Enprade Sornnett
Avi ogge Epioode Rowerrs
-40
-130
-32
-20
-80
-70
-200
-40
-90
3.5
0-2
0.4
0.5
0.
1.0
0.0
0.2
0.4
0.6
1.5
3.0
0.2
0.4
0.5
0.5
1.0
Steps
Steps
Staps
(c) Learning curves different algorithms under the privacy budget displayed for comparison.
MADDPG (non-priven (ion-private) is also
Cosperative Nev geiion
Cuoparatiog vw Communical Neviaation
Prediator-PPrry
-140
-12
-100
Avorag Rovands
Avaragu Edizoda Rlovands
-180
--2
Avorggs
200
-100
-220
DPSSA DPMACIe
DPM6C
-120
DFMIAC
-240
DPMACIE-
-59
DPMACUE 0.031
DPMACIZ 0.12)
DPMACI C.03)
DPMACIE 2.10)
-140
-200
3.0
0.2
1.0
0.0
5.5
1.0
0.3
08
Steps
100
Steps
1c5
Steps
200
(d) Learning curves of different privacy budgets (e = 0.01, 0.10, 1.00) for DPMAC.
Figure 2: Learning curves of DPMAC and baseline algorithms. The curves are averaged over seeds. Shaded areas denote 1 standard deviation.
8

---

Under review as a conference paper at ICLR 2023
7 EXPERIMENTS
In this section, we present the experiment results and corresponding experiment analyses. Please see Appendix G for more detailed analyses of experiment results.
Baselines We implement our DPMAC and evaluate it against TarMAC (Das al., 2019), I2C (Ding et al., 2020), and MADDPG (Lowe et al., 2017). All Algorithms are tested with and without the privacy requirement except for MADDPG, which involves no communication among agents. Since TarMAC and I2C do not have local sender and have no DP guarantee, we add Gaussian noise to their receiver according to the noise variance specified in Theorem 5.1 for a fair comparison. Please see Appendix D for more training details. We remark that the code will be made publicly available once this manuscript is accepted.
Environments We evaluate the algorithms on the multi-agent particle environment (MPE) (Mor- datch & Abbeel, 2017), which with continuous observation and discrete action space. This en- vironment is commonly used among existing literature (Lowe et al., 2017: Jiang & Lu, 2018; Ding et al., 2020; Kim et al., 2021). We evaluate a wide range tasks MPE, including cooperative navigation (CN), cooperative communication and navigation (CCN), and predator prey (PP). More details on the environmental settings are given in Appendix E.
Experiment results without privacy DPMAC is first compared with TarMAC, I2C, and MAD- DPG on three MPE tasks without the privacy requirement, as shown in Figure 2a. DPMAC outper- forms baselines on CCN & PP and has competitive performance on CN. Note that for the PP task we pick DPMAC with = 0.10 due to even better performance over its non-private variant. The comparison between DPMAC (non-private) and baselines is provided in Appendix F.
Experiment results with privacy We further add the privacy constraint on the communication algorithms. We set 5 10-4 all tasks. Figure 2b and Figure 2c show the performance under the privacy budget- e = 0.10, 6 = 1.0 and both with 5 = 10-1 We include MADDPG as a non-communication bascline method. We observe that DPMAC with the privacy requirement could still maintain good result compared to MADDPG, while the performance of TarMAC and I2C drops greatly. Figure 2d further gives the comparison between the performance of DPMAC under different privacy budgets. When = 0.01, DPMAC still gains remarkable performance, while other baselines' performance degraded greatly, as shown in Figure 2b.
Variance adjustment of DPMAC Experiments with privacy also support our claim that DPMAC could automatically adjust the variance of our stochastic message sender so that learns a noise- robust representation. As shown in Figure 2d, DPMAC gains very close performance when = 0.1 and e 1.0, though the privacy requirements of magnitude. However, one can see large gaps for the same baseline algorithms under different from Figure 2b and Figure 2c. Please see Figure 4 and Figure 5 for direct presentations of these gaps.
0.1 and e 1.0 differ by one order of
8 CONCLUSION
In this paper, we study the privacy-preserving communication in MARL. Motivated by simple yet effective example of the binary sums game, we propose DPMAC, a new efficient communicating MARL algorithm that preserves agents' privacy through differential privacy. Our algorithm is justi- fied both theoretically and empirically. Besides, to show that the crivacy-preserving communication problem is learnable, we analyze the single-step game and the multi-step game via the notion of Markov potential games (MPG) and show the existence of the Nash equilibrium. This existence further implies the learnability of several instances of MPG under privacy constraints. Extensive ex- periments are conducted on MPE and show the effecttivnness of DPMAC when compared to bascline methods on multiple tasks both with and without the privacy constraints.
Though we make the first step to establish efficient MARL algorithm with differential private communication, some interesting questions remain open. The first question is that is still unclear for whether there exists the Nash equilibrium in private competitive games. Besides, on the em- pirical side, investigating the performance of DPMAC in competitive games with communication might also be interesting and valuable.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martin Abadi, Andy Chu, lan Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, 2016. 2
Mário S Alvim, Miguel E Andrés, Konstantinos Chatzikokolakis, Pierpaolo Degano, and Catuscia Palamidessi. Differential privacy: on the trade-off between utility and information leakage. In International Workshop on Formal Aspects in Security and Trust, pp. 39-54. Springer, 2011. 2
Borja Balle, Maziar Gomrokchi, and Doina Precup. Differentially private policy evaluation. In International Conference Machine Learning, pp. 2130-2133 PMLR, 2016. 3
Amos Beimel, Kobbi Nissim, and Eran Omri. Distributed private data analysis: Simultaneously solving how and what. In David A. Wagner (ed.), Advances in Cryptology CRYPTO 2008, 28th Anmual International Cryptology Conference, Santa Barbara, CA, USA, August 17-21, 2008, Proceedings, volume 5157 of Lecture Notes in Computer Science, pp. 451-468, Springer, 2008.
Dingfan Chen, Tribhuvanesh Orekondy, and Mario Fritz. GS-WGAN: A gradient-sanitizee ap- proach for learning differentially private generators. In Advances in Neural Information Process- ing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, viriual, 2020. 2
Albert Cheu. Differential privacy in the shuffle model: A survey of separations. arXiv preprint 1xXv:2107.11133) 2021. 4
Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In Proceedings of the 36th Interna- fional Conference Machine Learning, ICML, 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 1538-1546. PMLR, 2019. 1, 2,
Ziluo Ding, Tiejun Huang, and Zongqing Lu. Learning individually inferred communication for multi-agent cooperation. Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. 1, 1,5,9,
Tao Dong, Xiangyu Bu, and Wenjie Hu. Distributed differentially private average consensus for multi-agent networks by additive functional laplace noise. Journal of the Franklin Institute, 357 (6):3565-355 -3544, 2020.
Abhimanyu Dubey. No-regret algorithms for private gaussian process bandit optimization. In The 24th International Conference on Artificial Intelligence and Statistics. AISTATS 2021, April 13- 15, 2021, Virtual Event, volume 130 of Proceedings Machine Learning Research, pp. 2062- 2070. PMLR, 2021. 3
Abhimanyu Dubey and Alex 'Sandy' Pentland. Differentially-prives federated linear bandits. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Informa- fion Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Cynthia Dwork. Differential privacy. In International Colloquiun Automata, Languages, and Programming, pp. 1-12. Springer, 2006. 2,4,5
Cynthia Dwork, Aaron Roth, al. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci., 9(3-4):211 1-)222111177, 2014. 2, 14
Jakob N. Foerster, Yannis M. Assael, Nando de Freitas, and Shimon Whiteson. Learning to commu- nicate with deep multi-agent reinforcement learning. In Advances in Neural Information Process- ing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 2137-2145, 2016. 3,5,
10

---

Under review as a conference paper at ICLR 2023
Evrard Garcelon, Vianney Perchet, Ciara BukeeBuree, and Matteo Pirotta. Local differential privacy for regret minimization reinforcement learning. In Advances Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 10561-10573, 2021. 3
Muneeb UI Hassan, Mubashir Husain Rehmani, and Jinjun Chen. Differential privacy techniques for cyber physical systems: A survey. IEEE Commun. Surv. Tutorials, 22(1):746-789, 2020.
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In 2015 aaai fall symposium series, 2015. 18
Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent cooperation. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Infor- mation Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp- 7265-7275, 2018.
5.9
Woojun Kim, Jongeui Park, and Youngchul Sung. Communication in multi-agent reinforcement learning: Intention sharing. In International Conference on Learning Representations, 2021. 1, 3, 5, 9, 19
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. 18
Valli Kumari and Srinivasa Chakravarthy. Cooperative privacy game: a novel strategy for preserving privacy in data publishing. Human-centric Computing and Information Sciences, 6(1):1-20, 2016. 17
Stefanos Leonardos, Will Overman, Ioannis Panageas, and Georgios Piliouras. Global convergence of multi-agent policy gradient markov potential games. arXiv preprint arXiv:2106.01969, 2021. 2. 15, 17
Ziyan Li, Quan Yuan, Guiyang Luo, and Jinglin Li. Learning effective multi-vehicle cooperation at unsignalized intersection via bandwidlth-constrineed communication. In 94th IEEE Vehicular Technology Conference, VTC Fall 2021, Norman, OK, USA, September 27-30, 2021, pp. 1-7.
IEEE, 2021. doi: 10. SAW.TTCC2221-Fal5222222 2021992577
Toru Lin, Jacob Huh, Christopher Stauffer, Ser-Nam Lim, and Phillip Isola. Learning to ground multi-agent communication with autoencoders. Advances Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 15230- 15242, 2021. 3
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed aeperaaive-competttie environments. Advances Neural Information Pro- cessing Systems 30: Annual Conference on Neural Information Processing Systems 2017, De- cember 4-9, 2017, Long Beach, CA, USA, pp. 6379-6390, 2017. 2, 9, 18,
Sergio Valcarcel Macua, Javier Zazo, and Santiago Zazo. Learning parametric closed-loop poli- cies for markov potential games. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Truck Proceedings. OpenReview.net, 2018. 2, 15,
Ilya Mironov. Rényi differential privacy. In 2017 IEEE 30th Computer Security Foundations Sym- posium (CSF), pp. 263-275. IEEE, 2017. 14
Dov Monderer and Lloyd S Shapley. Potential games. Games and economic behavior, 14(1):124- 143, 1996. 2, 15, 16
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. arXiv preprint arXiv: 00498, 2017.
John Nash et al. Equilibrium points in n-person games. Proceedings of the national academy of sciences, 36( 36(1):48 49, 1950. 15, 17
11

---

Under review as a conference paper at ICLR 2023
Balazs Pejo, Qiang Tang, and Gergely Biczók. Together alone: The price of privacy in collabora- tive learning. Proc. Priv. Enhancing Technol., 2019(2):47 65, 2019. 15, 16, 17
Peng Peng, Quan Yuan, Ying Wen, Yaodong Yang, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent bidrectionally-cordiiaacc nets for learning play starcraft combat games. CoRR, 1bs// 2017.
Murtaza Rangwala and Ryan Williams. Learning multi-agent communication through structured attentive reasoning. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: An- mual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. 3
Jun Sakuma, Shigenobu Kobayashi, and Rebecca N Wright. Privacy-preserving reinforcement learn- ing. In Proceedings of the 25th international conference on Machine learning, pp. 864-871, 2008.
Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. CoRR, abs/1610.03295, 2016a. URL URtp::////axxi abs/166 /oco/1610.. 03295.
Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. CoRR, abs/ abs//110,,3295, 2016b.
Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when communicate at scale in multiagent cooperative and competitive tasks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Learning multiagent communication with backpropagation. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Anmual Confer- ence on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 2244 2252, 2016. 3,5
Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural In- formation Processing Systems 12, [NIPS Conference, Denver, Colorado, USA, November 29- December 4, 1999], pp. 1057-1063. The MIT Press, 1999.
Youming Tao, Yulian Wu, Peng Zhao, and Di Wang. Optimal rates of (locally) differentially private heavy-tailed multi-armed bandits. In Proceedings of The 25th International Conference on Arti- ficial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pp- 1546 1546-1574. PMLR, 28-30 Mar 2022. 3
Jay Tenenbaum, Haim Kaplan, Yishay Mansour, and Uri Stemmer. Differentially private multi- armed bandits in the shuffle model. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp.
2021.
Qi Tian, Kun Kuang, Baoxiang Wang. Furui Liu, and Fei Wu. Multi-agent communication with graph information bottleneck under limited bandwidth. CoRR, abs/2112.10374, 2021. 3
Aristide Charles Yedia Tossou and Christos Dimitrakakis. Achieving privacy in the adversarial multi-armed bandit. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pp. 2653-2659, AAAI Press, 2017. 3
Mycal Tucker, Julie Shah, Roger Levy, and Noga Zaslavsky. Towards human- communication via the information bottleneck principle. CORR, 00./2077000880 2022. doi: 10.48550/arXiv. 2207.00088. 3
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural infor- mation processing systems, pp. 5998-6008, 2017. 18
12

---

Under review as a conference paper at ICLR 2023
Baoxiang Wang and Nidhi Hegde. Srivacy-presevving q-leaming with functional noise in continuous spaces. Advances in Neural Information Processing Systems, 32, 2019. 3
Lingxiao Wang, Bargav Jayaraman, David Evans, and Quanquan Gu. Efficient privacy-preserving nonconvex optimization. arXiv e-prints, pp. arXiv-1910, 2019. 14
Rundong Wang, Xu He, Runsheng Yu, Wei Qiu, Bo An, and Zinovi Rabinovich. Learning efficient multi-agent communication: An information bottleneck approach. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 9908-9918. PMLR, 2020a. 3
Tonghan Wang, Jianhao Wang, Chongyi Zheng, and Chongjie Zhang. Learning nearly decompos- able value functions via communication minimization. In 8th International Conference on Learn- ing Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020b. 1,3
Hua Wei, Nan Xu, Huichu Zhang, Guanjic Zheng, Xinshi Zang, Chacha Chen, Weinan Zhang, Yan- min Zhu, Kai Xu, and Zhenhui Li. Colight: Learning network-level cooperation for traffic signal control. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM 2019, Beijing, China, November 3-7, 2019, pp. 1913-1922. ACM, 2019.
Jiachen Yang. Alireza Nakhaei, David Isele, Kikuo Fujimura, and Hongyuan Zha. CM3: coopera- tive multi-goal multi-stage multi-agent reinforcement learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net, 2020.
Dayong Ye. Tianqing Zhu, Zishuo Cheng, Wanlei Zhou, and Yu Philip. Differential advising in multiagent reinforcement learning. IEEE Transactions on Cybernetics, 2020. 3
Chongjie Zhang and Victor R. Lesser. Coordinated multi-agent reinforcement learning in networked distributed pomdps. In Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelli- gence, AAAI 2011, San Francisco, California, USA, August 7-11, 2011. AAAI Press, 2011.
Sai Qian Zhang, Qi Zhang, and Jieyu Lin. Efficient communication in multi-agent reinforcement learning via variance based control. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 3230-3239, 2019. 3
Sai Qian Zhang, Qi Zhang, and Jieyu Lin. Succinet and robust multi-agent communication with temporal message control. In Advances Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. 3
Xin Zhang, Zhuqing Liu, Jia Liu, Zhengyuan Zhu, and Songtao Lu. Taming communication and sample complexities in decentralized policy evaluation for cooperative multi-agent reinforcement learning. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 18825-18838, 2021. 3
Canzhe Zhao, Yanjie Ze, Jing Dong, Baoxiang Wang, and Shuai Li. Differentially private tempo- ral difference learning with stochastic nonconvex www.-ssttoooooonccccoeee optimization, arXiv preprint 2004442001.10447, 2022. 3
Kai Zheng, Tianle Cai, Weiran Huang, Zhenguo Li, and Liwei Wang. Locally differentially private (contextual) bandits learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. 3
Xingyu Zhou Differentially private reinforcement learning with linear function approximation. Proc. ACM Meas. Anal. Comput. Syst. 6(1):811-8:2) 2022. 3
13


Reviews:
[{"review_id": "aIjHnPDW1c-", "reviewer": "Reviewer_mZfd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a novel differentially private communication algorithm for cooperative multi-agent reinforcement learning. Specifically, the authors proposed the differentially private multi-agent communication (DPMAC). The proposed algorithm protects the sensitive information of individual agents by equipping each agent with a local message sender with rigorous differential privacy guarantee. To achieve this, a stochastic message sender for each agent respectively with DP requirement into the sender. This can automatically adjusts the learned message distribution to alleviate the instability caused by DP noise. They also theoretically proved that the existence of a Nash equilibrium in cooperative MARL with privacy-preserving communication. To validate the proposed algorithm, the authors leveraged a decentralized navigation communication environment to show the superiority over the baseline methods.", "summary_of_the_review": "This paper presents a novel differentially private communication algorithm for cooperative multi-agent reinforcement learning. The authors develop a privacy-preserving local sender and privacy-aware receiver for individual agents. They also mathematically analyze the privacy guarantee and the equilibrium analysis. The experimental results look promising to outperform the baseline methods, but they are based on a simple environment. Overall I think the draft looks technically sound and solid, but some additional work are required to clarify and validate the theory.", "strength_and_weaknesses": "Strength: \n\nThis paper develops a novel algorithm for MARL to cope with differential privacy in individual agents. The authors design new mechanism, privacy-preserving local sender and privacy-aware receiver. The mechanism looks technically sound to protect sensitive information for agents. The authors also theoretically prove the differential privacy guarantees and analyze two scenarios of games to show the Nash equilibrium. The experimental results are also promising to demonstrate the outperforming capability, though the environment is simple. Overall, the paper is well written and easy to follow.\n\nWeaknesses:\n\nTypically, MARL suffers from high computation and communication overhead. This work fails to show the complexities, which is a weakness. Particularly, to handle DP issue, it would be great to see how those complexities are affected for the proposed DPMAC framework. Additionally, the analysis for the equilibrium analysis seems not sufficient. I would suggest the authors have more intuitive discussion after the theorem statement. \n\nMoreover, it is better to see algorithm framework in the main contents. Based on the current work, it is not clear what exactly underlying algorithm is. Is it Q-learning based or policy gradient-based? As MARL covers a large spectrum of different algorithm, such as the MADDPG in the comparison. Or DPMAC can be used with any centralized counterpart?\n\nHow can we tell from the experimental results that the Nash equilibrium has been achieved?\n\nThough the experimental results look convincing, the environment is way too simple. Results based on more complex environment can support better the theory.\n\n*************************Post-rebuttal*************************\nI appreciate the responses from the authors. After carefully reviewing the rebuttal, I would just keep my current score.\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SIlKQtTEWro", "reviewer": "Reviewer_qrFk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work focused on multi-agent reinforcement learning with a privacy guarantee. The author first shows that a privacy-aware receiver could \nalleviate the instability caused by random Gaussian noise. Based on this idea, the author proposes the differentially private multi-agent communication (DPMAC) framework and proves the existence of the Nash equilibrium. In addition, the result of the experiments shows that\nDPMAC outperforms other algorithms in privacy-preserving scenarios and gains competitive performance in non-private scenarios.", "summary_of_the_review": "In summary, this work first studied multi-agent reinforcement learning with a privacy guarantee and proposed a novel algorithm. The experiment results support the efficiency of the DPMAC algorithm. However, the motivating example cannot provide enough insights for the algorithm, and a minor issue exists with the differential privacy guarantee.", "strength_and_weaknesses": "Strength:\n\nThe author provided the first analyses on multi-agent reinforcement learning with a privacy guarantee and proposed a novel framework (DPMAC). In addition, both the theoretical results and the experiments show that the proposed algorithm is efficient and can provide a differential privacy guarantee.\n\nWeakness:\n\n1. The motivating example could not provide enough insights for the algorithm. In this example, the bit $b_i$ should be generated from {0,1} rather $[0,1]$. Though this example shows that a privacy-aware receiver could alleviate the instability caused by random Gaussian noise, it seems difficult to extend the result to a continuous situation and can only provide limited insights for the framework.\n\n2. The proof of Theorem 5.1 have some minor issue. In detail, the differential privacy guarantee is based on Lemma A.1 and Propositional A.1, which require the function $f_i$ to be fixed. However, the DPMAC algorithm's parameter $\\theta_i$ is optimized with the gradient method. Thus, the function $f_i$ may be related to the message, and in this situation, previous analyses cannot provide a guarantee for differential privacy.\n\n3. For the related work, the author mentions Zhou (2022) subsequently takes the first step to establish the sublinear regret in linear mixture Markov decision processes (MDPs). However, it seems the author is missing a concurrent work by Liao(2022), which also provided a differential guarantee in linear mixture MDP and was posted on arXiv 1.5 months before Zhou (2022).\n\nLiao C, He J, Gu Q. Locally differentially private reinforcement learning for linear mixture markov decision processes", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GM6fDH7_aA", "reviewer": "Reviewer_oYeH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies communication in multiagent reinforcement learning that preserves privacy. It adopts the concept of differential privacy to approach the problem and proposes an algorithm that is able to protect individual agents' information privacy. The authors also use a stochastic message sender for each agent to address the instability issue. The authors also prove the existence of a Nash equilibrium in the cooperative multiagent reinforcement learning model. Theoretical results are evaluated empirically.", "summary_of_the_review": "In summary, a paper motivated by an interesting idea but lacking clarity at some parts as well as convincing motivating applications. ", "strength_and_weaknesses": "Strength:\n\nThe idea of considering differential privacy in MARL setting is interesting and novel.\n\nWeakness:\n\nSome parts of the paper is not clear. For example, how does the MARL process look like when agents communicate with each other. Do they send messages to every other agent at every time step before they take an action? How many messages can be sent by each of them in a time step? What is the sequence of the messages and the order of message sending? \n\nThe methods described in Section 5.1 also seem to lack some necessary details, so it's hard to follow why the comparison between Equations (1) and (2) explains the claim that the method alleviates the performance degradation issue.  When the authors say the sender are sampled from a learned message distribution, it is also unclear based on what the message distribution is learned. (Maybe this has to do with the lack of clarity about the MARL process.)\n\nIn addition to the clarity issue, it's a bit hard to think of a scenario where the agents are fully cooperative and care about their privacy at the same time. The authors mentioned autonomous driving, but I think autonomous driving is hardly a fully cooperative scenario (where agents share exactly the same reward function). \n\nMinor issue: What is $\\mathcal{R}$ in Definition 3.1? In the previous paper, $\\mathcal{R}$ is the reward function.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BRNVyM8mUBL", "reviewer": "Reviewer_NQa3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the private-preserving problem in multi-agent cooperative communication. It proposes a differentially private multi-agent communication (DPMAC) algorithm, which protects the sensitive information of individual agents by adding noise when sending messages to others with rigorous $(ε, δ)$-differential privacy (DP) guarantee. Each agent is equipped with a sender that sends its partial observation disturbed by stochastic noise, and a receiver integrates all other agents’ messages. It proves the existence of the Nash equilibrium for cooperative games with privacy-preserving communication. Experiments on the MPE show that DPMAC clearly outperforms other algorithms in privacy-preserving scenarios and gains competitive performance in non-private scenarios.", "summary_of_the_review": "This work firstly proposes the private-preserving problem in multi-agent cooperative communication, it is original. Theoretical analysis is sufficient on privacy guarantee and Nash's existence. However, experiments are weak and do not prove the algorithm can protect the privacy.\n", "strength_and_weaknesses": "Strength\n1. This paper is the first attempt to develop a MARL private communication framework, with a theoretical analysis on the private guarantee.\n2. Experiments on MEP show proposed DPMAC gains comparable performance under non-private settings and better performance than other privatized baselines under private setting.\n \nWeaknesses:\n1. However, explanations of experiment results are lacking, for example why DPMAC with privacy constraints gains better performance on Predator-Prey than that without privacy constraints? What is the main conclusion on the ablation study of hyperparameters \\epsilon? The results on three environments seem not consistent, the hyperparameter \\epsilon has a significant impact only on predator-prey when training steps up to 1e6.\n2. It would be better to verify the privacy-preserving property in a more rational scenario that naturally possesses private requirements; agents in an MPE environment seem not to have the necessity of privacy-preserving.\n3. I wonder if it is necessary to do experiments on whether real private information (such as personal living routines) can be protected.\n4. Why the Nash existence is important in a cooperative game, the Nash does not always correspond to the solution with the greatest payoff in a cooperative game. And the first two lines in section 6 clarify that the “Many cooperative multi-agent games enjoy the existence of a unique NE”, but this paper proves the existence of NE instead of unique NE. It would be better to unify these statements.\n5.The theoretical analysis on Nash existence is given, but it would be better to conduct experiments about NE existence for completeness.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject