Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the private-preserving problem in multi-agent cooperative communication. It proposes a differentially private multi-agent communication (DPMAC) algorithm, which protects the sensitive information of individual agents by adding noise when sending messages to others with rigorous $(ε, δ)$-differential privacy (DP) guarantee. Each agent is equipped with a sender that sends its partial observation disturbed by stochastic noise, and a receiver integrates all other agents’ messages. It proves the existence of the Nash equilibrium for cooperative games with privacy-preserving communication. Experiments on the MPE show that DPMAC clearly outperforms other algorithms in privacy-preserving scenarios and gains competitive performance in non-private scenarios.
strength_and_weaknesses: Strength
1. This paper is the first attempt to develop a MARL private communication framework, with a theoretical analysis on the private guarantee.
2. Experiments on MEP show proposed DPMAC gains comparable performance under non-private settings and better performance than other privatized baselines under private setting.
 
Weaknesses:
1. However, explanations of experiment results are lacking, for example why DPMAC with privacy constraints gains better performance on Predator-Prey than that without privacy constraints? What is the main conclusion on the ablation study of hyperparameters \epsilon? The results on three environments seem not consistent, the hyperparameter \epsilon has a significant impact only on predator-prey when training steps up to 1e6.
2. It would be better to verify the privacy-preserving property in a more rational scenario that naturally possesses private requirements; agents in an MPE environment seem not to have the necessity of privacy-preserving.
3. I wonder if it is necessary to do experiments on whether real private information (such as personal living routines) can be protected.
4. Why the Nash existence is important in a cooperative game, the Nash does not always correspond to the solution with the greatest payoff in a cooperative game. And the first two lines in section 6 clarify that the “Many cooperative multi-agent games enjoy the existence of a unique NE”, but this paper proves the existence of NE instead of unique NE. It would be better to unify these statements.
5.The theoretical analysis on Nash existence is given, but it would be better to conduct experiments about NE existence for completeness.

clarity,_quality,_novelty_and_reproducibility: Clarity:
* The writing of section 5 is hard to understand. There are many notations in section 5.2, but the definition of these seems not complete and not clear. For example, what $m_i,p_i,u_i$, and $\mu_i$ indicate?
* What’s the relationship between equations (1) and (2), (1) hope to learn an unconstrained message sender, (1) hope to learn a constrained message sender, right? And why the inequality about KL- divergence holds in the third line under eq. (2)?
* I am confused about this description that in the second line of page 7, “utilize the prior knowledge $\sigma_i$ ”, the  $\sigma_i$ seems not defined as “prior knowledge” hereinbefore， it is defined as Gaussian noise, and it has not appeared in the equation at the last line of page 6.
* There exist many unclear descriptions in the experiments section, for example, fig 2b corresponds to $\epsilon=0.1$ instead of $\epsilon=0.01$ as in the paper.

Quality: Theoretical analysis is sufficient on privacy guarantee and Nash existence, but the experiments especially the explanation of the results are lacking.

Novelty: This paper firstly proposes the problem of private-preserving communication in cooperative MARL. It is original, however, the experiments are not conducted in environments with private-preserving requirements, thus I am concerned about the effectiveness of the proposed algorithm on privacy protection.

Reproducibility: The overall structure of the message sender and receiver is in the form of equations, but the detailed setting of the sender under privacy and non-privacy constraints is not clear to me. It would be clearer to give a pseudocode of the framework.

summary_of_the_review: This work firstly proposes the private-preserving problem in multi-agent cooperative communication, it is original. Theoretical analysis is sufficient on privacy guarantee and Nash's existence. However, experiments are weak and do not prove the algorithm can protect the privacy.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a novel differentially private communication algorithm for cooperative multi-agent reinforcement learning. Specifically, the authors proposed the differentially private multi-agent communication (DPMAC). The proposed algorithm protects the sensitive information of individual agents by equipping each agent with a local message sender with rigorous differential privacy guarantee. To achieve this, a stochastic message sender for each agent respectively with DP requirement into the sender. This can automatically adjusts the learned message distribution to alleviate the instability caused by DP noise. They also theoretically proved that the existence of a Nash equilibrium in cooperative MARL with privacy-preserving communication. To validate the proposed algorithm, the authors leveraged a decentralized navigation communication environment to show the superiority over the baseline methods.
strength_and_weaknesses: Strength: 

This paper develops a novel algorithm for MARL to cope with differential privacy in individual agents. The authors design new mechanism, privacy-preserving local sender and privacy-aware receiver. The mechanism looks technically sound to protect sensitive information for agents. The authors also theoretically prove the differential privacy guarantees and analyze two scenarios of games to show the Nash equilibrium. The experimental results are also promising to demonstrate the outperforming capability, though the environment is simple. Overall, the paper is well written and easy to follow.

Weaknesses:

Typically, MARL suffers from high computation and communication overhead. This work fails to show the complexities, which is a weakness. Particularly, to handle DP issue, it would be great to see how those complexities are affected for the proposed DPMAC framework. Additionally, the analysis for the equilibrium analysis seems not sufficient. I would suggest the authors have more intuitive discussion after the theorem statement. 

Moreover, it is better to see algorithm framework in the main contents. Based on the current work, it is not clear what exactly underlying algorithm is. Is it Q-learning based or policy gradient-based? As MARL covers a large spectrum of different algorithm, such as the MADDPG in the comparison. Or DPMAC can be used with any centralized counterpart?

How can we tell from the experimental results that the Nash equilibrium has been achieved?

Though the experimental results look convincing, the environment is way too simple. Results based on more complex environment can support better the theory.

*************************Post-rebuttal*************************
I appreciate the responses from the authors. After carefully reviewing the rebuttal, I would just keep my current score.




clarity,_quality,_novelty_and_reproducibility: The clarity and quality of this paper are good. The theoretical novelty also seems decent. While the empirical evidences may require some more work to validate the theory.
summary_of_the_review: This paper presents a novel differentially private communication algorithm for cooperative multi-agent reinforcement learning. The authors develop a privacy-preserving local sender and privacy-aware receiver for individual agents. They also mathematically analyze the privacy guarantee and the equilibrium analysis. The experimental results look promising to outperform the baseline methods, but they are based on a simple environment. Overall I think the draft looks technically sound and solid, but some additional work are required to clarify and validate the theory.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies communication in multiagent reinforcement learning that preserves privacy. It adopts the concept of differential privacy to approach the problem and proposes an algorithm that is able to protect individual agents' information privacy. The authors also use a stochastic message sender for each agent to address the instability issue. The authors also prove the existence of a Nash equilibrium in the cooperative multiagent reinforcement learning model. Theoretical results are evaluated empirically.
strength_and_weaknesses: Strength:

The idea of considering differential privacy in MARL setting is interesting and novel.

Weakness:

Some parts of the paper is not clear. For example, how does the MARL process look like when agents communicate with each other. Do they send messages to every other agent at every time step before they take an action? How many messages can be sent by each of them in a time step? What is the sequence of the messages and the order of message sending? 

The methods described in Section 5.1 also seem to lack some necessary details, so it's hard to follow why the comparison between Equations (1) and (2) explains the claim that the method alleviates the performance degradation issue.  When the authors say the sender are sampled from a learned message distribution, it is also unclear based on what the message distribution is learned. (Maybe this has to do with the lack of clarity about the MARL process.)

In addition to the clarity issue, it's a bit hard to think of a scenario where the agents are fully cooperative and care about their privacy at the same time. The authors mentioned autonomous driving, but I think autonomous driving is hardly a fully cooperative scenario (where agents share exactly the same reward function). 

Minor issue: What is $\mathcal{R}$ in Definition 3.1? In the previous paper, $\mathcal{R}$ is the reward function.
clarity,_quality,_novelty_and_reproducibility: I find the paper very clear at some parts but hard to follow at others. The key idea and the approach proposed look novel.  
summary_of_the_review: In summary, a paper motivated by an interesting idea but lacking clarity at some parts as well as convincing motivating applications.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work focused on multi-agent reinforcement learning with a privacy guarantee. The author first shows that a privacy-aware receiver could 
alleviate the instability caused by random Gaussian noise. Based on this idea, the author proposes the differentially private multi-agent communication (DPMAC) framework and proves the existence of the Nash equilibrium. In addition, the result of the experiments shows that
DPMAC outperforms other algorithms in privacy-preserving scenarios and gains competitive performance in non-private scenarios.
strength_and_weaknesses: Strength:

The author provided the first analyses on multi-agent reinforcement learning with a privacy guarantee and proposed a novel framework (DPMAC). In addition, both the theoretical results and the experiments show that the proposed algorithm is efficient and can provide a differential privacy guarantee.

Weakness:

1. The motivating example could not provide enough insights for the algorithm. In this example, the bit $b_i$ should be generated from {0,1} rather $[0,1]$. Though this example shows that a privacy-aware receiver could alleviate the instability caused by random Gaussian noise, it seems difficult to extend the result to a continuous situation and can only provide limited insights for the framework.

2. The proof of Theorem 5.1 have some minor issue. In detail, the differential privacy guarantee is based on Lemma A.1 and Propositional A.1, which require the function $f_i$ to be fixed. However, the DPMAC algorithm's parameter $\theta_i$ is optimized with the gradient method. Thus, the function $f_i$ may be related to the message, and in this situation, previous analyses cannot provide a guarantee for differential privacy.

3. For the related work, the author mentions Zhou (2022) subsequently takes the first step to establish the sublinear regret in linear mixture Markov decision processes (MDPs). However, it seems the author is missing a concurrent work by Liao(2022), which also provided a differential guarantee in linear mixture MDP and was posted on arXiv 1.5 months before Zhou (2022).

Liao C, He J, Gu Q. Locally differentially private reinforcement learning for linear mixture markov decision processes
clarity,_quality,_novelty_and_reproducibility: This paper is well-written and easy to follow.
summary_of_the_review: In summary, this work first studied multi-agent reinforcement learning with a privacy guarantee and proposed a novel algorithm. The experiment results support the efficiency of the DPMAC algorithm. However, the motivating example cannot provide enough insights for the algorithm, and a minor issue exists with the differential privacy guarantee.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper looks at a multi-agent scenario where they have to communicate, as usual. The question of privacy then arises naturally and this paper provides nice answer to them.

The major concern that I have is that the contributions and techniques used are not that novel and rather well-understood now, So I am not sure that the ICLR community will gain a lot from that paper (with respect to the existing literature).

As a consequence, I am unfortunately not sure that the contributions of this paper are significant enough in its current form to be accepted at ICLR (but I have no doubt it will be accepted elsewhere)
justification_for_why_not_higher_score: The techniques are rather classical and well-known. We do not want to see thousands of paper on a privacy variant of this or that problem, with similar tools used, rather new insights and/or techniques and/or models
Decision: Reject