Title: Information Design in Multi-Agent Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: multi-agent reinforcement learning

multi-agent communication

information design

signaling gradient

obedience constraints
Abstract: Reinforcement learning (RL) is inspired by the way human infants and animals learn from the environment. The setting is somewhat idealized because, in actual tasks, other agents in the environment have their own goals and behave adaptively to the ego agent. To thrive in those environments, the agent needs to influence other agents so their actions become more helpful and less harmful. Research in computational economics distills two ways to influence others directly: by providing tangible goods (mechanism design) and by providing information (information design). This work investigates information design problems for a group of RL agents. The main challenges are two-fold. One is the information provided will immediately affect the transition of the agent trajectories, which introduces additional non-stationarity. The other is the information can be ignored, so the sender must provide information that the receiver is willing to respect. We formulate the Markov signaling game, and develop the notions of signaling gradient and the extended obedience constraints that address these challenges. Our algorithm is efficient on various mixed-motive tasks and provides further insights into computational economics. Our code is publicly available at https://github.com/YueLin301/InformationDesignMARL.

Information Design in Multi-Agent Reinforcement Learning

Abstract

Reinforcement learning (RL) is inspired by the way human infants and animals learn from the environment. The setting is somewhat idealized because, in actual tasks, other agents in the environment have their own goals and behave adaptively to the ego agent. To thrive in those environments, the agent needs to influence other agents so their actions become more helpful and less harmful. Research in computational economics distills two ways to influence others directly: by providing tangible goods ( mechanism design ) and by providing information ( information design ). This work investigates information design problems for a group of RL agents. The main challenges are two-fold. One is the information provided will immediately affect the transition of the agent trajectories, which introduces additional non-stationarity. The other is the information can be ignored, so the sender must provide information that the receiver is willing to respect. We formulate the Markov signaling game , and develop the notions of signaling gradient and the extended obedience constraints that address these challenges. Our algorithm is efficient on various mixed-motive tasks and provides further insights into computational economics. Our code is publicly available at https://github.com/YueLin301/InformationDesignMARL .

1 Introduction

Reinforcement learning (RL) studies how a world-agnostic agent makes sequential decisions to maximize utility. It gains increasing popularity and achieves milestone progress in Atari [46, 22], Go [48], Poker [8, 9, 39], video games [2, 6], and bioinformatics [26], economics [59], etc. The canonical RL formulation requires the environment to be stationary, meaning that other agents could not respond to the ego agent's policy by adapting their own policies [50]. The assumption does not hold in real applications. Instead, the ego agent, in general, cannot dictate other agents and needs to influence others so their adaptation becomes more helpful and less harmful.

A substantial subarea of RL, multi-agent reinforcement learning (MARL), investigates the interaction and influence among multiple RL agents when they are placed in a shared environment. It has obtained promising results thanks to the generality and diversity of its settings. Examples are pure cooperative games and pure competitive games. The pure cooperative setting studies agents that work on a consentaneous goal. In this case, influencing other agents is relatively straightforward. The pure competitive setting studies zero-sum games. In this case, influencing other agents is less likely to be practical or even impossible for two-player games. In between these two extreme cases arises the large, more realistic but less charted area of mixed-motive MARL, where influencing other learning agents becomes a main challenge [31, 37, 30].

Studies in computational economics have distilled different ways of directly influencing a rational, self-interested agent into two types: by providing tangible goods ( mechanism design ) and by providing information ( information design ) [52, 15]. For the former type, tangible goods align perfectly with RL rewards because rewards are natural and utilitarian. Several works in RL design algorithms for the ego agent to use the reward to incentivize other agents [57, 58, 59, 29]. For the latter type, the ego agent must possess information that is helpful to the other agent while not observed by the other agent. The ego agent then sends a message that partially reveals this information in the hope that the other agents respect the message and act in a way that benefits the ego agent. The key observation in information design is that the message needs to be respected, which indicates that the message, apart from benefiting the ego agent, must also benefit the receiving agent.

Two difficulties persist in modeling and solving information design with reinforcement learning. The first challenge is the issue of non-stationarity. The receiver's environment changes as the sender's signaling scheme is updated. If the sender uses policy gradient, it does not consider how modifications to its scheme affect the receiver's learning [17]. On top of this, the signal impacts not only the updating phase but also the sampling phase. In fact, the receiver uses the signals from the sender as part of its input. The signaling scheme will, therefore, directly affect the trajectory generation. This means that specific techniques in mechanism design, such as hyper-gradient, are unsuitable, and new methods need to be formulated [57].

The second challenge lies in how the message can potentially be respected by the receiver. The most general signal space is the set of all state subsets and is exponentially large. An analysis analogous to the revelation principle proves that there is an optimal signaling scheme that uses a signal space of the same size as the action space of the receiver, which leads to the classic obedience constraints and the linear program formulation of information design [28, 40]. However, under the revelation principle, a signal suggests an exact action, which means the receiver will be dictated by the sender if it respects the message. When both parties are reinforcement learning agents, such want of dictatorship does not build trust and respect between them and instead inevitably drives them to the equilibrium where all signals are ignored. Counter-intuitively, we find that the revelation principle can be removed under the learning context, and one will need to resort to more persuasive signaling schemes and the corresponding update methods.

This paper investigates how an informative ego sender could learn to influence a self-interested receiver with its informational advantage. We propose Markov signaling game for mixed-motive communication, where in each timestep, the sender encodes and sends a message to the receiver. After the signaling step, the receiver will act based on the message and its observation. We derive the signaling gradient to learn the signaling scheme that addresses the non-stationarity problem. This gradient considers the additional gradient chain from the receiver's policy and is proved to be an unbiased gradient estimation. It agrees with our intuition that the influence of the sender on the behavior of the receiver should be reflected in the gradient term. Based on the signaling gradient, we design the extended obedience constraints for incentive compatibility of the signaling scheme. 2 We further provide an approximation of the gradient of such constraints, which solves the second challenge because it is suitable for learning algorithms while preserving the optimum of the ego agent. Information design in MARL is then end-to-end differentiable and learnable with our algorithm.

Numerical experiments on Recommendation Letter and Reaching Goals demonstrate the efficacy of our approach. Extended discussions are provided for the method and the empirical results.

2 Preliminaries and Related Works

Information Design For information design, the core insight is to send messages to change the posterior beliefs of the receiver, which persuades it to take actions that benefit the sender [52, 15]. The canonical formulation considers a task that a sender wants to persuade a myopic receiver for one step [28]. The sender and the receiver share a prior distribution P ( s ) over the state s , which affects both the payoffs of the sender r i ( s, a ) and the receiver r j ( s, a ) (some recent work lifts this assumption [60]). The sender needs to honestly commit its signaling scheme, which is a policy that determines the distribution of signals, to the receiver before the interaction. This is referred to as the commitment assumption.

The flow of the persuasion process is as follows: (1) The sender commits a signaling scheme to the receiver; (2) The environment generates a state s . The sender observes the state s and then samples a message according to the distribution of the committed signaling scheme; and (3) The receiver receives the message, and then calculates a posterior and chooses an optimal action for itself. Given the current state and the receiver's chosen action, the sender and the receiver get rewards from the environment.

Based on an analysis similar to the revelation principle [28, 19], there is an optimal signaling scheme that does not require more signals than the number of actions available to the receiver. Thus it is without loss of generality for the sender to recommend an action directly to the receiver rather than sending a message. From the self-interested receiver's perspective, as long as it believes that the recommended actions are optimal in its posterior belief, it will follow the sender's advice. This kind of constraints of the sender's signaling scheme is called obedience constraints. When such constraints are satisfied, the signaling scheme will be incentive compatible, meaning that the receiver will follow the sender's advice. In this way, the process of information design can be modeled as a constrained optimization problem

where w i ( s, a ) and w j ( s, a ) are the utility functions of the sender and the receiver respectively, φ ( a | s ) is the sender's signaling scheme, and the problem is formulated as a linear program.

Information design is applicable to a vast array of real-world scenarios, including voter coalition formation [1], law enforcement deployment [21], price discrimination [4], etc. See [27] for a summary.

Learning to Communicate Communication learning is a significant subarea of MARL. Existing research primarily focus on fully cooperative settings [16, 49, 42]. Among the proposed methods in MARL with communication, DIAL [16] is the closest work to ours. It highlights the importance of the receiver's feedback to the sender, where the receiver updates its critic network and passes the gradient back to the sender's network. This also implies that the sender is assisting the receiver in evaluating the environment. The altruistic design of the sender is appropriate in fully cooperative scenarios, but might not extend to mixed-motive scenarios.

MARL communication under mixed-motive settings was attempted in [25]. The work uses social influence, which is defined via the Kullback-Leibler divergence of the receiver's policy update, to measure how persuasive is the sender's message. It is a pioneer attempt and demonstrates efficacy in several contexts, but the authors observed that 'The listener agent is not compelled to listen to any given speaker. Listeners selectively listen to a speaker only when it is beneficial, and influence cannot occur all the time.' This describes a lack of the obedience constraints, and motivates us to propose algorithms that are more effective and more general.

Related works on mechanism design, sequential information design, and emergent communication are deferred to Appendix A.

3 Markov Signaling Games

Consider a signaling game involving a sender and a receiver. The sender i is assumed to have access to the global state s ∈ S , while the receiver j makes decisions based only on its local observation o ∈ O and received message σ ∈ Σ from i . At each timestep t , the observation o t ∈ O is sampled by the emission function q : S → O , where the information in o t is a proper subset of the information in s t . We overload the notation and write o t ⊂ s t . The receiver's observation at each timestep is common knowledge between the sender and receiver. The sender's informational advantage over the receiver at each timestep t is reflected by s t -o t . By the sender's informational advantage of the problem setting, assume that { s t -o t } t ≥ 0 affects j 's payoff. This ensures that the sender has information that the receiver wants to know but does not know.

The sender maintains a stochastic signaling scheme φ η : S × O → ∆(Σ) , where φ is parameterized by η and ∆( X ) denotes the set of all random variables on X . The receiver's stochastic action policy is denoted as π θ : O × Σ → ∆( A ) , where A is the receiver's action space and θ ∈ Θ is the policy parameter. Without loss of generality, assume that the sender takes no environmental action. The state

transition function p : S × A → ∆( S ) and the reward functions R i : S × A → R ( R j : S × A → R ) are dependent on j 's chosen action and are not dependent on i 's message. In this way, i needs to influence j through its signaling scheme, which indirectly affects its long-term payoff expectation.

Then, a Markov signaling game (MSG, Figure 1) is defined as a tuple

At each timestep t in G , i observes a state s t ∈ S and j 's observation o t ∈ O is sampled by q , and then i sends a message σ t based on φ η . Then j takes action a t ∈ A based on its policy π θ and the environment transits to the next state s ′ ∈ S according to the transition function p . Meanwhile, player i (respectively, j ) receives the reward r i t ( r j t ) via the reward function R i ( R j ). The agents and the environment repeat this process until the environment terminates the episode.

The value functions in MSGs are similar to the value functions in Markov decision processes (MDPs). The sender's state value function V i φ,π ( s ) = E φ,π [ G i t | s t = s ] defines the expected return of the sender at a state, where G i t = ∑ ∞ k = t γ k -t r i k +1 . The signal value function Q for player i denotes the same expectation but under an additional condition of sending signal σ at the state s , as Q i φ,π ( s, σ ) = E φ,π [ G i t | s t = s, σ t = σ ] . The action value function U is the same expectation that conditions on all variables s t , σ t , a t , as U i φ,π ( s, σ, a ) = E φ,π [ G i t | s t = s, σ t = σ, a t = a ] . Notice the transition function and the reward functions are not dependent on σ . We then define the marginal action value function W i φ,π ( s, a ) = E φ,π [ G i t | s t = s, a t = a ] = U i φ,π ( s, σ, a ) . The receiver's value functions are similarly defined, by replacing G i with G j .

t t

Assume that the agents involved are self-interested, i.e., they are focusing solely on optimizing their own payoffs. However, their payoffs are general and are not necessarily the environmental rewards. For instance, the sender may replace its reward function R i with ( R i + R j ) as its optimization goal, which would result in it 'selfishly' optimizing for the social welfare.

MSG is new to the community and the features of MSG are quite distinctive: (1) The sender and receiver are heterogeneous, with the sender having an informational advantage; (2) Their interests may differ and thus they are in a mixed-motive scenario; (3) The sender 'signal does not influence anyone's reward. Thus the sender can only influence the receiver's belief to improve its own expected payoffs; (4) The receiver can take actions to directly determine the payoffs for both parties, but lacks sufficient information to estimate the current state and needs to obtain information from the sender.

There are several extensions of the MSGs. For example, (1) One may allow the sender to take environmental actions; (2) The sender may only have a partial observation of the state; (3) There could be multiple senders and receivers. Discussions on these extensions are deferred to Appendix B.

4 Method

The fundamental concept of information design is the sender optimizing its payoffs while adhering to the obedience constraints, as referenced in Equation (1). However, persuasion becomes more challenging in sequential and learning scenarios, due to the non-stationarity and the existence of the trivial equilibrium where all messages are arbitrary and are ignored.

The first obstacle is non-stationarity, a recognized issue in MARL, that is especially problematic in communication since it requires bi-level optimization. During the early stages of training, the sender's signaling scheme is mostly random and provides little information, which can quickly break the trust between mixed-motive agents and lead to the trivial equilibrium. The second difficulty is the want

of dictatorship. In the original obedience constraints, the receiver's policy is set to be deterministic by the revelation principle. It means that recommending a wrong action is non-forgiving, and thus suitable for a trial-and-error algorithm. This will also lead to the trivial equilibrium. In this section, we present how these challenges are addressed and how information design is learned.

4.1 Signaling Gradient

The proposed signaling gradient is utilized to compute the gradient of the sender's long-term expected payoff w.r.t. its signaling scheme parameters. When calculating this gradient, it explicitly takes into account the chain of the receiver's policy. This helps alleviate the non-stationarity between agents.

In MSGs, the signaling scheme affects the distribution of signals, indirectly impacting the receiver's action, which then determines the payoffs for all agents. Specifically, the sender's expected payoff is expanded as E φ,π [ V i φ,π ( s ) ] = ∑ s,o,σ,a d φ,π ( s ) · q ( o | s ) · φ η ( σ | s, o ) · π θ ( a | o, σ ) · U i φ,π ( s, σ, a ) , where d φ,π is the state visitation frequency given φ η and π θ . Similar to the case of the policy gradient (PG), the relationship between state visitation frequency and π cannot be explicitly written. The introduction of communication φ further complicates this process as it involves deriving ∇ η d φ,π ( s ) .

We utilized a method similar to the policy gradient to derive the gradient ∇ η E φ,π [ V i φ,π ( s ) ] and obtain an unbiased gradient estimation. We call this estimation the signaling gradient . The proof of the lemma is deferred to Appendix D.

Lemma 4.1. Given a signaling scheme φ η of the sender and an action policy π θ of the receiver in an MSG G , the gradient of the sender's value function V i φ,π ( s ) w.r.t. the signaling parameter η is

̸

It is worth noting that E [ W i φ,π ( s, a ) ∇ η log π θ ( a | o, σ ) ] = 0 and W i φ,π ( s, a ) takes an action as an input. As a consequence of this derivation, the sender's updated term includes the receiver's policy and action. This result aligns with intuition, as this additional term reflects the sender's consideration of its impact on the receiver.

One may naturally regard the signal as an action and directly apply the policy gradient. If so, one will obtain E φ,π [ Q i φ,π ( s, σ ) · ∇ η log φ η ( σ | s ) ] . This gradient will then be independent of the receiver's action and is therefore biased.

Connections to Other MARL Methods There are three perspectives to gain insights from the derivation of the signaling gradient. The first perspective is that the signaling gradient can be regarded as policy-based feedback from the receiver instead of value-based feedback in DIAL [16]. The second perspective is that the signaling gradient and LOLA [17] are similar in alleviating the non-stationarity in MARL communication. (More discussions are deferred to Appendix H.4.) Since MSGs consider the coupling decision-making processes of both parties, the signaling gradient involves the sender taking the initiative to consider how to influence the receiver. In contrast, LOLA involves an agent proactively adapting to other people's updates. The third perspective is that the signaling gradient can be seen as a first-order gradient that is absent in LIO [57]. This new gradient chain can be explicitly derived because the signaling scheme directly affects the receiver's sampling phase.

4.2 Policy Gradient for the Receiver

From the receiver's perspective, the decision process can be modeled as a partially observable MDP (POMDP), in which its observation is O × Σ in the corresponding MSG. Therefore, the receiver can optimize its payoff expectation V j φ,π ( o, σ ) by calculating the gradient E φ,π [ A j φ,π ( o, σ, a ) · ∇ θ log π θ ( a | o, σ ) ] , where A ( o, σ, a ) = Q ( o, σ, a ) -V ( o, σ ) is the advantage function [51]. Compared to the receiver in Bayesian persuasion ([28], see Section 4.5.1 for more details), the policy of the receiver in MSGs is now stochastic rather than deterministic, which allows a larger capacity in taking different actions and a larger capacity in decoding the received information.

4.3 Extended Obedience Constraints

In the learning algorithm context, we consider the sender's informational advantage on the current state s and investigate the incentive compatibility of the receiver. As an analogous of (1), the prior

of such information is then the occupancy measure d φ,π ( s ) of the state condition on the current signaling scheme and action policy. The payoff function w j ( s, a ) corresponds to the action value function W j φ,π ( s, a ) in Markov signaling games.

It amounts to deciding the signal space Σ . The revelation principle states that there is an optimal signaling scheme that does not require more signals than the number of actions available to the receiver. If one follows the revelation principle, one would reasonably use Σ = A , resulting in the following obedience constraints.

These constraints are technically correct. However, in the learning context, having only | A | possible signals means that the receiver either completely follows the suggested action or completely ignores the message. The former happens only if the obedience constraints are satisfied. In sequential interactions, the constraints will, of course, be violated occasionally. However, the dictatorship nature of the signaling scheme fails to sway the receiver without consistent satisfaction with the constraints.

Moreover, consider the trivial equilibrium between the sender and the receiver: The sender does not reveal useful information, and the receiver ignores the message. At this point, neither side could escape from the equilibrium alone. This means it is likely to fail once the learning algorithm converges to the trivial equilibrium. One natural choice is to send the information to the receiver instead of dictating actions. In this way, the receiver, as a learning agent, is more likely to be able to utilize the information and is more likely to respect the message.

Therefore, we extend the obedience constraints to general, continuous signal space Σ that describes the state. A common choice is Σ = S . The following lemma asserts that the extended obedience constraints impose the same optimum as with the obedience constraints.

Lemma 4.2. Given a receiver's observation o , the extended obedience constraints (4) in MSGs yield the same optimum as the obedience constraints (3).

for all σ, σ ′ ∈ Σ .

The lemma assumes the sender can access the receiver's policy and observation. Otherwise, the sender may use inferring methods to maintain an estimation of that (an example is [35]). This lemma is proved using the same argument in [3]. For convenience, in later sections, the left-hand side of (4) is denoted as C φ ( σ, σ ′ ) .

4.4 Learning Markov Signaling Games

Given a joint policy π , the self-interested sender attempts to optimize its payoff expectation in an MSG while satisfying the extended obedience constraints. This optimization problem is

Since we are employing a learning-based approach, it is necessary to calculate the gradient ∇ η C φ ( σ, σ ′ ) . In this way, our method is model-free and does not require the prior knowledge of P ( s ) (the occupancy measure d φ,π ( s ) ). Unfortunately, when calculating the gradient of the extended obedience constraints, the technique used in the signaling gradient cannot be applied to reveal the dependency of d φ,π on φ . Instead, the gradient is estimated using the biased sampling method as below.

where τ is a sampled trajectory with T timesteps, and σ ′ is randomly sampled, instead of being sampled from the signaling scheme. Despite that σ is from the data that is generated from φ , the purpose of having these σ, σ ′ is to find the violation of the ∀ σ, σ ′ constraints. Therefore in extended obedience constraints ∇ η π θ ( a | o, σ ) = 0 , which is different from the signaling gradient.

There are various methods available to solve the constrained optimization problem (5) iteratively, e.g., the Lagrangian method, the dual gradient descent method (DGD) [7]. We tested both methods in the experiments and we found that the Lagrangian method has better performance. See Appendix H.5 for the details. Taking the Lagrangian method as an example, The update of the signaling scheme parameters η ( k ) for the k -th iteration is

where λ σ,σ ′ denotes the non-negative Lagrangian multipliers (predefined as hyperparameters), and ( · ) -= min { 0 , ·} .

4.5 Discussions on Method

4.5.1 Lift to the Commitment Assumption

The most controversial but reasonable assumption in information design is the commitment assumption. In Bayesian persuasion (one-to-one persuasion) [28], the sender will commit to a signaling scheme first. The sender determines its signaling scheme before the games start and honestly tells it to the receiver. A justification of the commitment assumption is that, in a repeated game where a long-term sender interacts with a sequence of short-term receivers, the commitment will naturally emerge in equilibria. This is due to the sender's need to establish its reputation for credibility, which is essential for optimizing its long-term payoff expectations [45].

However, the reputation system between the receivers still needs to be well-defined, so the receivers that have previously interacted cannot convey information about the sender to the receivers that will interact later. Without a reputation system, the sender can optimize its payoff by claiming a respectful signaling scheme while taking an exploitable one. Instead, RL allows for organic and repeated interactions between senders and receivers in a given environment, more closely resembling real-world scenarios. This unique feature enables the learning framework to capture policy evolution and better replicate phenomena in human society.

4.5.2 Lift to the Revelation Principle

The extended obedience constraints remove the revelation principle analysis from the obedience constraints, thereby reverting the sender's behavior from 'action recommending' to 'signal sending'. In this way, the sender's set of signals becomes redundant. The redundancy renders the signaling scheme more general and amenable to learning-based approaches. Previously, the signaling scheme required a one-to-one mapping to recommend a particular action, where recommending an undesired action can be non-forgiving. With the introduction of redundancy, the sender can now learn manyto-one mappings to refer to the wanted action distribution, which is a more lenient way for a trial-and-error method. This redundancy is similar to other areas of learning algorithms. For example, one could increase the size of the neural network beyond the information theory necessity to better encode and represent the mapping.

4.5.3 Far-sighted Receiver

The nature of RL determines that the receiver in MSGs is considering the cumulative reward. This lifts the commitment assumption and evolves the trustworthiness of the sender's signaling scheme. But meanwhile, the cumulative reward formulates that the receiver must be far-sighted, which is different from the common assumption of a myopic receiver in information design. In fact, a far-sighted receiver is inevitable once we lift the commitment assumption.

A far-sighted receiver is, in general, regarded hard in the literature. Gan et al. [18] prove that information design with a far-sighted receiver is NP-hard. One could intuitively see this from the Recommendation Letter example (See Appendix F). The HR can deliberately choose not to hire any students, even when the signaling scheme satisfies the obedience constraints, hoping to force the professor to be more honest (i.e., reveal more information) in the future. The optimality of the receiver's policy is undefined in this setting.

One way to empirically prevent this is to set an additional constraint ∫ σ,σ ′ C φ ( σ, σ ′ ) dσdσ ′ ≥ ϵ apart from the obedience constraints C φ ( σ, σ ′ ) ≥ 0 in Equation (5), where ϵ > 0 . A more substantial improvement in reward will incentivize the receiver to establish trust in the long run.

4.5.4 Hyper Gradient

Our approach shares similarities with LIO (discussed in Appendix A), as both methods allow agents to alter their parameters to indirectly enhance their payoff expectations by influencing others. In their cases, agents achieve influence by offering rewards to others. The main difference between rewarding and signaling is that the former solely impacts others' policy updates (as the gained rewards are used exclusively for updating). In contrast, the latter additionally affects the sample generation.

In methods to incentivize others, the gradients of the receiver's one-step policy update w.r.t. the sender's rewarding network parameters are required to capture the sender's influence on the receiver. This kind of gradient is second-order and can be viewed as a hyper-gradient.

Unlike incentive-based interventions, in communication methods, the sender's outputs are the inputs of the receiver's actor. The sender can achieve influence while generating trajectories. Hence, our primary focus is on studying the first-order gradient of the receiver's policy w.r.t. the sender's signaling parameters, i.e., ∇ η π θ ( a | o, σ ) . The second-order gradients can also be computed, as shown in Equation (21) in the appendix. The effect of the hyper gradient is left for future work.

4.5.5 Sender's Access to Receiver

In signaling gradient, the updating of the sender's signaling scheme needs to backpropagate through the receiver's policy during training. Accessing the opponent in mixed-motive settings might sound counterintuitive at a glance, but this setting is actually reasonable and technically feasible.

In MARL algorithms, the Centralized Training with Decentralized Execution (CTDE) framework is commonly used [35]. This means that in the simulated environment, data is centralized and accessible during training, which allows us to use all the required quantities. Once the training is completed, the agent no longer has access to other agents.

From the information design perspective, we may consider the receiver as a 'dummy' learning agent that is auxiliary in centralized training. Once a sender agent is trained, it finds the (coarse correlated) equilibrium that is no longer connected to specific receivers. The sender is supposed to persuade any rational, self-interested agents in subsequent interactions with them.

Technically, in communication methods, it is common for gradients to pass through other agents during training. We adopted the commonly used Gumbel-Softmax technique in the field of emergent communication research to allow for end-to-end differentiation, which is used to retain the gradients of sampled signals [24, 20].

5 Experiments

The method proposed in this paper is validated in Recommendation Letter and Reaching Goals . The receiver's action policy is implemented by the advantage actor-critic (A2C) [38]. Each curve in the experimental result graphs is drawn with at least 15 random seeds. All the seeds are included in the results (including those failed ones). Running 4 seeds with 2 NVIDIA GeForce RTX 3090, the longest time is Reaching Goals with 5 × 5 map, which takes up to a day.

Our designed algorithm takes the perspective of a self-interested sender. Therefore, the measure of success for our algorithm primarily depends on the sender's reward r i . However, we also present the curve of r i + r j (social welfare) in an attempt to demonstrate that even when the sender is self-interested and the signaling scheme is somewhat deceptive, it does not harm social welfare, and it might even enhance social welfare in some scenarios.

5.1 Recommendation Letter

Recommendation Letter is a classic example in information design [15, 28]. In this task, a professor will write recommendation letters for a number of graduating students, and a company's human resources department (HR) will receive the letters and decide whether to hire the students.

The professor and the HR share a prior distribution of the candidates' quality, with a probability of 1 / 3 that the candidate is strong and a probability of 2 / 3 that the candidate is weak. The HR does not know exactly what each student's quality is but wants to hire strong students, while the letters are the only source of information. The HR will get a reward of 1 for hiring a strong candidate, a penalty of -1 for hiring a weak candidate, and a reward of 0 for not hiring. The professor gets a 1 reward for each hire. In this section, we focus on analyzing the experimental results, while a classic analysis of the three situations is presented in Appendix F.

In the experiments, we primarily compared the performance of policy gradient class algorithms (PG), PG class algorithms with obedience constraints (PGOC), DIAL [16], signaling gradient (SG, our ablation), and signaling gradient with obedience constraints (SGOC, our proposed method) in learning a signaling scheme. More specifically, the PG class algorithm utilized in experiments refers to A2C. Furthermore, SG also employed A2C techniques, including the use of the actor-critic framework, target critic, and advantage function (adapted to W i ( s, a ) -V i ( s ) in MSGs). The algorithms with obedience constraints are also required to maintain an extra critic for estimating W j ( s, a ) . We let φ η ( σ | s, o ) = φ η ( σ | s ) and Σ = { 0 , 1 } . The performance comparisons are shown in Figure 2 (a-c).

SG O C

SG

PG

PG O C

D

I A L

(a)

r

i

(b)

r

j

(c)

r

i

+

r

j

(d)

r

i

in

5

×

5

map

(e)

r

i

in

3

×

3

map

(f)

r

j

in

3

×

3

map

(g)

r

i

+

r

j

in

3

×

3

map

(h)

r

i

+

r

j

in

5

×

5

map

一

Reward

Reward

0.7

0.7

400

0.6

0.3

0.6

0.5

0.2

0.5

200

0.4

0.3

0.1

0.4

0.3

0

0.2

0

0.2

-200

0.1

-0.1

0.1

0

0

-400

0

1

2

3

0

1

2

3

0

1

2

3

0

1

2

3

Step

1e8

Step

le8

Step

1e8

Episode

1e5

Reward

Reward

Reward

Reward

400

400

400

400

200

200

200

200

0

0

0

0

-200

-200

-200

-200

-400

-400

-400

-400

0

2

4

6

8

10

0

2

4

6

8

10

0

2

4

6

8

10

0

1

2

3

Episode

1e4

Episode

1e4

Episode

1e4

Episode

1e5

The experiment results show that the three examples (two equilibria plus the honest signaling scheme) introduced in Section 1 have emerged. The algorithms with obedience constraints (PGOC and SGOC) reach the third equilibrium (the best for the sender), DIAL reaches the second example (the best for the receiver), and PG and SG reach the first equilibrium (the worst for both parties). Given that the Recommendation Letter task is not a problem with interdependent states, it is justifiable to expect that the performance of PGOC and SGOC (PG and SG) would be similar.

5.2 Reaching Goals

To reflect the inconsistency of the interests and the information asymmetry, we evaluate the methods in the Reaching Goals task. In this challenging task, the sender and receiver have different goals: the sender wants the red apple, while the receiver wants the green apple. At any given time, there is one of each type of apple on the map. Only the receiver is able to move around to reach apples in the grid-world map. The conflict of interest arises because these two types of apples are independently generated at random, and their locations are typically not the same. The receiver will not get a reward for reaching the red apple, but it does not know where the green apple is. The sender, on the other hand, knows the location of both types of apples, but can only get a reward when the receiver reaches the red apple. The sender can only indirectly optimize its payoff by sending messages to influence the receiver's actions. Moreover, this conflict can be exacerbated by designing distance penalties. After each timestep, both agents will get a penalty based on the distance between the receiver and its desired goal. And we set a fixed horizon of 50 for each episode in this scenario. Example maps of Reaching Goals are shown in Figure 6.

To evaluate the efficacy of SGOC, we conducted experiments on maps of 3 × 3 and 5 × 5 , and the results are presented in Figure 2. Let the signal space Σ = S 1 , where S 1 is one channel out of three channels (the locations of two apples and the receiver's position) of the image. The receiver can only see its position. The experiments of having a larger o j are included in Appendix H.7. The smaller the o j , the greater the sender's persuasion ability. From the empirical results, the signaling gradient is shown to be an essential factor in sequential communication scenarios.

5.3 Discussions on Experiments

5.3.1 Symmetricity of the Signaling Schemes

An interesting phenomenon is observed in the Recommendation Letter experiments: training with different random seeds may result in different pairs of encoders and decoders. In other words, the professor may signal 1 to indicate a strong student (or recommend this student) in some seeds, while in others, this is signaled by 0 . However, regardless of which case it is, the paired HR can always understand the semantics of the signals (reflected in the evolution of the receiver's policy). Based on the outcomes, the seeds are divided into two parts (A seeds and B seeds).

This phenomenon is reasonable since we do not make any prior assumption about signaling semantics. The symmetric results of Recommendation Letter experiments are shown in Figure 3.

A  Se e d s

(a)

φ

(1

|

'W'

)

(b)

φ

(1

|

'S'

)

(c)

π

(

'H'

|

0)

(d)

π

(

'H'

|

1)

Prob.

Prob.

Prob.

Prob.

1

1

0

0

0

0

0

1

2

3

0

1

2

3

0

一

2

3

0

1

2

3

Step

1e8

Step

1e8

Step

1e8

Step

1e8

5.3.2 Honesty of the Sender

As discussed in Section 4.5.3, the obedience constraints can be associated with an additional constraint ∫ σ,σ ′ C j φ ( σ, σ ′ ) dσdσ ′ ≥ ϵ . The hyperparameter ϵ > 0 is set to improve the credibility of signaling schemes in practical situations and to cope with the RL capacity of the receiver. Define the honesty metric as | φ (1 | 'S' ) -φ (1 | 'W' ) | . The experiments show that the larger the ϵ and the Lagrangian λ , the more honest the signaling scheme will be, as shown in Figure 8 in the appendix.

6 Conclusion

We investigate information design, a substantial and open area, for MARL that discusses mixedmotive communication. Technically, we propose the Markov signaling games to describe the problem and provide its characterizations. We then prove the signaling gradient lemma, which gives an unbiased way to estimate the gradient and update the sender's signaling network. To learn the incentive compatibility of the signaling scheme, we propose extended obedience constraints. The new constraints are more suitable for learning algorithms and practically promote mutually beneficial signaling schemes. The commitment assumption and the revelation principle are lifted by investigating information design with MARL. Experiments and extended discussions are presented to demonstrate the efficacy of our framework and algorithm.

Acknowledgments

We extend our heartfelt gratitude to Cynthia Huang, Zongqing Lu, Shuai Li, Pascal Poupart, Dan Qiao, Han Wang, Zichuan Wan, Jiachen Yang, Zewu Zheng, Shuhui Zhu for their enlightening discussions. Part of the implementation regarding the signaling gradient was developed by Zewu Zheng.

The authors are partially supported by National Natural Science Foundation of China (62106213, 72150002), Postdoctoral Science Foundation of China (2022M723039), Shenzhen Science and Technology Program (RCBS20210609104356063, JCYJ20210324120011032), and Guangdong Provincial Key Laboratory of Big Data Computing of The Chinese University of Hong Kong, Shenzhen.

References

Ricardo Alonso and Odilon Câmara. Persuading voters. American Economic Review , 106(11): 3590-3605, 2016.

Kai Arulkumaran, Antoine Cully, and Julian Togelius. Alphastar: An evolutionary computation perspective. In Proceedings of the genetic and evolutionary computation conference companion , pages 314-315, 2019.

Dirk Bergemann and Stephen Morris. Information design: A unified perspective. Journal of Economic Literature , 57(1):44-95, 2019.

Dirk Bergemann, Benjamin Brooks, and Stephen Morris. The limits of price discrimination. American Economic Review , 105(3):921-957, 2015.

Martino Bernasconi, Matteo Castiglioni, Alberto Marchesi, Nicola Gatti, and Francesco Trovò. Sequential information design: Learning to persuade in the dark. In Advances in Neural Information Processing Systems , 2022.

Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław D˛ ebiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680 , 2019.

Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

Noam Brown and Tuomas Sandholm. Superhuman ai for heads-up no-limit poker: Libratus beats top professionals. Science , 359(6374):418-424, 2018.

Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker. Science , 365 (6456):885-890, 2019.

Kris Cao, Angeliki Lazaridou, Marc Lanctot, Joel Z Leibo, Karl Tuyls, and Stephen Clark. Emergent communication through negotiation. arXiv preprint arXiv:1804.03980 , 2018.

Andrea Celli, Stefano Coniglio, and Nicola Gatti. Private bayesian persuasion with sequential games. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 1886-1893, 2020.

E Choi, A Lazaridou, and N de Freitas. Compositional obverter communication learning from raw visual input. arXiv preprint arXiv:1804.02341 , 2018.

Vincent Conitzer and Tuomas Sandholm. Complexity of mechanism design. In UAI , 2002.

Alvin W Drake. Observation of a Markov process through a noisy channel . PhD thesis, Massachusetts Institute of Technology, 1962.

Shaddin Dughmi. Algorithmic information structure design: A survey. ACM SIGecom Exchanges , 15(2):2-24, 2017.

Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. Advances in Neural Information Processing Systems , 2016.

Jakob Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learning awareness. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems , pages 122-130, 2018.

Jiarui Gan, Rupak Majumdar, Goran Radanovic, and Adish Singla. Bayesian persuasion in sequential decision-making. In AAAI Conference on Artificial Intelligence , 2022.

Jiarui Gan, Rupak Majumdar, Goran Radanovic, and Adish Singla. Sequential decision making with information asymmetry. In International Conference on Concurrency Theory , 2022.

Serhii Havrylov and Ivan Titov. Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. Advances in Neural Information Processing Systems , 30, 2017.

Penélope Hernández and Zvika Neeman. How bayesian persuasion can help reduce illegal parking and other socially undesirable behavior. American Economic Journal: Microeconomics , 14(1):186-215, 2022.

Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence , 2018.

Dmitry Ivanov, Iskander Safiulin, Igor Filippov, and Ksenia Balabaeva. Optimal-er auctions through attention. Advances in Neural Information Processing Systems , 35:34734-34747, 2022.

Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144 , 2016.

Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Z Leibo, and Nando De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International Conference on Machine Learning , pages 3040-3049. PMLR, 2019.

John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature , 2021.

Emir Kamenica. Bayesian persuasion and information design. Annual Review of Economics , 11:249-272, 2019.

Emir Kamenica and Matthew Gentzkow. Bayesian persuasion. American Economic Review , 101(6):2590-2615, 2011.

Raphael Koster, Jan Balaguer, Andrea Tacchetti, Ari Weinstein, Tina Zhu, Oliver Hauser, Duncan Williams, Lucy Campbell-Gillingham, Phoebe Thacker, Matthew Botvinick, et al. Human-centred mechanism design with democratic ai. Nature Human Behaviour , 6(10): 1398-1407, 2022.

Joel Z Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel. Multi-agent reinforcement learning in sequential social dilemmas. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems , pages 464-473, 2017.

Joel Z Leibo, Edgar A Dueñez-Guzman, Alexander Vezhnevets, John P Agapiou, Peter Sunehag, Raphael Koster, Jayd Matyas, Charlie Beattie, Igor Mordatch, and Thore Graepel. Scalable evaluation of multi-agent reinforcement learning with melting pot. In International Conference on Machine Learning , 2021.

David Lewis. Convention: A philosophical study . John Wiley & Sons, 2008.

Changxi Li, Fenghua He, Hongsheng Qi, Daizhan Cheng, Longbiao Ma, Yonghong Wu, and Shuo Chen. Potential games design using local information. In 2018 IEEE Conference on Decision and Control , pages 1911-1916. IEEE, 2018.

Na Li and Jason R Marden. Designing games for distributed optimization. IEEE Journal of Selected Topics in Signal Processing , 7(2):230-242, 2013.

Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in Neural Information Processing Systems , 2017.

Donald McCloskey and Arjo Klamer. One quarter of gdp is persuasion. The American Economic Review , 85(2):191-195, 1995.

Kevin R McKee, Ian Gemp, Brian McWilliams, Edgar A Duèñez-Guzmán, Edward Hughes, and Joel Z Leibo. Social diversity and social preferences in mixed-motive reinforcement learning. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems , pages 869-877, 2020.

Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning , 2016.

Matej Moravˇ cík, Martin Schmid, Neil Burch, Viliam Lis` y, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling. Deepstack: Expert-level artificial intelligence in heads-up no-limit poker. Science , 356(6337):508-513, 2017.

Roger B Myerson. Incentive compatibility and the bargaining problem. Econometrica: Journal of the Econometric Society , pages 61-73, 1979.

Roger B Myerson. Optimal auction design. Mathematics of Operations Research , 6(1):58-73, 1981.

Peng Peng, Ying Wen, Yaodong Yang, Quan Yuan, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent bidirectionally-coordinated nets: Emergence of human-level coordination in learning to play starcraft combat games. arXiv preprint arXiv:1703.10069 , 2017.

Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, and Sergey Levine. Variational discriminator bottleneck: Improving imitation learning, inverse rl, and gans by constraining information flow. In International Conference on Learning Representations , 2019.

Lillian J Ratliff and Tanner Fiez. Adaptive incentive design. IEEE Transactions on Automatic Control , 66(8):3871-3878, 2020.

Luis Rayo and Ilya Segal. Optimal information disclosure. Journal of Political Economy , 118 (5):949-987, 2010.

Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature , 2020.

Tianmin Shu and Yuandong Tian. M ˆ 3RL: Mind-aware multi-agent management reinforcement learning. In International Conference on Learning Representations , 2019.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature , 529(7587):484-489, 2016.

Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backpropagation. Advances in Neural Information Processing Systems , 2016.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems , 1999.

Ina Taneva. Information design. American Economic Journal: Microeconomics , 11(4):151-85, 2019.

Leigh Tesfatsion and Kenneth L Judd. Handbook of computational economics: Agent-based computational economics . Elsevier, 2006.

Richard H Thaler. Anomalies: The ultimatum game. Journal of economic perspectives , 2(4): 195-206, 1988.

Mycal Tucker, Roger Levy, Julie A Shah, and Noga Zaslavsky. Trading off utility, informativeness, and complexity in emergent communication. Advances in Neural Information Processing Systems , 35:22214-22228, 2022.

Jibang Wu, Zixuan Zhang, Zhe Feng, Zhaoran Wang, Zhuoran Yang, Michael I Jordan, and Haifeng Xu. Sequential information design: Markov persuasion process and its efficient reinforcement learning. In Proceedings of the 23rd ACM Conference on Economics and Computation , pages 471-472, 2022.

Jiachen Yang, Ang Li, Mehrdad Farajtabar, Peter Sunehag, Edward Hughes, and Hongyuan Zha. Learning to incentivize other learning agents. Advances in Neural Information Processing Systems , 2020.

Jiachen Yang, Ethan Wang, Rakshit Trivedi, Tuo Zhao, and Hongyuan Zha. Adaptive incentive design with multi-agent meta-gradient reinforcement learning. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems , pages 1436-1445, 2022.

Stephan Zheng, Alexander Trott, Sunil Srinivasa, David C Parkes, and Richard Socher. The ai economist: Taxation policy design via two-level deep multiagent reinforcement learning. Science Advances , 8(18):eabk2607, 2022.

You Zu, Krishnamurthy Iyer, and Haifeng Xu. Learning to persuade on the fly: Robustness against ignorance. In Proceedings of the 22nd ACM Conference on Economics and Computation , pages 927-928, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studied a Markov signaling game and proposed to use the signalling gradient and extended obedience constraint to build a learning system that can effectively train the sender in the game to reach desirable equilibriums. Experiments performed on two toy problems suggest that the newly developed SGOC algorithm can perform well and reach desirable equilibriums, compared to several competing learning methods.\n\nSTRENGTHS:\nIt is interesting and important to study the information sharing problem in a multi-agent context. The Markov signaling game presented in this paper appears to be of some practical values and deserves further investigation. The design of the new SGOC algorithm also introduces some novel ideas.\n\nWEAKNESSES:\nIt is not clear whether the Markov signaling game introduced in this paper is new and has never been studied previously. The novelty of this game may need to be clarified. Its usefulness for practical applications also seems uncertain. I cannot immediately relate the two toy problems studied experimentally in this paper with any practically important scenarios where an effective machine learning-based solution of the Markov signaling game is essential. Hence, the motivation and research value of this paper may need to be significantly strengthened.\n\nThe above concern is partially triggered by the lack of discussion of related works in the main text of this paper. Because of this, the importance and novelty of studying the Markov signaling game remain doubtful to a certain extent. The true technical novelty of using the signaling gradient and the extended obedience constraint hence should be further investigated.\n\nIt is assumed that the sender can access the receiver's policy and observation (the receiver's observation is not necessarily accessible to the sender). I have no idea when such an assumption could be true in a practical scenario. In fact, the calculation of the signaling gradient may rely on this assumption. However, when the sender and receiver have potentially conflicting interests, it is entirely unclear why the receiver would be motivated to honestly support the sender by providing its internal information to the sender (in a conflicting scenario, the sender and receiver may not have the motivation to participate in a centralized training process). This further makes me question the practical value of the new SGOC algorithm.\n\nSome statements in the paper sound a bit questionable and needs further clarification and justification. For example, the authors mentioned several times that the signaling gradient can alleviate the non-stationarity between the sender and receiver. However, it remains unknown to me why the non-stationarity can be alleviated in this way. Moreover, since the receiver is handling a partially-observable problem, it may consider to adopt a recurrent neural network as its policy network. In view of this, the calculation of the signaling gradient may become significantly more complicated. Meanwhile, the idea of biased policy gradient has been mentioned multiple times in this paper. However, I can hardly understand its true meaning. Hence, related discussions in the paper do not sound highly convincing to me.\n\nBesides the above, I have no idea why $\\nabla_{\\eta}\\pi_{\\theta}(a|o,\\sigma)$ should be 0 and why the sender's signaling scheme and the receiver's action policy should be updated alternately rather than simultaneously. This as well as several other algorithmic design details appear to have some technical issues.\n\nThe authors discussed the necessity of establishing a reputation system. However, the feasibility, complexity, and difficulty of building such a reputation system have not been investigated with sufficient depth. While building such a reputation system may not be necessary for this study, its feasibility could significantly affect the importance and practicality of the newly developed algorithm.\n\nQUESTIONS:\nWhy is the Markov signaling game studied in this paper new and practically important?\n\nWhat is the practical difficulty of supporting/realizing the commitment assumption? When the sender and receiver have potentially conflicting interests, why would the receiver be motivated to honestly support the sender by providing its internal information to the sender?\n\nWhy will the non-stationarity issue be alleviated by using signalling gradient?\n\nWhat is the exact meaning of biased policy gradient?\n\nWhy should $\\nabla_{\\eta}\\pi_{\\theta}(a|o,\\sigma)$ be 0? Why should the sender's signaling scheme and the receiver's action policy be updated alternately rather than simultaneously?\n\nWhat are the feasibility, complexity, and difficulty of building a reputation system to support the basic assumptions of the new algorithm and the Markov signaling game?\n\nLIMITATIONS:\nI do not have any concerns regarding this question.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

comments: SUMMARY:
This paper proposes a new agent-interaction framework called Markov signaling games, and the corresponding algorithm for mixed-motive tasks. Various experiments and discussions are provided to show the efficacy of the proposed method.

STRENGTHS:
- The authors provide a new formulation of agent-interaction framework for multi-agent reinforcement learning and the corresponding detailed explanation.

- The proposed algorithm to solve Markov signaling game with theoretical results is provided.

WEAKNESSES:
It is unclear what makes Markov signaling games important. While related works on multi-agent RL with communication are provided, It seems that Markov signaling games appear to be just one concept of communication in multi-agent RL. In the proposed game, it is assumed that the sender observes a fully environmental state and sends a message to the receiver. Is this assumption reasonable and important? The authors should explain why Markov signaling games are important compared to conventional communication frameworks in multi-agent RL, where multiple agents aim to solve tasks while communicating with other agents. The conventional framework does not even differentiate between which agents are senders or receivers.

Furthermore, the paper should include a comparison with state-of-the-art multi-agent communication algorithms such as TarMAC[1], ATOC[2], and others. Currently, the authors only provide a comparison with DIAL.


[1] A. Das et al., "TarMAC: Targeted multi-agent communication"

[2] J. Jiang et al., "Learning attentional communication for multi-agent cooperation"

QUESTIONS:
See Weakness

LIMITATIONS:
The limitations are provided in Appendix.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of multi-agent reinforcement learning in environments with mixed motives, which lie on the spectrum from fully cooperative to fully competitive scenarios. Specifically, it studies the problem of information design in this setting: how a sender might craft messages so that the behaviour of a receiver is successfully influenced. There are non-trivial implications in this setup, including the non-stationarity introduced by the messages and the fact that the receiver may choose not to obey them. The authors propose an algorithm to derive a signaling policy that takes into account the effect of the messages on the receiver's policy, as well as a way to learn the receiver's policy in a way that provides compatibility between the incentives. The algorithm is validated on two types of environments with mixed incentives.\n\nSTRENGTHS:\nOriginality: the paper is original in its consideration of mixed-motive environments, which have not received much attention in MARL, and yet are highly relevant for practical scenarios.\n\nQuality: the paper is of good quality. It motivates the work well by drawing connections to computational economics. It proposes methods that are grounded and sound, and performs a reasonable evaluation.\n\nClarity: the paper is well-structured and clear overall, although I was not able to understand some of the technical details as I am not familiar with the area in depth.\n\nSignificance: the area is one of active interest in the NeurIPS community.\n\nWEAKNESSES:\nW1. In my opinion, the primary weakness of this work is the strength of a particular assumption made by the authors: that the sender has full access to the receiver's policy and observation at every step. I am struggling to think of a scenario in which the assumption is suitable (since it amounts to global visibility) and yet the setting is not fully cooperative. What are some examples of scenarios in which this is the case? This aspect needs to be discussed more thoroughly. Otherwise, the proposed method may be of limited practical use.\n\nQUESTIONS:\nQS1. Please see W1 above, I would appreciate an answer.\n\nQS2. In the evaluation, it is unclear to me what is the appropriate metric to judge the performance of the algorithm. Is it $r^i$, $r^j$, or $r^i + r^j$ (social welfare)? Can we even speak of a right metric given the incompatibility between the incentives? Given that e.g. DIAL reaches the best equilibrium for the receiver, are the methods even comparable, or is the evaluation merely meant to illustrate which equilibria are reached? The \"goal\" of the evaluation should be clarified.\n\nQS3. Note that the supplementary material also includes the main paper text, and that the main PDF is not rendered properly (text is inaccessible, references are not linked). Consider compiling the main paper and supplementary material separately instead of cutting the PDF.\n\nQS4. Some nitpicks: first phrase of the abstract, while RL definitely is inspired by reward-driven learning in humans and animals, it is not necessarily setting out to mimic it; line 90 nature -> environment?.\n\nLIMITATIONS:\nCode and detailed information is available, which means the work is reproducible. No limitations beyond those already discussed in previous points. The authors have discussed broader impacts. The supplementary material includes a high-quality discussion of possible extensions in great depth.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
The article tackles the scenario where two agents, a sender $i$ and a receiver $j$, act in an environment.
The sender $i$ can send a message to the receiver $j$ at each time step.
The sender's goal is to learn what information to send to the receiver such that it influences the latter's policy to maximize its own return.
The authors derive a learning signal analogous to the policy gradient in the single-agent setup, and they also propose a version with a supplementary obedience constraint.

STRENGTHS:
The strength of the paper lies in the extensive discussion of the setup and its challenges, and the different equilibria that could be reached by the two agents. These are also supported by empirical results.

WEAKNESSES:
The scenario is limited to the case where the sender can backpropagate through the receiver's estimator, which looks far from a realistic setup.

Formula (3) needs some introduction. It is not clear why that formula expresses an obedience constraint. Similarly, it would be helpful to restate the revelation principle, not just mention it.

QUESTIONS:
What is incentive compatibility?

LIMITATIONS:
The authors mention some limitations, but I would add to that the need to backpropagate through the receiver's policy estimator.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies how multi-agent reinforcement learning agents influence each other by hsaring different kinds of information. Additionally, the authors study how different information affects the transition of the agents trajectories and what information can be ignored. To ensure efficient learning, a suitable gradient for the problem is also proposed.\n\nSTRENGTHS:\nThis paper is well written and reads well. It is also well structured and contains only few spelling mistakes.\nSome other strengths in particular:\n- Proposes markov signalling games\n- Provides an extensive theoretical analysis of the approach\n- Develop the notions of a signalling gradient\n\nWEAKNESSES:\n1. some of the related works in the appendix could be moved (or summarised) to the main text; the provided related works in the main text are too scarce and do not provide enough relevant information that relates to the paper\n2. the introduced markov signalling games seem to have some links with the famous Lewis games and other similar referencial games. Would be interesting to see a discussion around these concepts, as I cannot find it in the paper. Some related works also use reinforcement learning to learn communication involving similar games and concepts, such as [1], [2], [3], [4].\n\nminor:\n- line 323: \"ours\" -> \"our\"\n\nOverall, this paper is interesting and well structured. Please find here and below some points that I would like the authors to comment on.\n\n[1] https://openreview.net/pdf?id=O5arhQvBdH\n\n[2] https://arxiv.org/pdf/1804.02341.pdf\n\n[3] https://arxiv.org/pdf/1804.03980.pdf\n\n[4] https://arxiv.org/pdf/1705.11192.pdf\n\nQUESTIONS:\n1. in line 306 it is stated that the receivers action policy is implemented using A2C. Is this the case for the sender too? Or do they share the same network as in multiple MARL approaches? This should be clarified in the paper.\n\nLIMITATIONS:\n1. It would be interesting to analyse the explainability of the messages exchanged by the receiver and speaker in these experiments, using the proposed method, as I can't find this is the paper and it is something very relevant when it comes to communication-driven games.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}