SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Moreover, consider the trivial equilibrium between the sender and the receiver: The sender does not reveal useful information, and the receiver ignores the message. At this point, neither side could escape from the equilibrium alone. This means it is likely to fail once the learning algorithm converges to the trivial equilibrium. One natural choice is to send the information to the receiver instead of dictating actions. In this way, the receiver, as a learning agent, is more likely to be able to utilize the information and is more likely to respect the message.
Therefore, we extend the obedience constraints to general, continuous signal space  $\Sigma$  that describes the state. A common choice is  $\Sigma = S$ . The following lemma asserts that the extended obedience constraints impose the same optimum as with the obedience constraints.
Lemma 4.2. Given a receiver's observation  $o$ , the extended obedience constraints (4) in MSGs yield the same optimum as the obedience constraints (3).
$$
\sum_ {s} d _ {\varphi , \pi} (s) \cdot \varphi_ {\eta} (\sigma \mid s, o) \cdot \sum_ {a} \left[ \pi_ {\theta} (a \mid o, \sigma) - \pi_ {\theta} (a \mid o, \sigma^ {\prime}) \right] \cdot W _ {\varphi , \pi} ^ {j} (s, a) \geq 0, \tag {4}
$$
for all  $\sigma, \sigma' \in \Sigma$ .
The lemma assumes the sender can access the receiver's policy and observation. Otherwise, the sender may use inferring methods to maintain an estimation of that (an example is [35]). This lemma is proved using the same argument in [3]. For convenience, in later sections, the left-hand side of (4) is denoted as  $C_{\varphi}(\sigma, \sigma')$ .
4.4 Learning Markov Signaling Games
Given a joint policy  $\pi$ , the self-interested sender attempts to optimize its payoff expectation in an MSG while satisfying the extended obedience constraints. This optimization problem is
$$
\max  _ {\eta} \mathbb {E} _ {\varphi , \pi} \left[ V _ {\varphi , \pi} ^ {i} (s) \right], \quad \text {s . t .} \quad C _ {\varphi} (\sigma , \sigma^ {\prime}) \geq 0, \quad \forall \sigma , \sigma^ {\prime}. \tag {5}
$$
Since we are employing a learning-based approach, it is necessary to calculate the gradient  $\nabla_{\eta}C_{\varphi}(\sigma ,\sigma^{\prime})$ . In this way, our method is model-free and does not require the prior knowledge of  $P(s)$  (the occupancy measure  $d_{\varphi ,\pi}(s)$ ). Unfortunately, when calculating the gradient of the extended obedience constraints, the technique used in the signaling gradient cannot be applied to reveal the dependency of  $d_{\varphi ,\pi}$  on  $\varphi$ . Instead, the gradient is estimated using the biased sampling method as below.
$$
\nabla_ {\eta} \hat {C} _ {\varphi} \left(\sigma , \sigma^ {\prime}\right) = \frac {1}{T} \sum_ {s _ {t} \in \tau} \left[ \sum_ {a} \left(\pi_ {\theta} \left(a \mid o _ {t}, \sigma\right) - \pi_ {\theta} \left(a \mid o _ {t}, \sigma^ {\prime}\right)\right) \cdot W _ {\varphi , \pi} ^ {j} \left(s _ {t}, a\right) \cdot \nabla_ {\eta} \varphi_ {\eta} \left(\sigma \mid s _ {t}, o _ {t}\right) \right], \tag {6}
$$
where  $\tau$  is a sampled trajectory with  $T$  timesteps, and  $\sigma'$  is randomly sampled, instead of being sampled from the signaling scheme. Despite that  $\sigma$  is from the data that is generated from  $\varphi$ , the purpose of having these  $\sigma, \sigma'$  is to find the violation of the  $\forall \sigma, \sigma'$  constraints. Therefore in extended obedience constraints  $\nabla_{\eta} \pi_{\theta}(a \mid o, \sigma) = 0$ , which is different from the signaling gradient.
There are various methods available to solve the constrained optimization problem (5) iteratively, e.g., the Lagrangian method, the dual gradient descent method (DGD) [7]. We tested both methods in the experiments and we found that the Lagrangian method has better performance. See Appendix H.5 for the details. Taking the Lagrangian method as an example, The update of the signaling scheme parameters  $\eta^{(k)}$  for the  $k$ -th iteration is
$$
\eta^ {(k + 1)} \leftarrow \eta^ {(k)} + \nabla_ {\eta} \mathbb {E} _ {\varphi , \pi} \left[ V _ {\varphi , \pi} ^ {i} (s) \right] + \sum_ {\sigma , \sigma^ {\prime}} \lambda_ {\sigma , \sigma^ {\prime}} \cdot \nabla_ {\eta} \left(\hat {C} _ {\varphi} (\sigma , \sigma^ {\prime})\right) ^ {-}, \tag {7}
$$
where  $\lambda_{\sigma, \sigma'}$  denotes the non-negative Lagrangian multipliers (predefined as hyperparameters), and  $(\cdot)^{-} = \min \{0, \cdot\}$ .
4.5 Discussions on Method
4.5.1 Lift to the Commitment Assumption
The most controversial but reasonable assumption in information design is the commitment assumption. In Bayesian persuasion (one-to-one persuasion) [28], the sender will commit to a signaling scheme first. The sender determines its signaling scheme before the games start and honestly tells it to the receiver. A justification of the commitment assumption is that, in a repeated game where a long-term sender interacts with a sequence of short-term receivers, the commitment will naturally emerge in equilibria. This is due to the sender's need to establish its reputation for credibility, which is essential for optimizing its long-term payoff expectations [45].
However, the reputation system between the receivers still needs to be well-defined, so the receivers that have previously interacted cannot convey information about the sender to the receivers that will interact later. Without a reputation system, the sender can optimize its payoff by claiming a respectful signaling scheme while taking an exploitable one. Instead, RL allows for organic and repeated interactions between senders and receivers in a given environment, more closely resembling real-world scenarios. This unique feature enables the learning framework to capture policy evolution and better replicate phenomena in human society.
4.5.2 Lift to the Revelation Principle
The extended obedience constraints remove the revelation principle analysis from the obedience constraints, thereby reverting the sender's behavior from "action recommending" to "signal sending". In this way, the sender's set of signals becomes redundant. The redundancy renders the signaling scheme more general and amenable to learning-based approaches. Previously, the signaling scheme required a one-to-one mapping to recommend a particular action, where recommending an undesired action can be non-forgiving. With the introduction of redundancy, the sender can now learn many-to-one mappings to refer to the wanted action distribution, which is a more lenient way for a trial-and-error method. This redundancy is similar to other areas of learning algorithms. For example, one could increase the size of the neural network beyond the information theory necessity to better encode and represent the mapping.
4.5.3 Far-sighted Receiver
The nature of RL determines that the receiver in MSGs is considering the cumulative reward. This lifts the commitment assumption and evolves the trustworthiness of the sender's signaling scheme. But meanwhile, the cumulative reward formulates that the receiver must be far-sighted, which is different from the common assumption of a myopic receiver in information design. In fact, a far-sighted receiver is inevitable once we lift the commitment assumption.
A far-sighted receiver is, in general, regarded hard in the literature. Gan et al. [18] prove that information design with a far-sighted receiver is NP-hard. One could intuitively see this from the Recommendation Letter example (See Appendix F). The HR can deliberately choose not to hire any students, even when the signaling scheme satisfies the obedience constraints, hoping to force the professor to be more honest (i.e., reveal more information) in the future. The optimality of the receiver's policy is undefined in this setting.
One way to empirically prevent this is to set an additional constraint  $\int_{\sigma, \sigma'} C_{\varphi}(\sigma, \sigma') d\sigma d\sigma' \geq \epsilon$  apart from the obedience constraints  $C_{\varphi}(\sigma, \sigma') \geq 0$  in Equation (5), where  $\epsilon > 0$ . A more substantial improvement in reward will incentivize the receiver to establish trust in the long run.
4.5.4 Hyper Gradient
Our approach shares similarities with LIO (discussed in Appendix A), as both methods allow agents to alter their parameters to indirectly enhance their payoff expectations by influencing others. In their cases, agents achieve influence by offering rewards to others. The main difference between rewarding and signaling is that the former solely impacts others' policy updates (as the gained rewards are used exclusively for updating). In contrast, the latter additionally affects the sample generation.
In methods to incentivize others, the gradients of the receiver's one-step policy update w.r.t. the sender's rewarding network parameters are required to capture the sender's influence on the receiver. This kind of gradient is second-order and can be viewed as a hyper-gradient.
Unlike incentive-based interventions, in communication methods, the sender's outputs are the inputs of the receiver's actor. The sender can achieve influence while generating trajectories. Hence, our primary focus is on studying the first-order gradient of the receiver's policy w.r.t. the sender's signaling parameters, i.e.,  $\nabla_{\eta}\pi_{\theta}(a\mid o,\sigma)$ . The second-order gradients can also be computed, as shown in Equation (21) in the appendix. The effect of the hyper gradient is left for future work.
4.5.5 Sender's Access to Receiver
In signaling gradient, the updating of the sender's signaling scheme needs to backpropagate through the receiver's policy during training. Accessing the opponent in mixed-motive settings might sound counterintuitive at a glance, but this setting is actually reasonable and technically feasible.
In MARL algorithms, the Centralized Training with Decentralized Execution (CTDE) framework is commonly used [35]. This means that in the simulated environment, data is centralized and accessible during training, which allows us to use all the required quantities. Once the training is completed, the agent no longer has access to other agents.
From the information design perspective, we may consider the receiver as a "dummy" learning agent that is auxiliary in centralized training. Once a sender agent is trained, it finds the (coarse correlated) equilibrium that is no longer connected to specific receivers. The sender is supposed to persuade any rational, self-interested agents in subsequent interactions with them.
Technically, in communication methods, it is common for gradients to pass through other agents during training. We adopted the commonly used Gumbel-Softmax technique in the field of emergent communication research to allow for end-to-end differentiation, which is used to retain the gradients of sampled signals [24, 20].
The method proposed in this paper is validated in Recommendation Letter and Reaching Goals. The receiver's action policy is implemented by the advantage actor-critic (A2C) [38]. Each curve in the experimental result graphs is drawn with at least 15 random seeds. All the seeds are included in the results (including those failed ones). Running 4 seeds with 2 NVIDIA GeForce RTX 3090, the longest time is Reaching Goals with  $5 \times 5$  map, which takes up to a day.
Our designed algorithm takes the perspective of a self-interested sender. Therefore, the measure of success for our algorithm primarily depends on the sender's reward  $r^i$ . However, we also present the curve of  $r^i + r^j$  (social welfare) in an attempt to demonstrate that even when the sender is self-interested and the signaling scheme is somewhat deceptive, it does not harm social welfare, and it might even enhance social welfare in some scenarios.
5.1 Recommendation Letter
Recommendation Letter is a classic example in information design [15, 28]. In this task, a professor will write recommendation letters for a number of graduating students, and a company's human resources department (HR) will receive the letters and decide whether to hire the students.
The professor and the HR share a prior distribution of the candidates' quality, with a probability of  $1/3$  that the candidate is strong and a probability of  $2/3$  that the candidate is weak. The HR does not know exactly what each student's quality is but wants to hire strong students, while the letters are the only source of information. The HR will get a reward of 1 for hiring a strong candidate, a penalty of  $-1$  for hiring a weak candidate, and a reward of 0 for not hiring. The professor gets a 1 reward for each hire. In this section, we focus on analyzing the experimental results, while a classic analysis of the three situations is presented in Appendix F.
In the experiments, we primarily compared the performance of policy gradient class algorithms (PG), PG class algorithms with obedience constraints (PGOC), DIAL [16], signaling gradient (SG, our ablation), and signaling gradient with obedience constraints (SGOC, our proposed method) in learning a signaling scheme. More specifically, the PG class algorithm utilized in experiments refers to A2C. Furthermore, SG also employed A2C techniques, including the use of the actor-critic framework, target critic, and advantage function (adapted to  $W^{i}(s,a) - V^{i}(s)$  in MSGs). The algorithms with obedience constraints are also required to maintain an extra critic for estimating  $W^{j}(s,a)$ . We let  $\varphi_{\eta}(\sigma \mid s,o) = \varphi_{\eta}(\sigma \mid s)$  and  $\Sigma = \{0,1\}$ . The performance comparisons are shown in Figure 2 (a-c).
(a)  $r^i$
(e)  $r^i$  in  $3\times 3$  map
Figure 2: Comparisons of the performance. (a-c) The results of Recommendation Letter. (d-h) The results of Reaching Goals. The rewards and penalties are amplified by 20 and 5 (12 and 3.5) respectively in  $3 \times 3$  ( $5 \times 5$ ) map.
(b)  $r^j$
(f)  $r^j$  in  $3\times 3$  map
(g)  $r^i + r^j$  in  $3 \times 3$  map
(h)  $r^i + r^j$  in  $5 \times 5$  map
The experiment results show that the three examples (two equilibria plus the honest signaling scheme) introduced in Section 1 have emerged. The algorithms with obedience constraints (PGOC and SGOC) reach the third equilibrium (the best for the sender), DIAL reaches the second example (the best for the receiver), and PG and SG reach the first equilibrium (the worst for both parties). Given that the Recommendation Letter task is not a problem with interdependent states, it is justifiable to expect that the performance of PGOC and SGOC (PG and SG) would be similar.
5.2 Reaching Goals
To reflect the inconsistency of the interests and the information asymmetry, we evaluate the methods in the Reaching Goals task. In this challenging task, the sender and receiver have different goals: the sender wants the red apple, while the receiver wants the green apple. At any given time, there is one of each type of apple on the map. Only the receiver is able to move around to reach apples in the grid-world map. The conflict of interest arises because these two types of apples are independently generated at random, and their locations are typically not the same. The receiver will not get a reward for reaching the red apple, but it does not know where the green apple is. The sender, on the other hand, knows the location of both types of apples, but can only get a reward when the receiver reaches the red apple. The sender can only indirectly optimize its payoff by sending messages to influence the receiver's actions. Moreover, this conflict can be exacerbated by designing distance penalties. After each timestep, both agents will get a penalty based on the distance between the receiver and its desired goal. And we set a fixed horizon of 50 for each episode in this scenario. Example maps of Reaching Goals are shown in Figure 6.
To evaluate the efficacy of SGOC, we conducted experiments on maps of  $3 \times 3$  and  $5 \times 5$ , and the results are presented in Figure 2. Let the signal space  $\Sigma = S^1$ , where  $S^1$  is one channel out of three channels (the locations of two apples and the receiver's position) of the image. The receiver can only see its position. The experiments of having a larger  $o^j$  are included in Appendix H.7. The smaller the  $o^j$ , the greater the sender's persuasion ability. From the empirical results, the signaling gradient is shown to be an essential factor in sequential communication scenarios.
5.3 Discussions on Experiments
5.3.1 Symmetricity of the Signaling Schemes
An interesting phenomenon is observed in the Recommendation Letter experiments: training with different random seeds may result in different pairs of encoders and decoders. In other words, the professor may signal 1 to indicate a strong student (or recommend this student) in some seeds, while in others, this is signaled by 0. However, regardless of which case it is, the paired HR can always understand the semantics of the signals (reflected in the evolution of the receiver's policy). Based on the outcomes, the seeds are divided into two parts (A seeds and B seeds).
This phenomenon is reasonable since we do not make any prior assumption about signaling semantics. The symmetric results of Recommendation Letter experiments are shown in Figure 3.
(a)  $\varphi (1\mid \mathrm{"W"}$
(b)  $\varphi(1 | "\mathbb{S}^{\prime \prime})$
(c)  $\pi (\mathrm{^{\prime\prime}H^{\prime\prime}}\mid 0)$
Figure 3: The signaling schemes and the action policy by SGOC. (a) The prob. of signaling 1 for Weak students. (b) The prob. of signaling 1 for Strong students. (c) The prob. of choosing to Hire when signaled 0. (d) The prob. of choosing to Hire when signaled 1.
(d)  $\pi (\mathrm{"H"}\mid 1)$
5.3.2 Honesty of the Sender
As discussed in Section 4.5.3, the obedience constraints can be associated with an additional constraint  $\int_{\sigma,\sigma'} C_{\varphi}^j(\sigma,\sigma') d\sigma d\sigma' \geq \epsilon$ . The hyperparameter  $\epsilon > 0$  is set to improve the credibility of signaling schemes in practical situations and to cope with the RL capacity of the receiver. Define the honesty metric as  $|\varphi(1| \text{"S"}) - \varphi(1| \text{"W"})|$ . The experiments show that the larger the  $\epsilon$  and the Lagrangian  $\lambda$ , the more honest the signaling scheme will be, as shown in Figure 8 in the appendix.
We investigate information design, a substantial and open area, for MARL that discusses mixed-motive communication. Technically, we propose the Markov signaling games to describe the problem and provide its characterizations. We then prove the signaling gradient lemma, which gives an unbiased way to estimate the gradient and update the sender's signaling network. To learn the incentive compatibility of the signaling scheme, we propose extended obedience constraints. The new constraints are more suitable for learning algorithms and practically promote mutually beneficial signaling schemes. The commitment assumption and the revelation principle are lifted by investigating information design with MARL. Experiments and extended discussions are presented to demonstrate the efficacy of our framework and algorithm.
We extend our heartfelt gratitude to Cynthia Huang, Zongqing Lu, Shuai Li, Pascal Poupart, Dan Qiao, Han Wang, Zichuan Wan, Jiachen Yang, Zewu Zheng, Shuhui Zhu for their enlightening discussions. Part of the implementation regarding the signaling gradient was developed by Zewu Zheng.
The authors are partially supported by National Natural Science Foundation of China (62106213, 72150002), Postdoctoral Science Foundation of China (2022M723039), Shenzhen Science and Technology Program (RCBS20210609104356063, JCYJ20210324120011032), and Guangdong Provincial Key Laboratory of Big Data Computing of The Chinese University of Hong Kong, Shenzhen.
[1] Ricardo Alonso and Odilon Camara. Persuading voters. American Economic Review, 106(11): 3590-3605, 2016.
[2] Kai Arulkumaran, Antoine Cully, and Julian Togelius. Alphastar: An evolutionary computation perspective. In Proceedings of the genetic and evolutionary computation conference companion, pages 314-315, 2019.
[3] Dirk Bergemann and Stephen Morris. Information design: A unified perspective. Journal of Economic Literature, 57(1):44-95, 2019.
[4] Dirk Bergemann, Benjamin Brooks, and Stephen Morris. The limits of price discrimination. American Economic Review, 105(3):921-957, 2015.
[5] Martino Bernasconi, Matteo Castiglioni, Alberto Marchesi, Nicola Gatti, and Francesco Trovò. Sequential information design: Learning to persuade in the dark. In Advances in Neural Information Processing Systems, 2022.
[6] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019.
[7] Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[8] Noam Brown and Tuomas Sandholm. Superhuman ai for heads-up no-limit poker: Libratus beats top professionals. Science, 359(6374):418-424, 2018.
[9] Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker. Science, 365 (6456):885-890, 2019.
[10] Kris Cao, Angeliki Lazaridou, Marc Lanctot, Joel Z Leibo, Karl Tuyls, and Stephen Clark. Emergent communication through negotiation. arXiv preprint arXiv:1804.03980, 2018.
[11] Andrea Celli, Stefano Coniglio, and Nicola Gatti. Private bayesian persuasion with sequential games. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1886-1893, 2020.
[12] E Choi, A Lazaridou, and N de Freitas. Compositional obverter communication learning from raw visual input. arXiv preprint arXiv:1804.02341, 2018.
[13] Vincent Conitzer and Tuomas Sandholm. Complexity of mechanism design. In UAI, 2002.
[14] Alvin W Drake. Observation of a Markov process through a noisy channel. PhD thesis, Massachusetts Institute of Technology, 1962.
[15] Shaddin Dughmi. Algorithmic information structure design: A survey. ACM SIGcom Exchanges, 15(2):2-24, 2017.
[51] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems, 1999.
[52] Ina Taneva. Information design. American Economic Journal: Microeconomics, 11(4):151-85, 2019.
[53] Leigh Tesfatsion and Kenneth L Judd. Handbook of computational economics: Agent-based computational economics. Elsevier, 2006.
[54] Richard H Thaler. Anomalies: The ultimatum game. Journal of economic perspectives, 2(4): 195-206, 1988.
[55] Mycal Tucker, Roger Levy, Julie A Shah, and Noga Zaslavsky. Trading off utility, informativeness, and complexity in emergent communication. Advances in Neural Information Processing Systems, 35:22214-22228, 2022.
[56] Jibang Wu, Zixuan Zhang, Zhe Feng, Zhaoran Wang, Zhuoran Yang, Michael I Jordan, and Haifeng Xu. Sequential information design: Markov persuasion process and its efficient reinforcement learning. In Proceedings of the 23rd ACM Conference on Economics and Computation, pages 471-472, 2022.
[57] Jiachen Yang, Ang Li, Mehrdad Farajtabar, Peter Sunehag, Edward Hughes, and Hongyuan Zha. Learning to incentivize other learning agents. Advances in Neural Information Processing Systems, 2020.
[58] Jiachen Yang, Ethan Wang, Rakshit Trivedi, Tuo Zhao, and Hongyuan Zha. Adaptive incentive design with multi-agent meta-gradient reinforcement learning. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, pages 1436-1445, 2022.
[59] Stephan Zheng, Alexander Trott, Sunil Srinivasa, David C Parkes, and Richard Socher. The ai economist: Taxation policy design via two-level deep multiagent reinforcement learning. Science Advances, 8(18):eabk2607, 2022.
[60] You Zu, Krishnamurthy Iyer, and Haifeng Xu. Learning to persuade on the fly: Robustness against ignorance. In Proceedings of the 22nd ACM Conference on Economics and Computation, pages 927-928, 2021.