Learning to Persuade

1 INTRODUCTION . The Bayesian persuasion model has recently been extensively studied in economics , computer science and operations research ( Kamenica & Gentzkow , 2011 ; Arieli & Babichenko , 2019 ; Dughmi & Xu , 2019 ; Lingenbrink & Iyer , 2019 ) . This model considers the problem of how an informed player ( the sender ) can influence the behavior of an uninformed , self-interested player ( the receiver ) . The model was firstly proposed by Kamenica & Gentzkow ( 2011 ) , and has been successfully applied to various domains such as security games ( Xu et al. , 2015 ; Rabinovich et al. , 2015 ; Shen et al. , 2020 ) , auctions ( Badanidiyuru et al. , 2018 ; Shen et al. , 2019 ) , voting ( Alonso & Câmara , 2016 ; Cheng et al. , 2015 ) and recommendation systems ( Mansour et al. , 2016 ) . In the Bayesian persuasion model , there are two players : a sender ( she ) and a receiver ( he ) . The sender can observe a state of nature that is randomly drawn from a common prior . The receiver does not have access to the state , but can play an action . Before the receiver chooses an action to play , the sender can send a message to the receiver about the observed state . Both the two players ’ utilities depend on the state as well as the receiver ’ s action . The core problem lies on how the sender can design a messaging scheme to maximize her utility by partially revealing the information to the receiver to influence the receiver ’ s behavior . Although the Bayesian persuasion model has nice mathematical structures in theory , it also relies heavily on some assumptions that often do not hold in reality . For example , it is assumed that the sender perfectly knows receiver ’ s utility and that the receiver is completely rational . Another crucial assumption is that the sender has the so-called commitment power and is able to convey her commitment to the receiver before the game begins . However , in many applications , the receiver can not get the accurate commitment due to lack of such communication channels , or simply because the whole commitment is too complicated to understand . The application of the model has been greatly restricted by such strong assumptions . In this paper , we aim to relax these unrealistic assumptions and consider the setting where the sender only knows his own utility but not the receiver ’ s . However , the sender can learn the receiver ’ s utility or behaviors through interacting with the receiver repeatedly . Such a setting captures how the sender could affect the receiver ’ s behavior in many applications ( e.g. , auctions , recommendation systems ) . Instead of game-theoretic analyses , we make use of tools from the AI domain to model the receiver , and apply a data-driven approach to optimize the sender ’ s strategy . In particular , we use two neural networks to model the sender and the receiver . The sender network maps observed states to strategies of sending different messages to the receiver , while the receiver network maps received messages to strategies of playing actions . Note that our receiver model is an end-to-end model and does not involve his utility function . Such a modeling choice is general enough to be applied to cases where the receiver is not fully rational . 1.1 OUR CONTRIBUTIONS . In this paper , we propose a framework for learning the sender ’ s optimal messaging scheme . The framework contains three components : a sender network , a receiver network , and an algorithm for optimizing the sender ’ s expected utility by repeatedly interacting with the receiver . When optimizing the sender ’ s messaging scheme , our algorithm takes into account how changes in sender ’ s scheme affects the receiver ’ s behavior . We analyze theoretic properties of the proposed framework . We show that the receiver model is PAC-learnable , but at the same time may take exponentially many interactions to train the receiver model . We also conduct extensive experiments to demonstrate the effectiveness of our framework . 1.2 ADDITIONAL RELATED WORKS . The Bayesian persuasion model was first studies by researchers from the domain of economics ( Brocas & Carrillo , 2007 ; Kamenica & Gentzkow , 2011 ) . Brocas & Carrillo ( 2007 ) considers the case with only a couple of states , and Kamenica & Gentzkow ( 2011 ) later generalized the model to any finite number of states and actions . Bergemann et al . ( 2015 ) studies the market segmentation problem , and Shen et al . ( 2018 ) showed that the problem is equivalent to a persuasion problem in designing optimal auctions . The persuasion model is also studied in security games to fight illegal poaching and urban crimes ( Xu et al. , 2016 ; 2018 ; Bondi et al. , 2020 ) . There is also a line of works that focuses on learning the persuasion scheme . The most relevant work is by Bhatt & Buro ( 2021 ) , who formulate the problem of learning a messaging scheme as a multiagent communication problem , and propose two algorithms called Info-Q and Info-Policy . However , they focus on fully cooperative settings and only consider deterministic messaging schemes . Zu et al . ( 2021 ) study the learning problem when the prior distribution over the states is unknown . They provide an algorithm with sub-linear regret . Camara et al . ( 2020 ) also study a mechanism design setting without the common prior assumption . They give regret bounds for the mechanism designer compared to the best mechanism in hind sight . Babichenko et al . ( 2021 ) also considers the setting where the receiver ’ s utility is unknown . However , they focus on the case with only two actions and aim to provide a messaging scheme that performs well with all possible receiver utilities . Castiglioni et al . ( 2020 ) study the setting where the receiver has an unknown type that is chosen adversarially . They show that there is no efficient algorithm in this case but can achieve sub-linear regret . 2 PRELIMINARIES . In the standard Bayesian persuasion model , there are two players : a sender and a receiver , denoted by subscripts 1 and 2 , respectively . The receiver needs to make a decision by choosing an action from a set of possible actions A . Assume that both players ’ utility functions depend on a state of nature s ∈ S , which is drawn from a publicly known prior distribution p. In particular , the players ’ utility functions are ui : S × A 7→ R , i ∈ { 1 , 2 } , respectively . We focus on the case where both A and S are finite . We assume that the sender can observe the actual , realized state s , while the receiver has no access to it throughout the game . The sender can send a message m ∈ M to the receiver before he makes the decision to reveal information about the state s. Before the game starts , the sender can commit to a certain messaging scheme π ( m|s ) ∈ [ 0 , 1 ] , which is the probability of sending message m conditioned on that she observes state s. The standard persuasion model also assumes that the sender has commitment power ( the sender acts exactly as the messaging scheme they commit to ) , and that the receiver also knows π ( m|s ) . Therefore , when the receiver gets message m , he will update his belief about the state according to the Bayes ’ rule : p ( s|m ) = π ( m|s ) p ( s ) ∑ s′ π ( m|s′ ) p ( s′ ) . ( 1 ) Based on the posterior belief , the receiver then selects an action to maximize his expected utility : a ∈ arg max a′ ∑ s u2 ( s , a ′ ) p ( s|m ) = arg max a′ ∑ s u2 ( s , a ′ ) π ( m|s ) p ( s ) , . ( 2 ) An example of how the persuasion process works is provided in Appendix A . The core problem is to design an optimal messaging scheme to maximize the sender ’ s expected utility , subject to the constraint that the receiver always chooses the best action to him . Such a problem can be formulated as a linear program , and thus can be efficiently solved . With arguments analogous to the revelation principle ( Gibbard , 1973 ; Myerson , 1981 ) in the mechanism design theory , it is not difficult to show that an optimal messaging scheme needs to use no more than n messages , where n = |A| is the total number of actions available to the receiver . Therefore , it suffices to focus on the case with n messages and each message can actually be interpreted as an action recommendation , i.e. , the sender recommends the receiver to take action ai by sending him message mi . However , in order for the receiver to follow the sender ’ s recommendation , we need to pose a constraint on the sender ’ s messaging scheme : ∑ s∈S π ( mi|s ) u2 ( s , ai ) ≥ ∑ s∈S π ( mi|s ) u2 ( s , ai′ ) , ∀1 ≤ i , i′ ≤ n. ( 3 ) With the above constraints , the receiver has no incentive to choose any action other than the one recommended by the sender . Of course , the messaging scheme π should also satisfy the following feasibility constraints : n∑ i=1 π ( mi|s ) = 1 , ∀s ∈ S and π ( mi|s ) ≥ 0 , ∀1 ≤ i ≤ n , ∀s ∈ S ( 4 ) Therefore , we can formulate the problem as the following linear program : maximize : ∑ s∈S n∑ i=1 p ( s ) π ( mi|s ) u1 ( s , ai ) subject to : Constraint ( 3 ) and ( 4 ) ( 5 ) In our setting , we allow irrational behaviors of the receiver . Let q ( a|m ) be the receiver ’ s probability of choosing action a after receiving message m. If the receiver is rational , q ( a|m ) is simply a pure strategy . Otherwise , q ( a|m ) can be any probability distribution over A . We make the assumption that q ( a|m ) only depends on the messaging scheme π ( m|s ) ( as well as the prior p ( s ) , which is constant ) and thus we sometimes write qπ ( a|m ) to emphasize the dependence on π . The receiver needs to learn π ( m|s ) in order to determine q ( a|m ) , since the receiver can not observe the scheme in advance . We further assume that if the sender ’ s empirical messaging scheme converges to π , then the receiver can correctly learn π and the empirical frequency of the receiver ’ s actions also converges to qπ . Note that this includes the no-regret learning algorithms in the contextual bandit setting . Because when the receiver is rational and applies a no-regret learning algorithm , his strategy would surely converge to the one defined in Equation ( 2 ) , since otherwise , the learning algorithm can not have sub-linear regrets .

Summary: The authors propose a differentiable, end-to-end formulation for the standard Bayesian persuasion problem.  The sender's and the receiver's strategies are modeled using neural networks, and the authors propose using a version of alternative training to simultaneously optimize the sender's strategy and predict the receiver's response.  The authors analyze some theoretical properties of the approach in an idealized setting, and conduct experiments on synthetic data to evaluate the performance of their approach.  The empirical results show that when the receiver observes the sender's entire strategy, their approach achieves about 90% of the optimal sender's utility for 32 states and 32 actions.  When the receiver only observes individual messages (and therefore needs to learn the best response), the proposed approach significantly outperform baseline methods (which are not specifically designed for this task).  The results also show diminishing marginal utility gain when the space of messages allowed becomes larger.