Learning to Persuade

Abstract: In the standard Bayesian persuasion model, an informed sender looks to design a signaling scheme to partially reveal the information to an uninformed receiver, so as to influence the behavior of the receiver. This kind of strategic interaction abounds in the real world. However, the standard model relies crucially on some stringent assumptions that usually do not hold in reality. For example, the sender knows the receiver's utility function and the receiver's behavior is completely rational.

In this paper, we aim to relax these assumptions using techniques from the AI domain. We put forward a framework that contains both a receiver model and a sender model. We first train a receiver model through interactions between the sender and the receiver. The model is used to predict the receiver's behavior when the sender's scheme changes. Then we update the sender model to obtain an approximately optimal scheme using the receiver model. Experiments show that our framework has comparable performance to the optimal scheme. 

Reviews and author responses:

The paper studies how to relax certain stringent assumptions made in a Bayesian persuasion framework. In particular, the paper looks to relax assumptions where the sender knows the receiver’s utility function and the receiver’s behavior is completely rational. The authors also claim that the proposed framework works on scenario where the receiver does not know the sender’s messaging scheme in advance and needs to learn to optimize his (receiver’s) own objective. The paper also provides some experiments.  The paper is easy to follow and the motivation for relaxing the stringent assumptions in a canonical economic model is fairly well-grounded.  

1. The paper mentions that the proposed framework can be applied to relax several important assumptions made in Bayesian persuasion, e.g., sender doesn’t know receiver’s utility, receiver is not rational, receiver is not knowing sender’s signaling scheme, the signaling space is limited. However, to my best knowledge, simply relax either one of these assumptions is far from trivial.  I’d suggest the authors to focus on relaxing one of them, instead trying to relax them in a single framework.
2. In the paper, the authors mention in several places that the receiver is not rational. There are two rationalities in the Bayesian persuasion: the receiver is expected utility maximizer, the receiver is Bayesian rational (i.e., update the posterior in Bayesian manner). So which rationality the authors refer to? Please clarify this, 
3. The linear program (5) is obtained by applying a revelation principle. When the receiver is not Bayesian rational, I’m not sure if the authors can still apply revelation principle and achieve a similar program as in (5), see the paper “Non-Bayesian Persuasion” by Clippel and Zhang (2019). 
4. Under program (5), “The receiver needs to learn π(m|s) in order to determine q(a|m), since the receiver cannot observe the scheme in advance.” I didn’t understand this sentence, why does the receiver need to determine q(a|m)? Isn’t this receiver’s own belief updating? 
5. “Note that this includes the no-regret learning algorithms in the contextual bandit setting. Because when the receiver is rational and applies a no-regret learning…” I’m not sure how this “no-regret learning algorithms in the contextual bandit setting” relates to the current setting? what is the “context” in the proposed  framework?
6. 1st paragraph of sec 3, “...needs to learn to optimize his own objective”, why does the receiver need to learn to optimize his own objective? so receiver doesn’t know his objective function? 
7. Before program (6), the authors mention that “We omit the constraints described by Equation (3) as they also require the knowledge of the receiver’s utility function.”, But the program (6) still includes the receiver’s utility function? Again, if receiver is not rational, then it is not trivial if you can achieve the same constraint in (6), as the revelation principle may not apply here.
8. in eq (8), it misses a minus sign? 
9. In sec 4, |S| = k, what is k here? If the authors already apply revelation principle, which may not accurate though, then one should have |S| = n. If authors simply just restrict |S| = k < n, I’m not sure if it is a without loss operation. 
10. The figures 2b and 3b seem to show that the proposed framework doesn’t converge to the optimal? 


==== Post-Rebuttal =====\
Thank the authors for the feedback. Taking everything into consideration, I'd like to keep my rating as-is. To summarize, the paper could be strengthened both from theoretical understanding of the assumption relaxations and experiments. Given the current stand, I think it not reaches the bar of ICLR. Hence, my rejection score.

 Thank you for your thoughtful feedback. Our framework is able to scale well. Our framework contains two networks, and both of them have quite simple structures. Thus our framework can still handle much larger games. When the game size increases, the game structure quickly becomes very complex. Consider the receiver's posterior belief space, which can be partitioned into different convex sets with each of them corresponding to a certain action as the receiver's best response. The number of boundary points increases exponentially in the game size (which is the underlying reason for proposition 2). So the performance is expected to degrade a bit for larger games. With such a complex and intrinsic structure of the game, we believe that the performance achieved by our framework is quite impressive.

Program (6) in Section 3 still assumes the receiver is rational. The program is reformulated from program (5) and is then used to extend our framework to allow for irrational receivers.

As for proposition 1, yes, you are right. Proposition 1 only says that with enough data points, we are able to fit the receiver's behaviors well in expectation. The final performance of our approach is difficult to analyze since there is no unified way to describe an irrational receiver. In the worst case, the sender's utility is just her expected utility of sending no message. This is possible in a fully competitive setting, which can be modeled as a zero-sum game between the players.

 Thanks for your feedback. Below we answer your questions point by point.

1. Our initial goal was just to relax the assumption that the sender knows the receiver's utility function. But later we discovered that our framework was also able to relax other assumptions thanks to the flexibility of neural networks. For example, by simply changing the output dimension, our framework can be used to handle the case with restricted signaling spaces.

2. We do not distinguish between these two types of rationalities. In fact, with our framework, there is no need to do so. In our framework, the receiver model is an end-to-end model that maps the sender's signaling scheme directly to a strategy. How the receiver actually does the mapping (whether by using a non-Bayesian belief update or by not maximizing the utility, or even both) is not important in our framework, as long as the receiver's behavior satisfies the assumption we made at the end of Section 2.

3. Yes, you are right. Without the rationality assumption, program (5) no longer holds. As we mentioned at the beginning of the preliminaries section, we consider rational receivers and briefly introduce the standard Bayesian persuasion model in this section to make our paper self-contained. And later, in Section 3, we derive our framework and modify program (5) to allow for irrational receivers.

4. In our setting, the sender may not be able to send the receiver her own signaling scheme (because of the complexity of the scheme or lack of such a communication channel). In this case, the receiver cannot do the belief update and has to learn the scheme through repeated interactions with the sender. We will move the last paragraph of Section 2 to Section 3 to avoid confusion.

5. The context is simply the message sent by the sender. From the receiver's point of view, the problem he faces is just a contextual bandit problem, where each action corresponds to a certain "arm" of the bandit. And if the receiver applies a no-regret learning algorithm, given enough time, he will be able to identify the best arm (the best action) almost surely.

6. The receiver needs to learn because he does not know the sender's signaling scheme in this case (mentioned in the same sentence in the paper). If the receiver does not know the signaling scheme in advance, he cannot update his belief about the state and thus cannot choose the action that actually maximizes his utility.

7. Yes, program (6) still contains the receiver's utility. However, the argmax operation in the program is supposed to be completed by the receiver instead of the sender (mentioned in the first sentence right below the program in our paper). In the entire process, the sender does not need to know the receiver's utility. And we do not require the best action a(m) given message m is different for different m's (which is required in Equation (3)), since the revelation principle does not apply here.

8. Yes, a minus sign is missing. Thanks for pointing that out.

9. k is the number of states, which is different from the number of actions (which is n). The revelation principle suggests that the number of messages should be no more than the number of actions if the receiver is rational. This means |M| should be equal to n.

10. In Figure 2(b), the optimal solution is obtained via solving the linear program. Our framework uses a softmax activation function at the end of the receiver model to approximate the receiver's best response. Thus the experiment result is expected to have worse performance compared to the optimal solution. For Figure 3(b), our framework is applied to the setting where the number of messages (16) is less than the number of actions (32). While the optimal solution is computed assuming the sender can still use 32 messages. So the optimal solution is simply an upper bound. We compare these two solutions because finding the actual optimal solution for the case with only 16 messages is an NP-hard problem, and requires enumerating all (32 choose 16 =) 601080390 possible messages combinations.

 Thank you for your comments. In the paper, we proposed a framework to address the problem that the sender does not know the receiver's utility function. We think that our framework is simple and practical. Our framework can address not only the problem that the receiver's utility function is unknown to the sender, but also the problem that the receiver is irrational. Furthermore, if the number of messages is restricted (which is an NP-hard problem), our framework can also perform well.

We consider deep learning as a powerful tool. It is easy to implement in practice, yet has reasonably good performance at the same time. There are also some works using neural networks to solve related problems, such as (Duetting, Paul, et al. "Optimal Auctions through Deep Learning." ICML. 2019), (Feng, Zhe, Harikrishna Narasimhan, and David C. Parkes. "Deep learning for revenue-optimal auctions with budgets." AAMAS. 2018.) which uses a neural network structure to find optimal auctions, and (Golowich, Noah, Harikrishna Narasimhan, and David C. Parkes. "Deep Learning for Multi-Facility Location Mechanism Design." IJCAI. 2018.). Another reason why we choose to use neural networks to solve the problem is the flexibility of neural networks. Such flexibility allows us to apply our framework to a large class of irrational receivers, while most known approaches do not. There are some existing works that apply robust optimization approaches to the Bayesian persuasion setting. However, most of these works make some assumptions and thus cannot be applied to our setting. On the other hand, the aim of this paper is to propose a simple yet powerful framework for optimizing the sender's utility other than the robustness. 

In section 3.1, the input of the sender's model is simply constant 1, which has no practical meaning but is necessary when using modern deep learning frameworks such as PyTorch and TensorFlow. The output of the sender's model is a table, which has |S| rows and |M| columns. Each row represents the conditional probability p(m|s). The whole table is the sender's messaging scheme.

We actually take into consideration the cost during the learning phase. In Section 4, Proposition 1 is about the relation between the number of sampled data and the generalization error. Proposition 2 says that we need to sample exponentially many data points in order to accurately capture the receiver's behavior pattern. So we use two loops in our proposed algorithm to accelerate the learning process. However, our empirical analysis shows that in most cases, we do not need to sample exponentially many data points to obtain a reasonably good performance.

\hat{R}_1 is the predicted payoff of the sender. It is computed through the following steps. When the sender's messaging scheme is generated. It will be the input of the receiver model. The receiver model will output the predicted probability of the receiver's actions. Based on the predicted action probability, the sender can then estimate her own expected payoff. Note that the predicted probability may be different from the receiver's actual strategy. If the receiver model can make accurate predictions, the predicted utility \hat{R}_1 will be close to the true utility R_1.

The worst-case performance guarantee is simply the sender's utility of sending no message, which is achieved when the players are playing a zero-sum game. It can also be achieved when an irrational receiver ignores his own utility and focuses on minimizing the sender's utility. In general, when the receiver is irrational, this problem is non-convex in most cases. In addition, it is usually very difficult to analyze the convergence and performance guarantee of a neural network. Our algorithm always converges to high-quality solutions in our experiments.

 Thank you for your comments. we believe that you have some misunderstandings about our paper. Although the standard Bayesian persuasion setting can be modeled in the Stackelberg fashion, it is actually very different from Stackelberg games in the following ways:
1. In a Stackelberg game, for any sender's strategy, the follower's best response is simply a single action. However, in the Bayesian persuasion model, the receiver's strategy depends on the sender's message. Or equivalently, the receiver's strategy is a function of the sender's message. Thus, there can be exponentially many different strategies for the receiver, making it very difficult to apply learning algorithms from Stackelberg games to this setting.
2. In the Bayesian persuasion model, upon receiving a message, the receiver needs to update his belief about the state using the Bayes rule. However, in Stackelberg games, the follower does not need this step to determine the best response to the leader's commitment.
Essentially, Bayesian persuasion models a setting with information asymmetry, and the persuasion scheme is used by the sender to leverage this information asymmetry. While in Stackelberg games, there is no information asymmetry.

When faced with a learning sender, it is possible that the receiver tries to deceive the sender. However, our setting also considers irrational receivers. It is unclear why an irrational receiver is even able to deceive the sender in such a complex setting. In our paper, we assume that the receiver uses a certain class of learning algorithms that include no-regret learning algorithms. We believe this class is general enough for our setting. We agree that the deception problem is an interesting future research direction, but it is not the focus of our paper.

The main contribution of our paper is not the irrational behavior model (see Section 1.1 for a summary of our contributions). Instead, we proposed a framework based on neural networks to tackle the problem where the sender does not know the receiver's utility function. The framework is intuitive and easy to implement. Then we discovered that this framework is also applicable when the receiver's behavior can be described using a mapping from messages to actions. This includes cases where the receiver is irrational, and where the receiver's behavior cannot even be described using utility functions. 

When the receiver is irrational, it is not always the case that the sender can get higher payoffs. Consider the following example. Assume the sender's and the receiver's utility functions are the same. So the sender and the receiver are cooperative. The optimal strategy of the sender is to reveal full information to the receiver. If the receiver is rational, the receiver will follow the recommendations of the sender. So the sender's and the receiver's payoffs will both reach the maximum. However, if the receiver is not rational, the receiver may not follow the recommendations (for example, the receiver may just ignore the message from the sender and always use a certain action). Then the sender's and the receiver's payoffs will not reach the maximum.

The baseline paper (AAMAS'16) mentioned by the reviewer considers a repeated security game, which is a special case of Stackelberg games. Their algorithm does not apply to our setting. Their setting is essentially an adversarial bandit setting and they propose a polynomial-time algorithm with low regret. However, this algorithm cannot even apply to general Stackelberg games, since it is known that no algorithm can guarantee to learn the optimal strategy in polynomial time.

The paper suggests using neural networks to solve the nested optimisation problem of designing a disclosure scheme in persuasion schemes. This allows authors to remove the assumption of prior knowledge of receiver utilities by the sender, as well as rationality assumption of the receiver. Authors theoretically show that their approach correctly simulates the necessary optimisation patterns, and the paper concludes with a set of experiments that demonstrate that alternative solutions, that do not solve the optimisation directly, are less effective in achieving same effective persuasion levels.  It arguable whether a receiver, that needs to learn the commitment strategy of the sender and is aware of the sender's parallel adaptation, will miss this opportunity to manipulate the outcome. In fact, such manipulation is a serious concern for adaptive senders. 

Consider, e.g., some of the missed related work:
"Optimally deceiving a learning leader in Stackelberg games" by Birmpas, Gan et al, JAIR, 72: 507-531, 2021 (a follow up paper to an older conference publication)
"Imitative follower deception in stackelberg games" by Gan et al, ACM EC, 2019
"Manipulating a Learning Defender and Ways to Counteract" by Gan et al, NeurIPS, 2019

The use of non-rational behaviour model is also not novel. Consider works by Azaria, Kraus et al, where a response model of a human actor (non-rational by nature) was learned. Milind Tambe's team has quite a few similar results as well. Again, missing from the related work. However, a further point of these previous results is that non-rational behaviour is quite often sub-optimal and open the possibility for further exploitation by the sender. In other words, if the receiver's model is in any way (near) stable, then the learned commitment strategy of the sender can potentially generate higher payoffs. Alas, this is not what Table-1 and Table-2 of the current paper show. Rather, once the non-rational assumption is engaged, the sender's payoff drops below its original visa vi rational-receiver levels. Authors do not notice this, nor discuss the issue. 

Finally, a better baseline to consider would be a regret-minimising approach:
"Playing Repeated Security Games with No Prior Knowledge"
by Xu, Tran-Thanh and Jennings
AAMAS 2016
 The paper suggests a straight forward application of ML to solve persuasion optimisation with a reduced set of assumptions. Which is ok, but rather week, contribution. If the overall approach of using learning to supplant unknown elements was novel -- would be a publishable paper -- however, there's a lot of previous work where this has been used before, but authors missed it.


The paper aims to remedy limitations of standard model-based approach of Bayesian persuasion by using machine learning methods. The authors presented an algorithm, which uses a neural network in the loop, provided theoretical analysis, and presented empirical results to prove the performance of their approach. The paper starts with a good motivation to remedy practical shortcomings of the Bayesian persuasion model, but the results presented fall largely short of getting there. 

First fo all, there are many approaches in the literature on human behavior modeling and robust optimization, which could potentially be used to address the issues of imperfect rationality and incomplete knowledge. It is unclear how necessary it is to use a compelely new approach based on neural networks to solve the problem. If the authors' motivation is to address the limitations of the standard idealized model, then I feel that they should try these well-studied approaches first, or at least demonstrate that these appraches work no better than their machine learning approach. There is no such comparison in the paper. The experiments, in particular, did not use any such benchmarks and therefore the results a bit weak in demonstrating the strength of their contribution.

The learning algorithm also has several shortcomings on its own. The sender's model described in Section 3.1 is unclear to me. What are the input to and output of the neural network? In Algorithm 1, the authors also seem to have completely ignored the cost during the learning phase, the time it takes for the algorithm to converge at Line 3, and then in the while loop from Line 5-10. The algorithm also looks loose in some other steps. E.g., how is $\hat{R}_1$ predicted? There is no rigorous analysis to the performance of the algorithm, in terms of convergence and the quality of solutions it produces. Section 4 did offer some analysis, but it falls back with the assumption of a completely rational receiver, making it questionable how meaningful the analysis is.  A paper with a good motivation but falls short in fulfilling the proposed objective. The contribution is not significant.  

The authors propose a differentiable, end-to-end formulation for the standard Bayesian persuasion problem.  The sender's and the receiver's strategies are modeled using neural networks, and the authors propose using a version of alternative training to simultaneously optimize the sender's strategy and predict the receiver's response.  The authors analyze some theoretical properties of the approach in an idealized setting, and conduct experiments on synthetic data to evaluate the performance of their approach.  The empirical results show that when the receiver observes the sender's entire strategy, their approach achieves about 90% of the optimal sender's utility for 32 states and 32 actions.  When the receiver only observes individual messages (and therefore needs to learn the best response), the proposed approach significantly outperform baseline methods (which are not specifically designed for this task).  The results also show diminishing marginal utility gain when the space of messages allowed becomes larger. ## Strengths

The problem of Bayesian persuasion is of great theoretical and practical importance.  The method proposed doesn't require complete knowledge of the receiver's utility model, or even the assumption that there is one (i.e., the receiver might be irrational).  The method appears fairly simple (which I think is good), and achieves impressive performance in the experiments, which all suggest its practical potential.


## Weaknesses

I think the main contribution of the paper is empirical, so my main concern is whether the proposed method scales well.  As the experiments show, the performance of the method degrades as the game becomes larger.  I'd be more excited if the authors could provide more evidence that the method would likely perform well in some practical scenarios.


## Detailed comments

Formulation in Section 3: technically speaking, this still assumes the receiver is rational (i.e., utility-maximizing), right?  (The model in Section 3.2 can actually model (at least some) irrational behavior so that's fine.)

First paragraph in Section 3.1: "Each entry in the table are ..."

Proposition 1: this doesn't make much practical sense to me for a number of reasons.  First, \Pi (presumably being the set of all messaging schemes obtainable from training) probably should be infinite (or really large if we somehow restrain the training procedure) for the framework considered in this paper.  Moreover (and more importantly), the bound conceptually corresponds to a somewhat twisted version of the problem: the sender, for whatever reason, decides to use a meta-randomized scheme, where upon seeing each state, the sender first flips a coin to decide which scheme she uses, and then she commits to and announces that scheme, and flips another coin to decide the actual message that she sends.  We (the learner), observe the receiver's best response for a while, and find a (deterministic) model that fits our observations best.  Then the generalization bound says if our model fits our observations well, then it will probably fit the receiver's behavior well in expectation over all the randomness, including that involved in the sender's meta-randomized scheme.  Note that this says nothing about how well our model fits the receiver's actions in response to any particular messaging scheme.  In particular, our model could still be far off when the sender uses the "optimal" messaging scheme (this could mean the optimal scheme against the receiver's actual actions, or what we think is optimal given our model of the receiver).  If that's the case, then the bound doesn't say much about the (idealized) performance of the proposed approach. Overall I think the paper proposes a novel differentiable approach to Bayesian persuasion, which is then validated experimentally.  The theoretical results are good to know, but I don't think they are particularly helpful for practical purposes; the experimental results overall support the claims, but may or may not scale well.

Meta-review: The paper studies the Bayesian persuasion model in a more realistic setting where the sender does not know the receiver’s utility but can interact with the receiver repeatedly to learn the utility. The paper proposes a learning-based framework to optimize the sender’s strategy, then analyze the theoretical properties of the proposed framework, and perform extensive experiments. The reviewers acknowledged that the paper investigates an important problem of relaxing the practical shortcomings of the Bayesian persuasion model. However, the reviewers pointed out several weaknesses in the paper, and there was a clear consensus that the work is not ready for publication. The reviewers have provided very detailed and constructive feedback to the authors. We hope that the authors can incorporate this feedback when preparing future revisions of the paper.

Decision: Reject