Learning to Persuade
Authors: Xiaodong Liu
Zhikang Fan
Xun Wang
Weiran Shen
Venue: ICLR 2022 Submitted (2022)

Abstract: In the standard Bayesian persuasion model, an informed sender looks to design a signaling scheme to partially reveal the information to an uninformed receiver, so as to influence the behavior of the receiver. This kind of strategic interaction abounds in the real world. However, the standard model relies crucially on some stringent assumptions that usually do not hold in reality. For example, the sender knows the receiver's utility function and the receiver's behavior is completely rational.

In this paper, we aim to relax these assumptions using techniques from the AI domain. We put forward a framework that contains both a receiver model and a sender model. We first train a receiver model through interactions between the sender and the receiver. The model is used to predict the receiver's behavior when the sender's scheme changes. Then we update the sender model to obtain an approximately optimal scheme using the receiver model. Experiments show that our framework has comparable performance to the optimal scheme. 

Under review as a conference paper at ICLR 2022
LEARNING TO PERSUADE
Anonymous authors Paper under double-blind review
ABSTRACT
In the standard Bayesian persuasion model, an informed sender looks to design a signaling scheme to partially reveal the information to an uninformed receiver, as to influence the behavior of the receiver. This kind of strategic interaction abounds in the real world. However, the standard model relies crucially on some stringent assumptions that usually do not hold in reality. For example, the sender knows the receiver's utility function and the receiver's ' behavior is completely ra- tional.
In this paper, we aim to relax these assumptions using techniques from the Al domain. We put forward a framework that contains both a receiver model and a sender model. We first train receiver model through interactions between the sender and the receiver. The model is used to predict the receiver's 's behavior when the sender's scheme changes. Then we update the sender model to obtain an approximately optimal scheme using the receiver model. Experiments show that our framework has comparable performance to the optimal scheme.
INTRODUCTION
The Bayesian persuasion model has recently been extensively studied in economics, computer sci- ence and operations research (Kamenica & Gentzkow 2011 Arieli & Babichenko, 2019 Dughmi & Xu 2019 Lingenbrink & lyer 2019). This model considers the problem of how an informed player (the sender) can influence the behavior of an uninformed, self-interested player (the receiver). The model was firstly proposed by Kamenica & Gentzkow (201T), and has been successfully applied to
various domains such as security games (Xu et al. 2015 Rabinovich et al.) 2015 (Shen et al. 2020).
auctions (Badanidiyuru et al. 2018 Shen et al. 2019). voting (Alonso & Câmara 2016 Cheng
et al. 2015) and recommendation systems (Mansour et al. 2016)).
In the Bayesian persuasion model, there are two players: sender (she) and a receiver (he). The sender can observe state of nature that is randomly drawn from common prior. The receiver does not have access to the state, but can play an action. Before the receiver chooses an action to play, the sender can send message to the receiver about the observed state. Both the two players' utilities depend on the state as well as the receiver's action. The core problem lies on how the sender can design messaging scheme maximize her utility by partially revealing the information the receiver to influence the receiver's 's behavior.
Although the Bayesian persuasion model has nice mathematical structures in theory, it also relies heavily on some assumptions that often do not hold in reality. For example, it is assumed that the sender perfectly knows receiver's utility and that the receiver is completely rational. Another crucial assumption is that the sender has the so-called commitment power and is able to convey her commitment to the receiver before the game begins. However, in many applications, the receiver cannot get the accurate commitment due to lack of such communication channels, or simply because the whole commitment is too complicated to understand. The application of the model has been greatly restricted by such strong assumptions.
In this paper, we aim to relax these unrealistic assumptions and consider the setting where the sender only knows his own utility but not the receiver's. However, the sender can leam the receiver's utility or behaviors through interacting with the receiver repeatedly. Such a setting captures how the sender could affect the receiver's is behavior in many applications (e.g., auctions, recommendation systems).
Instead of and apply data-driven approach optimize the sender's strategy. In particular, we use two neural
analyses, we make use of tools from the AI domain to model the receiver,

---

Under review as a conference paper at ICLR 2022
networks to model the sender and the receiver. The sender network maps observed states to strategies of sending different messages to the receiver, while the receiver network maps received messages to strategies of playing actions. Note that our receiver model is end-to-end model and does not involve his utility function. Such a modeling choice is general enough to be applied to cases where the receiver is not fully rational.
1.1 OUR CONTRIBUTIONS
In this paper, we propose framework for learning the sender's optimal messaging scheme. The framework contains three components: a sender network, receiver network, and an algorithm for optimizing the sender's expected utility by repeatedly interacting with the receiver. When optimizing the sender's messaging scheme, our algorithm takes into account how changes in sender's 's scheme affects the receiver's behavior. We analyze theoretic properties of the proposed framework. We show that the receiver model is PAC-learnable, but at the same time may take exponentially many interactions to train the receiver model. We also conduct extensive experiments to demonstrate the effectiveness of our framework.
1.2 ADDITIONAL RELATED WORKS
The Bayesian persuasion model was first studies by researchers from the domain of economics (Brocas & Carrillo 2007 Kamenica & Gentzkow 2011). Brocas & Carrillo (2007) considers the case with only a couple of states, and Kamenica Gentzkow (2011) later generalized the model to any finite number of states and actions. [Bergemann et al. (2015) studies the market segmentation problem, and [Shen et al.] (2018) showed that the problem is equivalent to a persuasion problem in designing optimal auctions. The persuasion model is also studied in security games to fight illegal poaching and urban crimes (Xu et al. 2016 2018 Bondi et al. 2020).
There is also a line of works that focuses on learning the persuasion scheme. The most relevant work is by Bhatt& & Buro (202I), who formulate the problem of learning a messaging scheme as a multi- agent communication problem, and propose two algorithms called Info-Q and Info-Policy. However, they focus on fully cooperative settings and only consider deterministic messaging schemes. Zu et al. (2021) study the learning problem when the prior distribution over the states is unknown. They provide an algorithm with sub-linear regret. Caaaara et al. (2020) also study a mechanism design setting without the common prior assumption. They give regret bounds for the mechanism designer compared to the best mechanism in hind sight. [Babichenko et al. (20TT) also considers the setting where the receiver's utility is unknown. However. they focus on the case with only two actions and aim to provide a messaging scheme that performs well with all possible receiver utilities. Castiglioni al. (2020) study the setting where the receiver has an unknown type that is chosen adversarially. They show that there is no efficient algorithm in this case but can achieve sub-linear regret.
PRELIMINARIES
In the standard Bayesian persuasion model, there are two players: sender and a receiver, denoted by subscripts 1 and 2, respectively. The receiver needs to make a decision by choosing an action from set of possible actions A. Assume that both players' utility functions depend on a state of nature E S, which is drawn from publicly known prior distribution p. In particular, the players' utility functions are u, x AA + I,, i E {1, 2}. respectively. We focus on the case where both A and S are finite. We assume that the sender receiver has no access to it throughout the game. The sender can send a message m E M to the receiver before he makes the decision to reveal information about the state s. Before the game starts, the sender can commit to a certain messaging scheme π [ s)) E [0, 1). which is the probability of sending message m conditioned on that she observes state 8. The standard persuasion model also assumes that the sender has commitment power (the sender acts exactly as the messaging scheme they commit to), and that the receiver also knows π (m s)) Therefore, when the receiver gets message m, he will update his belief about the state according to the Bayes' rule:
observe the actual, realized state s, while the
T(m]s)p(s)
p(s|m) Σ. ( s(ss'))
(I)
2

---

Under review as a conference paper at ICLR 2022
Based on the posterior belief, the receiver then selects an action maximize his expected utility:
a E arg max
Σ (')(s,))p(ssm) = arg max a'
Σ [ul(s.d)r(mlsspps))
(2)
a'
An example of how the persuasion process works is provided in Appendix A to design an optimal messaging scheme to maximize the sender's expected utility, subject the constraint that the receiver always chooses the best action to him. Such problem can be formulated as linear program, and thus can be efficiently solved.
The core problem
With arguments analogous to the revelation principle (Gibbard 1973 Myerson 1981) in the mecha- nism design theory, it not difficult to show that an optimal messaging scheme needs to use no more than n. messages, where n |A| is the total number of actions available to the receiver. Therefore, suffices to focus on the case with n messages and each message can actually be interpreted as an action recommendation, i.e., the sender recommends the receiver to take action a, by sending him message my. However, in order for the receiver to follow the sender's recommendation, we need to pose constraint the sender's messaging scheme:
Σ (mmlls)u((s,a) 2 Σ m(mals)u2(ssas) S i, S
#ES
With the above constraints, the receiver has no incentive choose any action other than the one recommended by the sender. Of course, the messaging scheme x should also satisfy the following feasibility constraints:
Σ (m;ls)) 1, vs E S and (m,,ss) 2 0, L1 visisn,Vse
(4)
Therefore, formulate the problem the following linear program:
maximize:
((s)
(5)
#ES
subject to: Constraint
3
and
In our setting, we allow irrational behaviors of the receiver. Let g(a (m) be the receiver's probability of choosing action a after receiving message m. If the receiver is rational, g(a(m) is simply a pure strategy. Otherwise, q(a m)) can be any probability distribution over A. We make the assumption that (aamm only depends on the messaging scheme ा [m/ss (as well as the prior p(s), which is constant) and thus we sometimes write 9x (a/m)) emphasize the dependence on म. The receiver needs to learn (m]s) in order to determine g(a|m), since the receiver cannot observe the scheme in advance. We further assume that if the sender's empirical messaging scheme converges to m then the receiver can correctly learn and the empirical frequency of the receiver's actions also converges to qa Note that this includes the no-regret learning algorithms in the contextual bandit setting. Because when the receiver is rational and applies no-regret learning algorithm, his strategy would surely converge to the one defined in Equation 2 since otherwise, the learning algorithm cannot have sub-linear regrets.
OUR FRAMEWORK
In this section, we present our framework for learning the optimal messaging scheme. We consider the setting where the sender does not know the receiver's utility function (hence the receiver's 's be- havior pattern). Our framework also applies to the setting where the receiver is irrational or where the receiver does not know the sender's messaging scheme in advance and needs to learn to optimize his own objective.
We mainly focus the messaging scheme design problem where the number of the sender's mes- sages are equal to the number of the receiver's actions. We omit the constraints described by Equa- tion (3) as they also require the knowledge of the receiver's utility function. Thus message m, does
3

---

Under review as a conference paper at ICLR 2022
not necessarily represent that action o; is indeed the best choice for the receiver. However, re-formulate the above linear program as follows:
maximize:
Σ Σ
p(s)m((ls)uusss, a(m))
#ES MEM
subject to: a(m) E arg max Σ
E((s)m)u2(ssa) vm E M
(6)
#ES
Constraint
The above optimization problem actually becomes bi-level optimization problem, where the re- ceiver completes the "arg max" operation. Therefore, when optimizing the sender's messaging scheme, must take the receiver's behavior into account. Otherwise, the receiver's ' behavior may change as we update the messaging scheme, which can even lower the sender's expected utility. So cannot simply view the receiver or bandit-based algorithm to optimize the sender's s scheme, since the "environment" not stable due the strategic behaviors of the receiver. We propose resenting the sender's 's messaging scheme and the other describing receiver's behavior. The overall structure of our framework is shown in Figure
given "environment" and apply any reinforcement learning
neural networks, with one rep-
Lowal
Figure I: The structure of our framework.
3.1 SENDER'S 'S MODEL: TABLENET
We first consider the case with finitely many states and actions. this case, the messaging scheme can be represented entry in the table are simply the conditional probability p(mls).
table, where each row represents state and each column message. Each
In most applications, neural network usually takes possible input and outputs possible output y. In our model, however, the sender's ' neural network does not depend on any input, and the output is the table described above. This is because the table already represents the entire messaging scheme. But in order to fit in existing neural network structures, we use constant as the input and then let the network output the whole table.
Our sender network simple enough. The constant input goes into [S] fully connected layers with dimension [M]. Each fully connected layer is followed by softmax activation layer. Then stack the |S] outputs together to form the scheme table. We call our sender network the "TableNet".
3.2 RECEIVER'S S MODEL
The receiver's network is a function that maps scheme to the receiver's behavior (or strategy), When the sender sends message to the receiver, the receiver first computes the posterior according to Equation (T). and then chooses an action based on Equation (2). Note that in this process, the receiver only uses column of the scheme table p( p(s[m), S to make the decision. So the input of our receiver's s network is a column of the messaging scheme table which is chosen based on the received message, and the output is the action distribution when the receiver gets the corresponding column. We simply use multi-layer perceptron followed by softmax activation function as the receiver model, representing probability distribution
the action set A.
Recall that in our setting, we assume that the receiver's strategy 9x depends on the sender's mes- saging scheme and that if the empirical joint distribution (s, m) converges n, the receiver's
4

---

Under review as a conference paper at ICLR 2022
empirical action distribution converges to 9r- This means even if the sender cannot send the scheme a to the receiver, they can just stick to и and let the receiver learn, and the receiver can indeed con- verge to certain strategy eventually. Our receiver's model is built only to predict the asymptotic behavior of the receiver.
It is worth mentioning that this receiver model just how the sender models the receiver's behavior, does not necessarily reflect how the receiver actually behaves. Even if the receiver does not know the messaging scheme in advance, he can still learn the sender's messaging pattern through repeated interactions with the sender. Therefore, to simplify the receiver model, we directly use the messaging scheme as its input. In this paper. we only care about the asymptotic performance of our framework, which is not affected by such design.
Notice that our receiver model is an end-to-end model and does not involve the receiver's utility. This means our model can be easily extended to the setting where the receiver is not rational or even does not have certain utility function. We can still maximize the sender's utility as long as we can predict the receiver's response accurately.
3.3 LEARNING ALGORITHM
Our learning algorithm consists of two operations: optimizing the receiver's model and optimizing the sender's model. These two operations are executed alternatively just like the GAN framework. Our algorithm learns the receiver's model from scratch, and the training data is collected on the fly as we train our networks. To speed up the learning process, we also add a "random query" phase at the beginning that just uses random schemes. Our algorithm sketch is shown in Algorithm
Algorithm 1: Learning algorithm
Input: Threshold
Initialize D < 9 and generate random messaging schemes r(m|ss;;
foreach generated scheme do
Interact with the receiver with each scheme until convergence and then add receiver's response data D:
Train the receiver model ((a[m) with the collected data;
while True do
Use the trained receiver model to optimize the sender's messaging scheme;
Stick to the new messaging scheme for a period of time and interact with the receiver. Wait
until the receiver's 's learning algorithm converges and then compare the sender's true utility R₁ with the sender's predicted utility fú; Collect the receiver's converged response data and add to the data set D;
R R I then return;
else Use the new data set fine tune the receiver model;
Receiver's model. For the receiver's s model, our goal is to obtain predictions that are as close to the receiver's true strategy as possible. Denote by ((amm) the predicted strategy of the receiver model and ((((m) the true strategy of the receiver given that the sender's message is m. We use the cross entropy as the metric measure the distance between the predicted strategy ((amm) and the true strategy g(alm):
f)) (( ~
1
Σ DEDi
log ((a)m),
[Dml
where Dm is the set of samples of receiver actions when receiving message m. D. is populated with data points through repeated plays.
Note that care about the strategy rather than the action of the receiver. Therefore, unlike in multi- class classification problems, we cannot use the most frequent action as our prediction. Our choice of the cross entropy loss also has the property that it is minimized when q
Sender's model. Recall that in the bi-level optimization problem (6). the sender's strategy and the receiver's strategy influence each other. Therefore, when updating the parameters of the sender's
5

---

Under review as a conference paper at ICLR 2022
network, need to consider how the update would influence the receiver's behavior. Fortunately, such changes of the receiver's behavior can be captured by our receiver's network. To further sim- plify the optimization process, connect the receiver's network to the sender's 's network similar to the GAN framework. With such a network structure, the influence on the receiver's 's behavior can be automatically taken into consideration with the automatic differentiation feature of all major machine learning packages.
We simply set the loss function to be the negated expected utility for the sender.
&1 1= 1-
(8)
#ES mEM
where Tmm]s) and @(al ((alm) are the outputs of the TableNet and the receiver's model, respectively.
4 THEORETIC ANALYSIS
In this section, we analyze our framework in theory. We focus on the case where the receiver is rational and acts according to Equation (1) and 2 For simplicity, let [S] where each element represents the
and define the
partial scheme to be a [T (1mms1)),,))))))))) π (m)) .......(((ssss))) probability of sending a certain message m when the sender observes state s,, Note that partial scheme can be any vector in [0, 1] as each element is just a conditional probability. As we already mentioned in Section 3.2 such a partial scheme a suffices for the receiver to calculate his best response a(o). The receiver's behavior f can thus be seen as map from the partial scheme vector a to an action a, i.e., a classifier with n classes.
Given certain messaging scheme Tmm[s), there be at most n different partial schemes, since we only need n different messages. And each partial scheme a will be sampled with probability E, p(s) (m))s(mls)) This means we can only get most different data points, which is clear not enough for training such a classifier. However, we can simply obtain more training data by gener- ating different messaging schemes. Denote by II be the set of such messaging schemes, and Q the underlying distribution over all II[I] possible partial schemes that are induced by sampling mes- saging schemes uniformly from II. Let FF be the set of all possible receiver behavior functions f- Clearly, there most n²" II| different possible such functions, i.e., |Fl S = The objective of building the receiver model is to learn a function f E F to minimize the following prediction error:
err(f) Pro~Q [e (f(a), a(o)))
(9)
where {(.) is a loss function. Without loss of generality, we assume that the loss function normal- ized and always lies in the interval [0, 1).
When building the receiver model, the sender collects data while interacting with the receiver. Let D be the set of all collected data far, with N = [D] being the size of the data set that grows as time goes. The j-th data point in D is a tuple (a², ((o)), where a is a partial scheme sampled according to q and a(o²) is the receiver's best response to od, Define empirical risk to be:
emT
Σ ((((o),(()) (T,(()))ED
N
Denote by fERM the model that minimizes the above empirical risk.
Proposition 1. Given any set of messaging schemes II, a sampled data set 'D with [D] N, and any e, > O when N > zz [In 3 + n|[l] Inm], any training algorithm that minimizes the empirical risk can produce a classifier fERM that satisfies err(fERM - eer(f*) > e with probability 1 - 8, regardless of the prior distribution p(s).
The proof follows from standard PAC learning arguments. For completeness, we provide proof in Appendix A that is tailored for our setting.
rational receiver always chooses the best response based on the posterior belief (m) over the set [S]. Thus the messaging design problem is equivalent to the posterior design problem, subject to the constraint that the weighted posterior p( mm) ) (m) should sum up to the prior p(s), where ((m) = force the receiver induce the same posterior for every
Σ. T(m] ( ((s'). However, even if round and let the receiver leam the corresponding best response, it still could take exponentially many different designs for the sender learn perfect receiver model.

---

Under review as a conference paper at ICLR 2022
Proposition 2. Even if the sender can force the receiver form any posterior belief over possi- ble states, in the worst case, the sender still needs exponentially many episodes to learn a perfect receiver model, where episode is a set of consecutive rounds where the sender uses the same messaging scheme x.
Compare our setting with the Stackelberg game where leader first commits to a certain strategy and then a follower best responds. A posterior belief similar commitment in the Stackelberg game in that rational receiver follows exactly the same reasoning to choose a best response. The proof of Proposition 2 is omitted since it immediately follows from the result below.
Theorem (Peng et al. (. There exists a Stackelberg game instance, such that any algorithm takes at least (1)( queries to learn perfect follower model, where the leader makes query by sending a committed strategy to the follower and asks for the follower's best response.
5 EXPERIMENTS
We conduct experiments on multiple matrix games with different game sizes, ranging from [S] |A| 4 x 4 all the way to 32 x 32. In all games, the sender and the receiver's 's utility matrices are generated randomly between 1. We generated 25 games for each game size and report the averaged results.
5.1 A WARM-UP SETTING WITH KNOWN RECEIVER UTILITY
In this section, we consider a warm-up setting which is exactly the same as the standard persuasion model. In this case, the receiver's behavior is known and thus the parameters of the receiver model. The detailed network structure is deferred to Appendix C
hard code, instead of learn,
Due to the non-convex nature of the neural networks, different initialized parameters may lead to different locally optimal solutions. So for each game we run our algorithm 10 times and report the best result. We compare our framework results the optimal solution obtained by directly solving the linear program (5) using the Gurobi LP solver (Gurobi Optimization, LLC (202)).
0.9
0.75
reward
sender scenderrewar reward
0,70
0.8
0.65
0.7
framework
optimal our framework
0.55
optimal
12
16
20
24
28
32
100
200
300
400
500
game
iterations
(a) Performance in games with various sizes
(b) Convergence rate for 8 x 8 games
Figure 2: Comparison with the optimal solution.
Figure 2a compares the results of our framework and the optimal ones. The curve shows the average performance over the 25 random games for each size. The vertical line at each data point indicates the standard variance. shows that, as the game size grows, the sender's utility increases while the variance decreases. This means the sender can get higher utility more steadily in more complicated games. It also shows that the performance gap is larger for larger games. This is because in larger games, the scheme design space is much larger due higher dimensions. Thus it is more difficult to optimize the networks. But even for the 32 x 32 games, the performance gap is still reasonably small (about 4% on average).
Figure 2b shows the convergence our framework for the 25 games of size 8 x 8. It takes only about 200 rounds for our algorithm converge.
7

---

Under review as a conference paper at ICLR 2022
5.2 THE UNKNOWN RECEIVER SETTING
In this section, we report results on the setting where the receiver's utility is unknown to the sender. Thus the sender needs build a receiver model to predict the receiver's response. This process is computationally hard in general (see Proposition
5.2.1 PERSUADING RATIONAL RECEIVERS WITH EXPLICIT COMMITMENTS
In this setting, we relax the assumption that the sender knows the receiver's ' utility beforehand, but
let the sender reveal his messaging schemes before he sends messages to the receiver. In this setting, the receiver does not need to learn, and can simply choose the best action according to Equation (
after receiving a certain message. As stated in Section 3.3 when building the receiver model, the
sender first uses random commitments to collect data about the receiver's behaviors. We split the
data into training set and a test set. The results of our experiments are listed in the table below.
Game Size
Accuracy Training
Accuracy Set
Sender Expected Unility
Optimal Expected Utility
4
0,9843
0,9844
0.6874 (97.52%)
0.7049
8
0,9893
0.9890
0.6998 (92.37%)
0.7576
12 12
0,9864
0.9863
0.7476 (91. (91.60%)
0.8162
16 x 16
0,9861
0.9858
0.7678 (89.87%)
0.8543
20 20
0,9842
0,9840
0,7516 (86. 66%)
0.8671
24
0,9850
0,9847
0,7860 (88. 93%93%)
0.8838
28
0.9821
0.9817
0,8052 (89.96%)
0.8951
32 32
0.9835
0.9827
0.8033 (88. 13%)
0.9115
Table 1: Experiment results for the explicit commitment setting. The percentages in the brackets are obtained by comparing the last two columns.
The above table shows that the receiver model can predict the receiver's actual behavior very ac- curately. The Optimal Expected Utility is computed by solving LP (5). From Table we can see that our framework achieves comparable performance with the optimal solution, even though the comparison is not quite fair since the LP formulation requires the knowledge of the receiver's utility.
5.2.2 PERSUADING LEARNING RECEIVERS WITH IMPLICIT COMMITMENTS
In this section, we further relax the assumption that the receiver knows the sender's commitment before the game begins. Therefore, the receiver also needs to learn to optimize her behavior. Note that in this setting, the receiver can be regarded as having only bounded rationality. Because the receiver's learned response may deviate from the optimal one due to the non-stationary behaviors of some learning algorithms, or even worse, the set of strategies that can be expressed by a certain learning algorithm may not even contain the optimal one.
As mentioned in Section 3.2 in this setting, we simply have the sender stick to a certain scheme for a certain period of time and let the receiver learn his desired strategy. Even though the sender may change her scheme during the learning process, she can still implicitly reveal her new scheme the receiver she sticks the new scheme for an enough amount of time. And according to our assumption, if the empirical frequency of the tuple (s, m) converges to certain scheme, the receiver is always able to learn the correct response (which may not be rational though). Thus the sender can always obtain the receiver's asymptotic behavior for a certain scheme, and then use this new data point to improve the receiver model. To the best of our knowledge, there is no existing algorithm that solves the same problem in this paper. Therefore, we compare with the following baseline algorithms that solve similar problems:
Model the receiver (ModelR), the sender calculates his payoff assuming that the receiver follows the modeled policy and selects the message that maximizes his payoff.
Info-Q: the sender uses the Bayesian inference- hased messaging policy while the receiver uses Q-learning with optimistically initialized Q-values.
Info-Policy: the sender uses the Bayesian inference-based messaging policy while the re- ceiver uses REINFORCE update rule (Sutton et al. (1999))
The ModelR algorithm is a variant of the one proposed by Sen et al. 2003). Both Info-Q and Info-Policy are proposed by Bhatt& & Buro (021)) The experiment results are listed in Table
8

---

Under review as a conference paper at ICLR 2022
Game Size
Sender's Predicted Utility
Sender's Actual Utility
ModelR
Info-Q
Info-Policy
4
4
0.6649
0.6668
0.5878
0.5857
0.5902
8
8
0.6497
0.6589
0.5118
0.5114
0.5161
12
0.7173
0.6901
0.5254
0.5179
0.5014
16
16
0.6889
0.6801
0.4994
0.4953
0,4932
20
20
0.6740
0.6521
0.5080
0.4908
0,4932
24 24
0.6592
0.6391
0.4907
0.4851
0.5017
28
28
0.6745
0.6484
0.5264
0,5193
0.5020
32
32
0.6478
0.6289
0.5099
0.5145
0.5048
Table 2: Comparison between our framework and the baselines. The sender's predicted utility is the prediction given by framework, while the other columns obtained by actually using the learned scheme by respective algorithms.
The predictions our framework are quite accurate general. All baseline algorithms give sim- ilar performances, while our framework achieves significantly higher utilities. An important reason that all the baseline algorithms are too "greedy", i.c., always choose the message that can give the sender the highest utility, and thus form deterministic schemes. However, game-theoretic analyses show that randomization crucial to achieving better performances. In fact, only by being "lazy" and "temperate" the sender become a leader and take advantage of the information asymmetry.
5.3 PERSUADING WITH RESTRICTED SCHEME SIZE
Consider the setting where the receiver's utility is unknown and there is limit to the number of messages that the sender can send. This usually happens when there are physical restrictions on the messages there is bandwidth limit. We assume that the sender the receiver, where & < [.]] known that finding the optimal scheme in this setting NP-hard (Dughmi et al. 2016). The sender's messaging scheme can still be represented by table with size S x k. But there is no correspondence between the messages and the actions.
only send & messages to
0.80
Bender NoWArO
0.28
0.30
0.
optiral
cotmal
08
tramwock
75
24
23
20
120
200
300
400
200
massage number
Earations
(a) Performance vs. number of messages
(b) Convergence rate for 16 messages
Figure 3: Performance with restricted number of messages.
Figure 3a shows that as the number of messages increases, the marginal utility gain of adding a new message diminishes. Surprisingly, with only messages, our framework achieves more than 90% of the optimal utility, which is obtained by using 32 messages. Figure 3b shows that our algorithm converges in approximately 200 rounds, which similar Figure 2b
CONCLUSION
In this paper, we relaxed the assumption that the sender knows the receiver's utility function in the standard Bayesian persuasion model. We put forward framework that learns the optimal messaging scheme for the sender. Our framework also works when the sender cannot reveal her scheme to the receiver before the game begins, by letting the sender commit underlying scheme implicitly. Experiments show that our framework can achieve reasonably good performance compared to the optimal scheme in theory.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Ricardo Alonso and Odilon Câmara. Persuading voters. 3590-3605, 2016.
American Economic Review, 106(11) I):
Itai Arieli and Yakov Babichenko. Private bayesian persuasion. Journal of Economic Theory, 182: 185-217, 2019.
Yakov Babichenko, Inbal Talgam-Cohen, Haifeng Xu, and Konstantin Zabarnyi. Regret-minimizing bayesian persuasion. In Proceedings of the 22nd ACM Conference on Economicy and Com- putation, EC "21, pp. 128, New York, NY. USA, 2021. Association for Computing Machin- ery. ISBN 9781450385541. doi: 10.1145/346655663447733 URL /ttps::/doi.orgg/ 1145/3465456.1 3467574
Ashwinkumar Badanidiyuru, Kshipra Bhawalkar, and Haifeng Xu. Targeting and signaling in ad auctions. In Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algo- rithms, pp. 2545-2563, SIAM, 2018.
Dirk Bergemann, Benjamin Brooks, and Stephen Morris. The limits price discrimination. Amer- ican Economic Review, 105(3):211-77, 2015.
Varun Bhatt and Michael Buro. Inference-basee deterministic messaging for multi-agent communi- cation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11228- 11236, 2021.
Elizabeth Bondi, Hoon Oh, Haifeng Xu, Fei Fang, Bistra Dilkina, and Milind Tambe. To signal or not to signal: Exploiting uncertain real-time information in signaling games for security and sustainability. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 1369-1377, 2020.
Isabelle Brocas and Juan D Carrillo, Influence through ignorance. The RAND Journal of Economics, 38(4))931-94) 2007.
Modibo K Camara, Jason D Hartline, and Aleck Johnsen. Mechanisms for a no-regret agent: Beyond the common prior. 2020 IEEE s Annual Symposium on Foundations of Computer Science (FOCS), pp. 259-270. IEEE, 2020.
Matteo Castiglioni, Andrea Celli, Alberto Marchesi, and Nicola Gatti. Online Bayesian Persuasion. In Advances in Neural Information Processing Systems, volume 33, pp. sociates, Inc., 2020, URL http:://pppoeeeed.....e..i.eeeee html ba5451d3 a
Curran As-
Yu Cheng, Ho Yee Cheung, Shaddin Dughmi, Ehsan Emamjomeh-Zadel! Li Han, and Shang-Hua Teng. Mixture selection, mechanism design, and signaling. In 20/5 IEEE 56th Annual Symposium on Foundations of Computer Science, pp. 1426-1445. IEEE, 2015.
Shaddin Dughmi and Haifeng Xu. Algorithmic bayesian persuasion. SIAM Journal on Computing, (0):STOC16-6) 68, 2019.
Shaddin Dughmi, David Kempe, and Ruixin Qiang. Persuasion with limited communication. In Vincent Conitzer, Dirk Bergemann, and Yiling Chen (eds.), Proceedings of the 2016 ACM Con- ference on Economics and Computation, EC 16, Maastricht, The Netherlands, July 24-28, 2016, pp. 663 680. ACM, 2016. doi: 12.1145/9440166.29999 1145/2940716.0 2940781
URL https: //os:////oo.. or0/
Allan Gibbard. Manipulation of voting schemes: general result. Econometrica: journal of the Econometric Society, pp. 587-601, 1973.
Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2021. URL https: /www.
gurobi com
Emir Kamenica and Matthew Gentzkow. Bayesian persuasion. American Economic Review, 101 (6):2590-2615, 20
10

---

Under review as a conference paper at ICLR 2022
David Lingenbrink and Krishnamurthy Iyer. Optimal signaling mechanisms in unobservable queues. Operations research, 67(5):1397-1466, 2019.
Yishay Mansour, Aleksandrs Slivkins, Vasilis Syrgkanis, and Zhiwei Steven Wu. Bayesian explo- ration: Incentivizing exploration in bayesian games. arXiv preprint arXi:16622.05500 2016.
Roger B Myerson. Optimal auction design. Mathematics of operations research, 6(1):58-73, 1981.
Binghui Peng, Weiran Shen, Pingzhong Tang, and Song Zuo. Learning optimal strategies to commit to. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 2119-21491220 2019.
Zinovi Rabinovich, Albert Xin Jiang, Manish Jain, and Haifeng Xu. Information disclosure as a means security. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, pp. 645-653. Citeseer, 2015.
Sandip Sen, Stephane Airiau, and Rajatish Mukherjee. Towards pareto-optimal solution in general- sum games. In Proceedings of the second international joint conference on Autonomous agents and multiagent systems, pp. 153-160, 2003.
solutioon
Weiran Shen, Pingzhong Tang, and Yulong Zeng- A closed-form characterization of buyer sig- naling schemes in monopoly pricing. In Proceedings of the I 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 1531-1539, 2018.
Weiran Shen, Pingzhong Tang, and Yulong Zeng. Buyer signaling games in auctions. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp. 1591- 1599, 2019.
Weiran Shen, Weizhe Chen, Taoan Huang, Rohit Singh, and Fei Fang. Security games with strategic informants. In IJCAI, pp. 371-377, 2020.
When to follow the tip:
Richard Sutton, David A McAllester, Satinder P Singh, Yishay Mansour, al. Policy gradient methods for reinforcement learning with function approximation. In NIPs, volume 99, pp. 1057- 1063. Citeseer, 1999.
Haifeng Xu, Zinovi Rabinovich, Shaddin Dughmi, and Milind Tambe. Exploring information asym- metry in two-stage security games. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Haifeng Xu, Rupert Freeman, Vincent Conitzer, Shaddin Dughmi, and Milind Tambe. Signaling in bayesian stackelberg games. In AAMAS, pp. 150-158, 2016.
Haifeng Xu, Kai Wang, Phebe Vayanos, and Milind Tambe. Strategic coordination of human pa-
trollers and mobile sensors with signaling for security games. In Proceedings of the AAAI Con- ference Artificial Intelligence, volume 32, 2018
You Zu, Krishnamurthy Iyer, and Haifeng Xu. Learning to persuade on the fly: Robustness against ignorance. In Proceedings of the 22nd ACM Conference on Economics and Computation, EC 211, pp. 927-928, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450385541.
doi: 10.1145//46555634465555
URL https://dos .ow./org//// 11.11.....1.....
3465456. 3467593

---

Under review as a conference paper at ICLR 2022
APPENDIX
A AN EXAMPLE OF PERSUASION
Consider the example shown in Table 3 where salesman wants sell item to buyer by vincing the buyer that the item is of high quality in expectation. The two rows H and L denote that
Buyer
B
N
H
1,1
0,0
Salesman
L
1,0
0,1
Table 3:
Bayesian persuasion example
the item has high quality and low quality, respectively. And the two columns B and N denote the buyer's is action of buying the item and not buying it. Suppose that the buyer has prior belief about the item's quality with HH = & and p(L) = 5.4 The salesman has access the true quality item and is able to send message to the buyer revealing information about the quality of the item. After receiving the message, the buyer decides whether to buy the item or not.
Clearly, the salesman always prefers the buyer buy the item regardless of the actual quality. So a naïve strategy would to always send the message that the item has high quality. However, the salesman always sends the same message, the buyer will think the salesman's message doesn't 't convey any valuable information. So the buyer will choose actions based on the prior, which is not buying. Now assumed the salesman commits to the following messaging scheme: when the item has high quality, the salesman, with probability 1, sends the message indicating that the item is of high quality; When the item has low quality, the salesman tells the buyer that the item has high quality and low quality with probability 0.5 and 0.5 + e. Then after the buyer gets the message, they will update their belief via Bayes' theorem, and then choose an action according to their posterior belief. In this case, the buyer will still buy the item with a certain probability even when the quality low.
B PROOF OF PROPOSITION
Proof. Note that
erT(FRRM) - err(f*) =er((EERM) - (frR() eeTN (fERM) - efI)(f*)
+ erryy (ʃ*) err(J*).
Because JERM the model that minimizes the empirical risk. So the second line above is non-positive. Thus
enI(FERas) - err(f*) Sen(fERM) eIT (fERM) + cTT - err(f*) <(eTT(FERM) eIIN CIIN(fERM) + (eTIN(f") -(r(f)
<2 sup |err(f) - (f)l
(10)
fEF
For any function F, have According to Hoeffding's inequality.
err(f) - esr = Ep |err (f) esr
[er(f)) eIT (f)l 6
2e N4²
Pr
(11)
2.
Therefore, by the union bound
Pr sup feF
(err(f) eml((f) PPr E FF
- e 2 N=2
12

---

Under review as a conference paper at ICLR 2022
Combining the above with Equation equation Iquation[0yyellaa yields Pr (em((fERas)) - arm(f") e) S Pr
No²
22n" x/²
sup (emr(s) e/)) JEF
2 2
Therefore, when N
In 2 + II Inn], have
Pr Pr[(eT(fERM) err( er(())>
C DETAILED DESCRIPTION OF THE NETWORK STRUCTURE FOR THE WARM-UP SETTING
Recall that the input of the receiver's model is column of the output of the TableNet indexed by m: (m]s)) Vs E S. Then the receiver's model the input compute the posterior and the expected utility of taking each possible action:
r(a) Σ (2ss)))u2(ss.))) #ES
(12)
where r is |A -Aldimension vector, with each element being the expected utility of taking the corresponding action. As discussed in Section 3.2 we use a softmax activation for our receiver model, to represent distribution over his actions:
= softmax(AA.r) .r)
(13)
where is a hyper-parameter. Another reason for using such an activation that, in order to train the network, we need to replace the non-differentiable arg max operation that are used by a rational receiver when optimizing his utility. Thus À actually measures how well the softmax activation approximates the arg max operation. Note that such behavior model is also known the "quantal response" model in game theory, and is widely adopted to describe agents with bounded rationality. The structure of the receiver's model is shown in Figure
element-wise muttipled by the prior
"mutipty"
P(s)
Posterir P(a(m)
(a(4.))
Adtion Distribttion
P(m|s)
Sofhmax layer
P(a/m)
normalzation
elempent wise multiay
matrix multilly
Figure Network structure of the receiver model.
13


Reviews:
[{"review_id": "y0HTMOld10a", "reviewer": "Reviewer_oPdR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper starts with a good motivation to remedy practical shortcomings of the Bayesian persuasion model, but the results presented fall largely short of getting there. \n\nFirst fo all, there are many approaches in the literature on human behavior modeling and robust optimization, which could potentially be used to address the issues of imperfect rationality and incomplete knowledge. It is unclear how necessary it is to use a compelely new approach based on neural networks to solve the problem. If the authors' motivation is to address the limitations of the standard idealized model, then I feel that they should try these well-studied approaches first, or at least demonstrate that these appraches work no better than their machine learning approach. There is no such comparison in the paper. The experiments, in particular, did not use any such benchmarks and therefore the results a bit weak in demonstrating the strength of their contribution.\n\nThe learning algorithm also has several shortcomings on its own. The sender's model described in Section 3.1 is unclear to me. What are the input to and output of the neural network? In Algorithm 1, the authors also seem to have completely ignored the cost during the learning phase, the time it takes for the algorithm to converge at Line 3, and then in the while loop from Line 5-10. The algorithm also looks loose in some other steps. E.g., how is $\\hat{R}_1$ predicted? There is no rigorous analysis to the performance of the algorithm, in terms of convergence and the quality of solutions it produces. Section 4 did offer some analysis, but it falls back with the assumption of a completely rational receiver, making it questionable how meaningful the analysis is. ", "summary_of_the_paper": "The paper aims to remedy limitations of standard model-based approach of Bayesian persuasion by using machine learning methods. The authors presented an algorithm, which uses a neural network in the loop, provided theoretical analysis, and presented empirical results to prove the performance of their approach.", "summary_of_the_review": "A paper with a good motivation but falls short in fulfilling the proposed objective. The contribution is not significant.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "otd84Bjt_8H", "reviewer": "Reviewer_qN5Y", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It arguable whether a receiver, that needs to learn the commitment strategy of the sender and is aware of the sender's parallel adaptation, will miss this opportunity to manipulate the outcome. In fact, such manipulation is a serious concern for adaptive senders. \n\nConsider, e.g., some of the missed related work:\n\"Optimally deceiving a learning leader in Stackelberg games\" by Birmpas, Gan et al, JAIR, 72: 507-531, 2021 (a follow up paper to an older conference publication)\n\"Imitative follower deception in stackelberg games\" by Gan et al, ACM EC, 2019\n\"Manipulating a Learning Defender and Ways to Counteract\" by Gan et al, NeurIPS, 2019\n\nThe use of non-rational behaviour model is also not novel. Consider works by Azaria, Kraus et al, where a response model of a human actor (non-rational by nature) was learned. Milind Tambe's team has quite a few similar results as well. Again, missing from the related work. However, a further point of these previous results is that non-rational behaviour is quite often sub-optimal and open the possibility for further exploitation by the sender. In other words, if the receiver's model is in any way (near) stable, then the learned commitment strategy of the sender can potentially generate higher payoffs. Alas, this is not what Table-1 and Table-2 of the current paper show. Rather, once the non-rational assumption is engaged, the sender's payoff drops below its original visa vi rational-receiver levels. Authors do not notice this, nor discuss the issue. \n\nFinally, a better baseline to consider would be a regret-minimising approach:\n\"Playing Repeated Security Games with No Prior Knowledge\"\nby Xu, Tran-Thanh and Jennings\nAAMAS 2016\n", "summary_of_the_paper": "The paper suggests using neural networks to solve the nested optimisation problem of designing a disclosure scheme in persuasion schemes. This allows authors to remove the assumption of prior knowledge of receiver utilities by the sender, as well as rationality assumption of the receiver. Authors theoretically show that their approach correctly simulates the necessary optimisation patterns, and the paper concludes with a set of experiments that demonstrate that alternative solutions, that do not solve the optimisation directly, are less effective in achieving same effective persuasion levels. ", "summary_of_the_review": "The paper suggests a straight forward application of ML to solve persuasion optimisation with a reduced set of assumptions. Which is ok, but rather week, contribution. If the overall approach of using learning to supplant unknown elements was novel -- would be a publishable paper -- however, there's a lot of previous work where this has been used before, but authors missed it.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "ii8zX8Yf9cP", "reviewer": "Reviewer_yzkQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is easy to follow and the motivation for relaxing the stringent assumptions in a canonical economic model is fairly well-grounded.  \n\n1. The paper mentions that the proposed framework can be applied to relax several important assumptions made in Bayesian persuasion, e.g., sender doesn’t know receiver’s utility, receiver is not rational, receiver is not knowing sender’s signaling scheme, the signaling space is limited. However, to my best knowledge, simply relax either one of these assumptions is far from trivial.  I’d suggest the authors to focus on relaxing one of them, instead trying to relax them in a single framework.\n2. In the paper, the authors mention in several places that the receiver is not rational. There are two rationalities in the Bayesian persuasion: the receiver is expected utility maximizer, the receiver is Bayesian rational (i.e., update the posterior in Bayesian manner). So which rationality the authors refer to? Please clarify this, \n3. The linear program (5) is obtained by applying a revelation principle. When the receiver is not Bayesian rational, I’m not sure if the authors can still apply revelation principle and achieve a similar program as in (5), see the paper “Non-Bayesian Persuasion” by Clippel and Zhang (2019). \n4. Under program (5), “The receiver needs to learn π(m|s) in order to determine q(a|m), since the receiver cannot observe the scheme in advance.” I didn’t understand this sentence, why does the receiver need to determine q(a|m)? Isn’t this receiver’s own belief updating? \n5. “Note that this includes the no-regret learning algorithms in the contextual bandit setting. Because when the receiver is rational and applies a no-regret learning…” I’m not sure how this “no-regret learning algorithms in the contextual bandit setting” relates to the current setting? what is the “context” in the proposed  framework?\n6. 1st paragraph of sec 3, “...needs to learn to optimize his own objective”, why does the receiver need to learn to optimize his own objective? so receiver doesn’t know his objective function? \n7. Before program (6), the authors mention that “We omit the constraints described by Equation (3) as they also require the knowledge of the receiver’s utility function.”, But the program (6) still includes the receiver’s utility function? Again, if receiver is not rational, then it is not trivial if you can achieve the same constraint in (6), as the revelation principle may not apply here.\n8. in eq (8), it misses a minus sign? \n9. In sec 4, |S| = k, what is k here? If the authors already apply revelation principle, which may not accurate though, then one should have |S| = n. If authors simply just restrict |S| = k < n, I’m not sure if it is a without loss operation. \n10. The figures 2b and 3b seem to show that the proposed framework doesn’t converge to the optimal? \n\n\n==== Post-Rebuttal =====\\\nThank the authors for the feedback. Taking everything into consideration, I'd like to keep my rating as-is.", "summary_of_the_paper": "The paper studies how to relax certain stringent assumptions made in a Bayesian persuasion framework. In particular, the paper looks to relax assumptions where the sender knows the receiver’s utility function and the receiver’s behavior is completely rational. The authors also claim that the proposed framework works on scenario where the receiver does not know the sender’s messaging scheme in advance and needs to learn to optimize his (receiver’s) own objective. The paper also provides some experiments. ", "summary_of_the_review": "To summarize, the paper could be strengthened both from theoretical understanding of the assumption relaxations and experiments. Given the current stand, I think it not reaches the bar of ICLR. Hence, my rejection score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "H4bQ22wNC20", "reviewer": "Reviewer_MTb8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\nThe problem of Bayesian persuasion is of great theoretical and practical importance.  The method proposed doesn't require complete knowledge of the receiver's utility model, or even the assumption that there is one (i.e., the receiver might be irrational).  The method appears fairly simple (which I think is good), and achieves impressive performance in the experiments, which all suggest its practical potential.\n\n\n## Weaknesses\n\nI think the main contribution of the paper is empirical, so my main concern is whether the proposed method scales well.  As the experiments show, the performance of the method degrades as the game becomes larger.  I'd be more excited if the authors could provide more evidence that the method would likely perform well in some practical scenarios.\n\n\n## Detailed comments\n\nFormulation in Section 3: technically speaking, this still assumes the receiver is rational (i.e., utility-maximizing), right?  (The model in Section 3.2 can actually model (at least some) irrational behavior so that's fine.)\n\nFirst paragraph in Section 3.1: \"Each entry in the table are ...\"\n\nProposition 1: this doesn't make much practical sense to me for a number of reasons.  First, \\Pi (presumably being the set of all messaging schemes obtainable from training) probably should be infinite (or really large if we somehow restrain the training procedure) for the framework considered in this paper.  Moreover (and more importantly), the bound conceptually corresponds to a somewhat twisted version of the problem: the sender, for whatever reason, decides to use a meta-randomized scheme, where upon seeing each state, the sender first flips a coin to decide which scheme she uses, and then she commits to and announces that scheme, and flips another coin to decide the actual message that she sends.  We (the learner), observe the receiver's best response for a while, and find a (deterministic) model that fits our observations best.  Then the generalization bound says if our model fits our observations well, then it will probably fit the receiver's behavior well in expectation over all the randomness, including that involved in the sender's meta-randomized scheme.  Note that this says nothing about how well our model fits the receiver's actions in response to any particular messaging scheme.  In particular, our model could still be far off when the sender uses the \"optimal\" messaging scheme (this could mean the optimal scheme against the receiver's actual actions, or what we think is optimal given our model of the receiver).  If that's the case, then the bound doesn't say much about the (idealized) performance of the proposed approach.", "summary_of_the_paper": "The authors propose a differentiable, end-to-end formulation for the standard Bayesian persuasion problem.  The sender's and the receiver's strategies are modeled using neural networks, and the authors propose using a version of alternative training to simultaneously optimize the sender's strategy and predict the receiver's response.  The authors analyze some theoretical properties of the approach in an idealized setting, and conduct experiments on synthetic data to evaluate the performance of their approach.  The empirical results show that when the receiver observes the sender's entire strategy, their approach achieves about 90% of the optimal sender's utility for 32 states and 32 actions.  When the receiver only observes individual messages (and therefore needs to learn the best response), the proposed approach significantly outperform baseline methods (which are not specifically designed for this task).  The results also show diminishing marginal utility gain when the space of messages allowed becomes larger.", "summary_of_the_review": "Overall I think the paper proposes a novel differentiable approach to Bayesian persuasion, which is then validated experimentally.  The theoretical results are good to know, but I don't think they are particularly helpful for practical purposes; the experimental results overall support the claims, but may or may not scale well.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the Bayesian persuasion model in a more realistic setting where the sender does not know the receiver’s utility but can interact with the receiver repeatedly to learn the utility. The paper proposes a learning-based framework to optimize the sender’s strategy, then analyze the theoretical properties of the proposed framework, and perform extensive experiments. The reviewers acknowledged that the paper investigates an important problem of relaxing the practical shortcomings of the Bayesian persuasion model. However, the reviewers pointed out several weaknesses in the paper, and there was a clear consensus that the work is not ready for publication. The reviewers have provided very detailed and constructive feedback to the authors. We hope that the authors can incorporate this feedback when preparing future revisions of the paper.
Decision: Reject