Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper suggests using neural networks to solve the nested optimisation problem of designing a disclosure scheme in persuasion schemes. This allows authors to remove the assumption of prior knowledge of receiver utilities by the sender, as well as rationality assumption of the receiver. Authors theoretically show that their approach correctly simulates the necessary optimisation patterns, and the paper concludes with a set of experiments that demonstrate that alternative solutions, that do not solve the optimisation directly, are less effective in achieving same effective persuasion levels. 
main_review: It arguable whether a receiver, that needs to learn the commitment strategy of the sender and is aware of the sender's parallel adaptation, will miss this opportunity to manipulate the outcome. In fact, such manipulation is a serious concern for adaptive senders. 

Consider, e.g., some of the missed related work:
"Optimally deceiving a learning leader in Stackelberg games" by Birmpas, Gan et al, JAIR, 72: 507-531, 2021 (a follow up paper to an older conference publication)
"Imitative follower deception in stackelberg games" by Gan et al, ACM EC, 2019
"Manipulating a Learning Defender and Ways to Counteract" by Gan et al, NeurIPS, 2019

The use of non-rational behaviour model is also not novel. Consider works by Azaria, Kraus et al, where a response model of a human actor (non-rational by nature) was learned. Milind Tambe's team has quite a few similar results as well. Again, missing from the related work. However, a further point of these previous results is that non-rational behaviour is quite often sub-optimal and open the possibility for further exploitation by the sender. In other words, if the receiver's model is in any way (near) stable, then the learned commitment strategy of the sender can potentially generate higher payoffs. Alas, this is not what Table-1 and Table-2 of the current paper show. Rather, once the non-rational assumption is engaged, the sender's payoff drops below its original visa vi rational-receiver levels. Authors do not notice this, nor discuss the issue. 

Finally, a better baseline to consider would be a regret-minimising approach:
"Playing Repeated Security Games with No Prior Knowledge"
by Xu, Tran-Thanh and Jennings
AAMAS 2016

summary_of_the_review: The paper suggests a straight forward application of ML to solve persuasion optimisation with a reduced set of assumptions. Which is ok, but rather week, contribution. If the overall approach of using learning to supplant unknown elements was novel -- would be a publishable paper -- however, there's a lot of previous work where this has been used before, but authors missed it.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper aims to remedy limitations of standard model-based approach of Bayesian persuasion by using machine learning methods. The authors presented an algorithm, which uses a neural network in the loop, provided theoretical analysis, and presented empirical results to prove the performance of their approach.
main_review: The paper starts with a good motivation to remedy practical shortcomings of the Bayesian persuasion model, but the results presented fall largely short of getting there. 

First fo all, there are many approaches in the literature on human behavior modeling and robust optimization, which could potentially be used to address the issues of imperfect rationality and incomplete knowledge. It is unclear how necessary it is to use a compelely new approach based on neural networks to solve the problem. If the authors' motivation is to address the limitations of the standard idealized model, then I feel that they should try these well-studied approaches first, or at least demonstrate that these appraches work no better than their machine learning approach. There is no such comparison in the paper. The experiments, in particular, did not use any such benchmarks and therefore the results a bit weak in demonstrating the strength of their contribution.

The learning algorithm also has several shortcomings on its own. The sender's model described in Section 3.1 is unclear to me. What are the input to and output of the neural network? In Algorithm 1, the authors also seem to have completely ignored the cost during the learning phase, the time it takes for the algorithm to converge at Line 3, and then in the while loop from Line 5-10. The algorithm also looks loose in some other steps. E.g., how is $\hat{R}_1$ predicted? There is no rigorous analysis to the performance of the algorithm, in terms of convergence and the quality of solutions it produces. Section 4 did offer some analysis, but it falls back with the assumption of a completely rational receiver, making it questionable how meaningful the analysis is. 
summary_of_the_review: A paper with a good motivation but falls short in fulfilling the proposed objective. The contribution is not significant.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper studies how to relax certain stringent assumptions made in a Bayesian persuasion framework. In particular, the paper looks to relax assumptions where the sender knows the receiver’s utility function and the receiver’s behavior is completely rational. The authors also claim that the proposed framework works on scenario where the receiver does not know the sender’s messaging scheme in advance and needs to learn to optimize his (receiver’s) own objective. The paper also provides some experiments. 
main_review: The paper is easy to follow and the motivation for relaxing the stringent assumptions in a canonical economic model is fairly well-grounded.  

1. The paper mentions that the proposed framework can be applied to relax several important assumptions made in Bayesian persuasion, e.g., sender doesn’t know receiver’s utility, receiver is not rational, receiver is not knowing sender’s signaling scheme, the signaling space is limited. However, to my best knowledge, simply relax either one of these assumptions is far from trivial.  I’d suggest the authors to focus on relaxing one of them, instead trying to relax them in a single framework.
2. In the paper, the authors mention in several places that the receiver is not rational. There are two rationalities in the Bayesian persuasion: the receiver is expected utility maximizer, the receiver is Bayesian rational (i.e., update the posterior in Bayesian manner). So which rationality the authors refer to? Please clarify this, 
3. The linear program (5) is obtained by applying a revelation principle. When the receiver is not Bayesian rational, I’m not sure if the authors can still apply revelation principle and achieve a similar program as in (5), see the paper “Non-Bayesian Persuasion” by Clippel and Zhang (2019). 
4. Under program (5), “The receiver needs to learn π(m|s) in order to determine q(a|m), since the receiver cannot observe the scheme in advance.” I didn’t understand this sentence, why does the receiver need to determine q(a|m)? Isn’t this receiver’s own belief updating? 
5. “Note that this includes the no-regret learning algorithms in the contextual bandit setting. Because when the receiver is rational and applies a no-regret learning…” I’m not sure how this “no-regret learning algorithms in the contextual bandit setting” relates to the current setting? what is the “context” in the proposed  framework?
6. 1st paragraph of sec 3, “...needs to learn to optimize his own objective”, why does the receiver need to learn to optimize his own objective? so receiver doesn’t know his objective function? 
7. Before program (6), the authors mention that “We omit the constraints described by Equation (3) as they also require the knowledge of the receiver’s utility function.”, But the program (6) still includes the receiver’s utility function? Again, if receiver is not rational, then it is not trivial if you can achieve the same constraint in (6), as the revelation principle may not apply here.
8. in eq (8), it misses a minus sign? 
9. In sec 4, |S| = k, what is k here? If the authors already apply revelation principle, which may not accurate though, then one should have |S| = n. If authors simply just restrict |S| = k < n, I’m not sure if it is a without loss operation. 
10. The figures 2b and 3b seem to show that the proposed framework doesn’t converge to the optimal? 


==== Post-Rebuttal =====\
Thank the authors for the feedback. Taking everything into consideration, I'd like to keep my rating as-is.
summary_of_the_review: To summarize, the paper could be strengthened both from theoretical understanding of the assumption relaxations and experiments. Given the current stand, I think it not reaches the bar of ICLR. Hence, my rejection score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a differentiable, end-to-end formulation for the standard Bayesian persuasion problem.  The sender's and the receiver's strategies are modeled using neural networks, and the authors propose using a version of alternative training to simultaneously optimize the sender's strategy and predict the receiver's response.  The authors analyze some theoretical properties of the approach in an idealized setting, and conduct experiments on synthetic data to evaluate the performance of their approach.  The empirical results show that when the receiver observes the sender's entire strategy, their approach achieves about 90% of the optimal sender's utility for 32 states and 32 actions.  When the receiver only observes individual messages (and therefore needs to learn the best response), the proposed approach significantly outperform baseline methods (which are not specifically designed for this task).  The results also show diminishing marginal utility gain when the space of messages allowed becomes larger.
main_review: ## Strengths

The problem of Bayesian persuasion is of great theoretical and practical importance.  The method proposed doesn't require complete knowledge of the receiver's utility model, or even the assumption that there is one (i.e., the receiver might be irrational).  The method appears fairly simple (which I think is good), and achieves impressive performance in the experiments, which all suggest its practical potential.


## Weaknesses

I think the main contribution of the paper is empirical, so my main concern is whether the proposed method scales well.  As the experiments show, the performance of the method degrades as the game becomes larger.  I'd be more excited if the authors could provide more evidence that the method would likely perform well in some practical scenarios.


## Detailed comments

Formulation in Section 3: technically speaking, this still assumes the receiver is rational (i.e., utility-maximizing), right?  (The model in Section 3.2 can actually model (at least some) irrational behavior so that's fine.)

First paragraph in Section 3.1: "Each entry in the table are ..."

Proposition 1: this doesn't make much practical sense to me for a number of reasons.  First, \Pi (presumably being the set of all messaging schemes obtainable from training) probably should be infinite (or really large if we somehow restrain the training procedure) for the framework considered in this paper.  Moreover (and more importantly), the bound conceptually corresponds to a somewhat twisted version of the problem: the sender, for whatever reason, decides to use a meta-randomized scheme, where upon seeing each state, the sender first flips a coin to decide which scheme she uses, and then she commits to and announces that scheme, and flips another coin to decide the actual message that she sends.  We (the learner), observe the receiver's best response for a while, and find a (deterministic) model that fits our observations best.  Then the generalization bound says if our model fits our observations well, then it will probably fit the receiver's behavior well in expectation over all the randomness, including that involved in the sender's meta-randomized scheme.  Note that this says nothing about how well our model fits the receiver's actions in response to any particular messaging scheme.  In particular, our model could still be far off when the sender uses the "optimal" messaging scheme (this could mean the optimal scheme against the receiver's actual actions, or what we think is optimal given our model of the receiver).  If that's the case, then the bound doesn't say much about the (idealized) performance of the proposed approach.
summary_of_the_review: Overall I think the paper proposes a novel differentiable approach to Bayesian persuasion, which is then validated experimentally.  The theoretical results are good to know, but I don't think they are particularly helpful for practical purposes; the experimental results overall support the claims, but may or may not scale well.
review_title: null

Meta-review: The paper studies the Bayesian persuasion model in a more realistic setting where the sender does not know the receiver’s utility but can interact with the receiver repeatedly to learn the utility. The paper proposes a learning-based framework to optimize the sender’s strategy, then analyze the theoretical properties of the proposed framework, and perform extensive experiments. The reviewers acknowledged that the paper investigates an important problem of relaxing the practical shortcomings of the Bayesian persuasion model. However, the reviewers pointed out several weaknesses in the paper, and there was a clear consensus that the work is not ready for publication. The reviewers have provided very detailed and constructive feedback to the authors. We hope that the authors can incorporate this feedback when preparing future revisions of the paper.
Decision: Reject