SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Martino Bernasconi*
Bocconi University martino.bernasconi@unibocconi.it
Matteo Castiglioni
Politecnico di Milano matteo.castiglioni@polimi.it
Alberto Marchesi
Politecnico di Milano alberto.marchesi@polimi.it
Mirco Mutti*
Technion mirco.m@technion.ac.il
Bayesian persuasion studies the problem faced by an informed sender who strategically discloses information to influence the behavior of an uninformed receiver. Recently, a growing attention has been devoted to settings where the sender and the receiver interact sequentially, in which the receiver's decision-making problem is usually modeled as a Markov decision process (MDP). However, previous works focused on computing optimal information-revelation policies (a.k.a. signaling schemes) under the restrictive assumption that the receiver acts myopically, selecting actions to maximize the one-step reward and disregarding future rewards. This is justified by the fact that, when the receiver is farsighted and thus considers future rewards, finding an optimal Markovian signaling scheme is NP-hard. In this paper, we show that Markovian signaling schemes do not constitute the "right" class of policies. Indeed, differently from most of the MDPs settings, we prove that Markovian signaling schemes are not optimal, and general history-dependent signaling schemes should be considered. Moreover, we also show that history-dependent signaling schemes circumvent the negative complexity results affecting Markovian signaling schemes. Formally, we design an algorithm that computes an optimal and  $\epsilon$ -persuasive history-dependent signaling scheme in time polynomial in  $1/\epsilon$  and in the instance size. The crucial challenge is that general history-dependent signaling schemes cannot be represented in polynomial space. Nevertheless, we introduce a convenient subclass of history-dependent signaling schemes, called promise-form, which are as powerful as general history-dependent ones and efficiently representable. Intuitively, promise-form signaling schemes compactly encode histories in the form of honest promises on future receiver's rewards.
Bayesian persuasion [Kamenica and Gentzkow, 2011] is the problem faced by an informed sender who wants to influence the behavior of an uninformed, self-interested receiver through the provision of payoff relevant information. Bayesian persuasion captures many fundamental problems arising in real-world applications, e.g., online advertising [Bro Miltersen and Sheffet, 2012], voting [Alonso and Camara, 2016, Castiglioni et al., 2020a, Castiglioni and Gatti, 2021], traffic routing [Bhaskar et al., 2016, Castiglioni et al., 2021a], recommendation systems [Mansour et al., 2016], security [Rabinovich et al., 2015, Xu et al., 2016], marketing [Babichenko and Barman, 2017, Candogan, 2019], medical research [Kolotilin, 2015], and financial regulation [Goldstein and Leitner, 2018].
*Work done while the author was at Politecnico di Milano.
Most of the previous works study the classical, one-shot version of the Bayesian persuasion problem. However, in many application scenarios it is natural to assume that the sender and the receiver interact multiple times in a sequential manner. In spite of this, only a few very recent works addressed sequential versions of the Bayesian persuasion problem [Wu et al., 2022, Bernasconi et al., 2022, Gan et al., 2022a,b]. In particular, Wu et al. [2022] and Gan et al. [2022a,b] study settings where the sender and the receiver interact sequentially in a Markov decision process (MDP).
In Bayesian persuasion problems in MDPs, at each step of the interaction both the sender and the receiver know the current state of the MDP, and the former has also access to some private observation drawn according to a commonly-known, state-dependent distribution. The sender commits beforehand to an information-revelation policy, which is implemented by means of a signaling scheme that sends randomized action recommendations to the receiver, conditioned on the current (public) state and the sender's private observation. Specifically, the sender commits to a persuasive signaling scheme, meaning that the receiver is always incentivized to follow recommendations. At the end of each step, the next state of the MDP and the agents' rewards are determined as a function of the current state, the action actually played by the receiver, and the sender's private observation in the current step.
Wu et al. [2022] and Gan et al. [2022a,b] provide algorithms that compute an optimal (i.e., reward-maximizing) persuasive signaling scheme under the restrictive assumption that the receiver acts myopically, selecting actions to maximize the one-step reward and disregarding future ones. This is justified by the fact that, when the receiver is farsighted and thus considers future rewards, finding an optimal Markovian signaling scheme is NP-hard to approximate, as shown by Gan et al. [2022a] for infinite-horizon MDPs. An analogous result also holds in finite-horizon MDPs for non-stationary Markovian signaling schemes, as we prove in this work as a preliminary result.
In this paper, we show that Wu et al. [2022] and Gan et al. [2022a,b] failed to provide positive results with farsighted receivers since Markovian signaling schemes do not constitute the "right" class of policies to consider. This is in stark contrast with most of the MDPs settings in which Markovian policies are optimal. Indeed, we prove that Markovian signaling schemes are not optimal, and general history-dependent signaling schemes should be considered. As a result, we focus on the problem of computing an optimal persuasive history-dependent signaling scheme. Surprisingly, we show that taking history into account allows to circumvent the negative result affecting Markovian signaling schemes. We do that by providing an approximation scheme that finds an optimal  $\epsilon$ -persuasive (i.e., one approximately incentivizing the receiver to follow action recommendations) history-dependent signaling scheme in time polynomial in  $1 / \epsilon$  and the size of the problem instance.
The crucial challenge in designing our approximation scheme is that general, history-dependent signaling schemes cannot be represented in polynomial space. Our algorithm overcomes such an issue by using a convenient subclass of history-dependent signaling schemes, which we call *promise-form* signaling schemes. The core idea of such signaling schemes is to compactly encode all the relevant information contained in an history into a *promise* on future receiver's rewards. At each step of the process, a *promise-form* signaling scheme does not only determine an action recommendation for the receiver, but it also makes a promise to them. First, we prove that promise-form signaling schemes are as powerful as general history-dependent ones. Then, we show how an optimal  $\epsilon$ -persuasive *promise-form* signaling scheme can be computed in polynomial time by means of a recursive procedure. To do that, we rely on a crucial result showing that, for signaling schemes that honestly keep promises made to the receiver, persuasiveness constraints can be expressed as conditions defined locally at each step of the MDP, since the receiver only cares about sender's promises on their future rewards.[2]
In this work, we study Bayesian persuasion problems where a farsighted receiver takes actions in a time-inhomogeneous finite-horizon MDP [Puterman, 2014]. Formally, a problem instance is a tuple
$$
\left(\mathcal {S}, \mathcal {A}, \mathcal {H}, \Theta , \{r _ {h} ^ {\mathrm {S}} \} _ {h \in \mathcal {H}}, \{r _ {h} ^ {\mathrm {R}} \} _ {h \in \mathcal {H}}, \{p _ {h} \} _ {h \in \mathcal {H}}, \{\mu_ {h} \} _ {h \in \mathcal {H}}, \beta\right), \text {w h e r e :}
$$
$S$  is a finite set of states,  $\mathcal{A}$  is a finite set of receiver's actions available in each state,  $\mathcal{H} \coloneqq [1, \dots, H]$  is a set of time steps with  $H$  being the time horizon,  $\Theta$  is a finite set of sender's private observations,  $r_h^{\mathrm{S}}, r_h^{\mathrm{R}}: S \times \mathcal{A} \times \Theta \to [0, 1]$  are reward functions for the sender and the receiver, respectively,
2The complete proofs of all the results in the paper can be found in Appendices C, D, and E.
$p_h: \mathcal{S} \times \mathcal{A} \times \Theta \to \Delta(\mathcal{S})$  is a transition function,  $\mu_h: \mathcal{S} \to \Delta(\Theta)$  is a function defining probabilities of sender's private observations at each state, while  $\beta \in \Delta(\mathcal{S})$  is the initial state distribution.
We consider the most general setting in which the sender commits to a non-stationary and non-Markovian signaling scheme (henceforth called history-dependent signaling scheme for short). For every step  $h \in \mathcal{H}$  and state  $s_h \in S$  reached at that step, a history-dependent signaling scheme defines a randomized mapping from sender's private observations to action recommendations for the receiver, based on the whole history of states and receiver's actions observed up to step  $h$ . Formally, in the following we let  $\mathcal{T}_h$  be the set of all the possible histories up to step  $h$ , which is defined as
$$
\mathcal {T} _ {h} := \left\{\tau \mid \tau = \left(s _ {1}, a _ {1}, \dots , s _ {h - 1}, a _ {h - 1}, s _ {h}\right): s _ {i} \in \mathcal {S}, a _ {i} \in \mathcal {A} \right\},
$$
while we let  $\mathcal{T} := \mathcal{T}_1 \cup \ldots \cup \mathcal{T}_H$  be the set of all the possible histories (of any length). Then, a history-dependent signaling scheme is defined as a set  $\phi := \{\phi_{\tau}\}_{\tau \in \mathcal{T}}$  of functions  $\phi_{\tau}: \Theta \to \Delta(\mathcal{A})$  which define a mapping from sender's private observations to probability distributions over action recommendations for every possible history.
The interaction between the sender and the receiver goes as follows (Algorithm 1). (i) The sender publicly commits to a history-dependent signaling scheme  $\phi \coloneqq \{\phi_{\tau}\}_{\tau \in \mathcal{T}}$ . (ii) An initial state  $s_1 \sim \beta$  is drawn. (iii) At each step  $h \in \mathcal{H}$ , both agents observe the current state  $s_h \in S$  and the sender also gets a private observation  $\theta_h \in \Theta$  drawn according to  $\mu_h(s_h)$ , with the function  $\mu_h$  being known to both the sender and the receiver. (iv) The sender communicates to the receiver an action recommendation  $a_h \in \mathcal{A}$  sampled according to  $\phi_{\tau_h}(\theta_h)$ , where  $\tau_h \in T_h$  is the history up to step  $h$ . (v) The receiver plays an action  $\hat{a}_h \in \mathcal{A}$ , possibly different from  $a_h$ . (vi) The sender and the receiver get rewards  $r_h^{\mathbb{R}}(s_h, a_h, \theta_h)$  and  $r_h^{\mathbb{S}}(s_h, a_h, \theta_h)$ , respectively. (vii) If  $h = H$ , the interaction ends, otherwise
Algorithm 1 Sender-receiver interaction
1: Sender publicly commits to  $\phi \coloneqq \{\phi_{\tau}\}_{\tau \in \mathcal{T}}$
2:  $s_1 \sim \beta, \tau_1 \leftarrow (s_1)$ , deviated  $\leftarrow$  False
3: for each step  $h = 1, \ldots, H$  do
4: Sender observes  $\theta_h \sim \mu_h(s_h)$
5: if deviated is False then
6: Sender recommends  $a_{h}\sim \phi_{\tau_{h}}(\theta_{h})$
7: Receiver plays  $\widehat{a}_h\in \mathcal{A}$
8: else
9: Receiver plays  $\widehat{a}_h\in \mathcal{A}$
10: Sender collects reward  $r_h^{\mathrm{S}}(s_h,\widehat{a}_h,\theta_h)$
11: Receiver collects reward  $r_h^{\mathsf{R}}(s_h,\widehat{a}_h,\theta_h)$
12: Next state:  $s_{h + 1}\sim p_h(s_h,\widehat{a}_h,\theta_h)$
13: Update history:  $\tau_{h + 1}\gets \tau_h\oplus (\widehat{a}_h,s_{h + 1})$
14: if  $a_{h}\neq \widehat{a}_{h}$  then
15: deviated  $\leftarrow$  True
the next state  $s_{h + 1} \sim p_h(s_h, \widehat{a}_h, \theta_h)$  is drawn and the interaction continues to step  $h + 1$  starting from the third point. As customary in the literature (see, e.g., [Bernasconi et al., 2022]), we assume that, if the receiver does not follow recommendations at some step  $h \in \mathcal{H}$  by playing an action  $\widehat{a}_h \neq a_h$ , then the sender stops issuing future recommendations to the receiver.
For ease of presentation, we introduce the sender's value function  $V_{h}^{\mathbf{S},\phi}:\mathcal{T}_{h}\to \mathbb{R}$  to encode sender's expected rewards by using a history-dependent signaling scheme  $\phi \coloneqq \{\phi_{\tau}\}_{\tau \in \mathcal{T}}$  from step  $h\in \mathcal{H}$  onwards, assuming the receiver always follows recommendations. Given a history  $\tau = (s_1,a_1,\dots ,s_{h - 1},a_{h - 1},s_h)\in \mathcal{T}_h$  up to step  $h$ , such a value function is recursively defined as:
$$
V _ {h} ^ {\mathbf {S}, \phi} (\tau) = \sum_ {\theta \in \Theta} \sum_ {a \in \mathcal {A}} \mu_ {h} (\theta | s _ {h}) \phi_ {\tau} (a | \theta) \left(r _ {h} ^ {\mathbf {S}} (a, s _ {h}, \theta) + \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {h} (s ^ {\prime} | s _ {h}, a, \theta) V _ {h + 1} ^ {\mathbf {S}, \phi} (\tau \oplus (a, s ^ {\prime}))\right).
$$
Similarly, we introduce the receiver's action-value function  $V_h^{\mathsf{R},\phi} : \mathcal{A} \times \mathcal{T}_h \to \mathbb{R}$  to encode the receiver's expected rewards by following sender's action recommendations from step  $h \in \mathcal{H}$  onwards. Formally, given a history  $\tau = (s_1, a_1, \ldots, s_{h-1}, a_{h-1}, s_h) \in \mathcal{T}_h$  up to step  $h$ , the receiver's expected reward by following the recommendation to play  $a \in \mathcal{A}$  is recursively defined as follows:
$$
V _ {h} ^ {\mathsf {R}, \phi} (a, \tau) = \sum_ {\theta \in \Theta} \mu_ {h} (\theta | s _ {h}) \phi_ {\tau} (a | \theta) \left(r _ {h} ^ {\mathsf {R}} (a, s _ {h}, \theta) + \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {h} (s ^ {\prime} | s _ {h}, a, \theta) V _ {h + 1} ^ {\mathsf {R}, \phi} (\tau \oplus (a, s ^ {\prime}))\right),
$$
<sup>3</sup>In this paper, we let  $\Delta(X)$  be the set of all the probability distributions over a finite set  $X$ , with  $d(x)$  denoting the probability assigned to  $x \in X$  by a distribution  $d \in \Delta(X)$ . Moreover, given a function  $f: X \to \Delta(Y)$  with  $X, Y$  any two finite sets, for every  $x \in X$  we denote by  $f(y|x)$  the probability that  $f(x)$  assigns to  $y \in Y$ .
In this work, we use direct signaling schemes which send signals in the form of action recommendations for the receiver. This is w.l.o.g. by well-known revelation principle arguments [Kamenica and Gentzkow, 2011].
<sup>5</sup>By a simple revelation-principle-style argument, we can focus w.l.o.g. on signaling schemes which depend on histories that do not include the sequence of private observations observed by the sender.
where  $V_h^{\mathsf{R},\phi}:\mathcal{T}_h\to \mathbb{R}$  is such that  $V_{h}^{\mathsf{R},\phi}(\tau) = \sum_{a\in \mathcal{A}}V_{h}^{\mathsf{R},\phi}(a,\tau)$  for every  $h\in \mathcal{H}$  and  $\tau \in \mathcal{T}_h$
Finally, we introduce an additional receiver's value function, denoted by  $\widehat{V}_h^{\mathsf{R}}: \mathcal{S} \to \mathbb{R}$ , to encode receiver's expected rewards from step  $h \in \mathcal{H}$  onwards after having deviated from recommendations. Formally, for every state  $s \in \mathcal{S}$ , such a value function is recursively defined as follows:
$$
\widehat {V} _ {h} ^ {\mathsf {R}} (s) = \max _ {a \in \mathcal {A}} \sum_ {\theta \in \Theta} \mu_ {h} (\theta | s) \left(r _ {h} ^ {\mathsf {R}} (a, s, \theta) + \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {h} (s ^ {\prime} | s, a, \theta) \widehat {V} _ {h + 1} ^ {\mathsf {R}} (s ^ {\prime})\right),
$$
where the maximum operator encodes the fact that the receiver plays so as to maximize future rewards without knowledge of realized sender's private observations after having deviated.6
By the revelation principle [Kamenica and Gentzkow, 2011], it is well known that in order to find an optimal signaling scheme it is possible to focus w.l.o.g. on (direct) history-dependent signaling schemes under which the receiver is always incentivized to follow sender's action recommendations. These are called persuasive signaling scheme, and they are formally defined as follows:
Definition 1 (ε-persuasiveness). Let  $\epsilon \geq 0$ . A history-dependent signaling scheme  $\phi \coloneqq \{\phi_{\tau}\}_{\tau \in \mathcal{T}}$  is said to be  $\epsilon$ -persuasive if, for every step  $h \in \mathcal{H}$ , history  $\tau = (s_1, a_1, \ldots, s_{h-1}, a_{h-1}, s_h) \in \mathcal{T}_h$  up to step  $h$ , and pair of actions  $a, a' \in \mathcal{A}$ , the following holds:
$$
V _ {h} ^ {\mathsf {R}, \phi} (a, \tau) \geq \sum_ {\theta \in \Theta} \mu_ {h} (\theta | s _ {h}) \phi_ {\tau} (a | \theta) \left(r _ {h} ^ {\mathsf {R}} (s _ {h}, a ^ {\prime}, \theta) + \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {h} (s ^ {\prime} | s _ {h}, a ^ {\prime}, \theta) \widehat {V} _ {h + 1} ^ {\mathsf {R}} (s ^ {\prime}) - \epsilon\right).
$$
Moreover, we say that the signaling scheme is persuasive if the conditions above hold for  $\epsilon = 0$ . We denote the set of all the persuasive signaling schemes by  $\Phi$ .
Whenever the sender commits to an  $\epsilon$ -persuasive history-dependent signaling scheme, we assume that the receiver always follows the recommendation issued by the sender. Notably, the recommended action is guaranteed to provide the receiver expected rewards that are at worst  $\epsilon$  less than those attained by the best possible action.
In conclusion, the goal is to find an optimal signaling scheme for the sender, which is one achieving a sender's expected reward (from step one)  $V^{\mathbb{S},\phi}$  greater than or equal to OPT, defined as follows:
$$
\mathsf {O P T} := \max  _ {\phi \in \Phi} V ^ {\mathsf {S}, \phi}, \quad \text {w h e r e w e l e t} V ^ {\mathsf {S}, \phi} := \sum_ {s \in \mathcal {S}} \beta (s) V _ {1} ^ {\mathsf {S}, \phi} ((s)).
$$
3 History-dependent signaling schemes are necessary
Previous works studying Bayesian persuasion in MDPs [Wu et al., 2022, Gan et al., 2022a] focus on Markovian signaling schemes, in which the action recommendation at step  $h$  only depends on the current state  $s_h$  and private observation  $\theta_h$ . Indeed, considering this class of signaling schemes is sufficient to optimize the utility of a sender facing a myopic receiver. Here, we show that this is not the case when the receiver is farsighted. In particular, we show that non-stationary Markovian signaling scheme are suboptimal. To do so, we show that there exists an MDP (see Appendix A) in which the optimal persuasive signaling scheme is history-dependent. Intuitively, an history-dependent signaling scheme can adjust action recommendations depending on the choices available to the receiver in previous steps. Thus, if the receiver had profitable opportunities in the past, the sender must provide a larger expected reward to the receiver in order to be persuasive. Otherwise, the sender can aggressively maximize their expected rewards irrespective of receiver's ones. Formally:
Theorem 1. There exist instances in which a persuasive history-dependent signaling scheme guarantees sender's expected rewards strictly greater than that obtained by an optimal persuasive non-stationary Markovian signaling scheme.
Notice that, in all the steps reached after having deviated from recommendations, the receiver does not get any clue about the sender's private information, and, thus, their expected reward only depends on the current state (not on the history). In other words, after deviating the receiver is playing a new MDP in which, by taking expectations with respect to  $\mu_h$ , rewards and transition probabilities are defined as  $\tilde{r}_h(s,a) \coloneqq \sum_{\theta \in \Theta} \mu_h(\theta | s) r_h^{\mathsf{R}}(s,a,\theta)$  and  $\tilde{p}_h(s'|s,a) \coloneqq \sum_{\theta \in \Theta} \mu_h(\theta | s) p_h(s'|s,a,\theta)$ , respectively, for all  $h \in \mathcal{H}$ ,  $s \in S$ ,  $a \in \mathcal{A}$ , and  $s' \in S$ .
<sup>7</sup>We defer to Appendix A the complete description of the instance, together with calculations of the optimal signaling schemes (for each class) and their corresponding value functions.
Moreover, we show that optimal non-stationary Markovian signaling schemes are NP-hard to approximate in polynomial time, even when the persuasiveness requirement is relaxed. This further motivates the use of history-dependent signaling schemes when addressing Bayesian persuasion problems in finite-horizon MDPs with a farsighted receiver. Formally, we prove the following:
Theorem 2. There exist two constants  $\alpha < 1$  and  $\epsilon > 0$  such that computing an  $\epsilon$ -persuasive non-stationary Markovian signaling that provides the sender with at least a fraction  $\alpha$  of the optimal sender's expected reward OPT is NP-hard.
4 A sufficient subclass of efficiently-representable signaling schemes
Working with history-dependent signaling schemes begets unavoidable computational issues. These are due to the fact that explicitly representing such signaling schemes requires a number of bits growing exponentially in the size of the problem instance, since the number of possible histories is exponential in the time horizon  $H$ . In this section, we show how to circumvent such an issue by introducing a convenient subclass of signaling schemes—called *promise-form* signaling schemes—which are efficiently representable while being as good as history-dependent ones. In particular, our main result in this section (Theorem 3) shows that there always exists a promise-form signaling scheme which results in sender's expected rewards equal to its optimal value OPT.
4.1 Promise-form signaling schemes
A promise-form signaling scheme is defined by a set  $\sigma := \{(I_h, \varphi_h, g_h)\}_{h \in \mathcal{H}}$  of triplets, where:
-  $I_h: \mathcal{S} \to 2^{[0, H]}$  is a function defining, for every state  $s \in \mathcal{S}$ , a finite set  $I_h(s) \subseteq [0, H]$  of promises for step  $h \in \mathcal{H}$ . We add the additional requirement that  $0 \in I_1(s)$  for all  $s \in \mathcal{S}$ , and, for ease of notation, we set  $I_{H+1}(s) := \{0\}$  for every  $s \in \mathcal{S}$  and  $\mathcal{I} := \bigcup_{h \in \mathcal{H}} \bigcup_{s \in \mathcal{S}} I_h(s)$ .
-  $\varphi_h: \mathcal{S} \times \mathcal{I} \times \Theta \to \Delta(\mathcal{A})$  is an action-recommendation strategy to be employed at step  $h \in \mathcal{H}$ , where  $\varphi_h(a|s, \iota, \theta)$  is the probability of recommending action  $a \in \mathcal{A}$  in state  $s \in \mathcal{S}$  when the promise is  $\iota \in I_h(s)$  and the sender's private observation is  $\theta \in \Theta$ .
-  $g_{h} : \mathcal{S} \times \mathcal{A} \times \mathcal{I} \times \mathcal{S} \to \mathcal{I}$  is a promise function for step  $h \in \mathcal{H}$  such that, whenever  $h \leq H$  and  $\iota \in I_{h}(s)$ ,  $g_{h}(s, a, \iota, s') \in I_{h+1}(s')$  represents the promise for step  $h+1$  if the next state is  $s' \in \mathcal{S}$  and, at the current step  $h$ , action  $a \in \mathcal{A}$  is recommended in state  $s \in \mathcal{S}$ .<sup>8</sup>
Intuitively, the rationale behind promise-form signaling schemes is that, when reaching a state  $s_h \in S$  at step  $h \in \mathcal{H}$ , the sender "promises" a value  $\iota \in I_h(s_h)$  to the receiver, representing a lower bound on future rewards obtained by following recommendations. Moreover, sender's action recommendations only depend on the current state  $s_h \in S$ , the sender's private observation  $\theta_h \in \Theta$ , and the current promise  $\iota \in I_h(s_h)$ , through the distribution  $\varphi_h(s_h, \iota_h, \theta_h)$ . Notice that it is always possible to infer the current promise by looking at the history of past states and action recommendations, by "composing" the functions  $g_{h'}$  for the steps  $h' < h$ . This crucially avoids having to specify an explicit dependency on the full history of past states and action recommendations.
Notice that a promise-form signaling scheme as defined above does not automatically guarantee that the sender honestly keeps their promises. Indeed, in order to ensure that this is the case, we need to enforce additional constraints on the components of the signaling scheme, as we show in Section 4.3.