Title: Persuading Farsighted Receivers in MDPs: the Power of Honesty
Venue: NeurIPS 2023
Accepted: yes
Keywords: Bayesian Persuasion

MPD

information design

signaling
Abstract: Bayesian persuasion studies the problem faced by an informed sender who strategically discloses information to influence the behavior of an uninformed receiver. Recently, a growing attention has been devoted to settings where the sender and the receiver interact sequentially, in which the receiver's decision-making problem is usually modeled as a Markov decision process (MDP). However, the literature focuses on computing optimal information-revelation policies (a.k.a. signaling schemes) under the restrictive assumption that the receiver acts myopically, selecting actions to maximize the one-step utility and disregarding future rewards. This is justified by the fact that, when the receiver is farsighted and thus considers future rewards, finding an optimal Markovian signaling scheme is NP-hard. In this paper, we show that Markovian signaling schemes do not constitute the "right" class of policies. Indeed, differently from most of the MDPs settings, we show that Markovian signaling schemes are not optimal, and general history-dependent signaling schemes should be considered. Moreover, we also show that history-dependent signaling schemes circumvent the negative complexity results affecting Markovian signaling schemes. Formally, we design an algorithm that computes an optimal and $\epsilon$-persuasive history-dependent signaling scheme in time polynomial in ${1}/{\epsilon}$ and in the instance size. The crucial challenge is that general history-dependent signaling schemes cannot be represented in polynomial space. Nevertheless, we introduce a convenient subclass of history-dependent signaling schemes, called promise-form, which are as powerful as general history-dependent ones and efficiently representable. Intuitively, promise-form signaling schemes compactly encode histories in the form of honest promises on future receiver's rewards.

Persuading Farsighted Receivers in MDPs: the Power of Honesty

Abstract

Bayesian persuasion studies the problem faced by an informed sender who strategically discloses information to influence the behavior of an uninformed receiver. Recently, a growing attention has been devoted to settings where the sender and the receiver interact sequentially , in which the receiver's decision-making problem is usually modeled as a Markov decision process (MDP). However, previous works focused on computing optimal information-revelation policies (a.k.a. signaling schemes ) under the restrictive assumption that the receiver acts myopically , selecting actions to maximize the one-step reward and disregarding future rewards. This is justified by the fact that, when the receiver is farsighted and thus considers future rewards, finding an optimal Markovian signaling scheme is NP -hard. In this paper, we show that Markovian signaling schemes do not constitute the 'right' class of policies. Indeed, differently from most of the MDPs settings, we prove that Markovian signaling schemes are not optimal, and general history-dependent signaling schemes should be considered. Moreover, we also show that history-dependent signaling schemes circumvent the negative complexity results affecting Markovian signaling schemes. Formally, we design an algorithm that computes an optimal and glyph[epsilon1] -persuasive history-dependent signaling scheme in time polynomial in 1 /glyph[epsilon1] and in the instance size. The crucial challenge is that general history-dependent signaling schemes cannot be represented in polynomial space. Nevertheless, we introduce a convenient subclass of history-dependent signaling schemes, called promise-form , which are as powerful as general history-dependent ones and efficiently representable. Intuitively, promise-form signaling schemes compactly encode histories in the form of honest promises on future receiver's rewards.

1 Introduction

Bayesian persuasion [Kamenica and Gentzkow, 2011] is the problem faced by an informed sender who wants to influence the behavior of an uninformed, self-interested receiver through the provision of payoff relevant information. Bayesian persuasion captures many fundamental problems arising in real-world applications, e.g. , online advertising [Bro Miltersen and Sheffet, 2012], voting [Alonso and Câmara, 2016, Castiglioni et al., 2020a, Castiglioni and Gatti, 2021], traffic routing [Bhaskar et al., 2016, Castiglioni et al., 2021a], recommendation systems [Mansour et al., 2016], security [Rabinovich et al., 2015, Xu et al., 2016], marketing [Babichenko and Barman, 2017, Candogan, 2019], medical research [Kolotilin, 2015], and financial regulation [Goldstein and Leitner, 2018].

Matteo Castiglioni

Mirco Mutti ∗

Technion

Most of the previous works study the classical, one-shot version of the Bayesian persuasion problem. However, in many application scenarios it is natural to assume that the sender and the receiver interact multiple times in a sequential manner. In spite of this, only a few very recent works addressed sequential versions of the Bayesian persuasion problem [Wu et al., 2022, Bernasconi et al., 2022, Gan et al., 2022a,b]. In particular, Wu et al. [2022] and Gan et al. [2022a,b] study settings where the sender and the receiver interact sequentially in a Markov decision process (MDP).

In Bayesian persuasion problems in MDPs, at each step of the interaction both the sender and the receiver know the current state of the MDP, and the former has also access to some private observation drawn according to a commonly-known, state-dependent distribution. The sender commits beforehand to an information-revelation policy, which is implemented by means of a signaling scheme that sends randomized action recommendations to the receiver, conditioned on the current (public) state and the sender's private observation. Specifically, the sender commits to a persuasive signaling scheme, meaning that the receiver is always incentivized to follow recommendations. At the end of each step, the next state of the MDP and the agents' rewards are determined as a function of the current state, the action actually played by the receiver, and the sender's private observation in the current step.

Wu et al. [2022] and Gan et al. [2022a,b] provide algorithms that compute an optimal ( i.e. , rewardmaximizing) persuasive signaling scheme under the restrictive assumption that the receiver acts myopically , selecting actions to maximize the one-step reward and disregarding future ones. This is justified by the fact that, when the receiver is farsighted and thus considers future rewards, finding an optimal Markovian signaling scheme is NP -hard to approximate, as shown by Gan et al. [2022a] for infinite-horizon MDPs. An analogous result also holds in finite-horizon MDPs for non-stationary Markovian signaling schemes, as we prove in this work as a preliminary result.

In this paper, we show that Wu et al. [2022] and Gan et al. [2022a,b] failed to provide positive results with farsighted receivers since Markovian signaling schemes do not constitute the 'right' class of policies to consider. This is in stark contrast with most of the MDPs settings in which Markovian policies are optimal. Indeed, we prove that Markovian signaling schemes are not optimal, and general history-dependent signaling schemes should be considered. As a result, we focus on the problem of computing an optimal persuasive history-dependent signaling scheme. Surprisingly, we show that taking history into account allows to circumvent the negative result affecting Markovian signaling schemes. We do that by providing an approximation scheme that finds an optimal glyph[epsilon1] -persuasive ( i.e. , one approximately incentivizing the receiver to follow action recommendations) history-dependent signaling scheme in time polynomial in 1 /glyph[epsilon1] and the size of the problem instance.

The crucial challenge in designing our approximation scheme is that general, history-dependent signaling schemes cannot be represented in polynomial space. Our algorithm overcomes such an issue by using a convenient subclass of history-dependent signaling schemes, which we call promise-form signaling schemes. The core idea of such signaling schemes is to compactly encode all the relevant information contained in an history into a promise on future receiver's rewards. At each step of the process, a promise-form signaling scheme does not only determines an action recommendation for the receiver, but it also makes a promise to them. First, we prove that promise-form signaling schemes are as powerful as general history-dependent ones. Then, we show how an optimal glyph[epsilon1] -persuasive promiseform signaling scheme can be computed in polynomial time by means of a recursive procedure. To do that, we rely on a crucial result showing that, for signaling schemes that honestly keep promises made to the receiver, persuasiveness constraints can be expressed as conditions defined locally at each step of the MDP, since the receiver only cares about sender's promises on their future rewards. 2

2 Preliminaries

In this work, we study Bayesian persuasion problems where a farsighted receiver takes actions in a time-inhomogeneous finite-horizon MDP [Puterman, 2014]. Formally, a problem instance is a tuple

S is a finite set of states, A is a finite set of receiver's actions available in each state, H := [1 , . . . , H ] is a set of time steps with H being the time horizon, Θ is a finite set of sender's private observations, r S h , r R h : S × A × Θ → [0 , 1] are reward functions for the sender and the receiver, respectively,

p h : S ×A× Θ → ∆( S ) is a transition function, µ h : S → ∆(Θ) is a function defining probabilities of sender's private observations at each state, while β ∈ ∆( S ) is the initial state distribution. 3

We consider the most general setting in which the sender commits to a non-stationary and nonMarkovian signaling scheme (henceforth called history-dependent signaling scheme for short). For every step h ∈ H and state s h ∈ S reached at that step, a history-dependent signaling scheme defines a randomized mapping from sender's private observations to action recommendations for the receiver, based on the whole history of states and receiver's actions observed up to step h . 4 Formally, in the following we let T h be the set of all the possible histories up to step h , which is defined as

while we let T := T 1 ∪ . . . ∪ T H be the set of all the possible histories (of any length). 5 Then, a history-dependent signaling scheme is defined as a set φ := { φ τ } τ ∈T of functions φ τ : Θ → ∆( A ) , which define a mapping from sender's private observations to probability distributions over action recommendations for every possible history.

The interaction between the sender and the receiver goes as follows (Algorithm 1). (i) The sender publicly commits to a history-dependent signaling scheme φ := { φ τ } τ ∈T . (ii) An initial state s 1 ∼ β is drawn. (iii) At each step h ∈ H , both agents observe the current state s h ∈ S and the sender also gets a private observation θ h ∈ Θ drawn according to µ h ( s h ) , with the function µ h being known to both the sender and the receiver. (iv) The sender communicates to the receiver an action recommendation a h ∈ A sampled according to φ τ h ( θ h ) , where τ h ∈ T h is the history up to step h . (v) The receiver plays an action ̂ a h ∈ A , possibly different from a h . (vi) The sender and the receiver get rewards r R h ( s h , a h , θ h ) and r S h ( s h , a h , θ h ) , respectively. (vii) If h = H , the interaction ends, otherwise glyph[negationslash]

glyph[negationslash]

the next state s h +1 ∼ p h ( s h , ̂ a h , θ h ) is drawn and the interaction continues to step h +1 starting from the third point. As customary in the literature (see, e.g. , [Bernasconi et al., 2022]), we assume that, if the receiver does not follow recommendations at some step h ∈ H by playing an action ̂ a h = a h , then the sender stops issuing future recommendations to the receiver.

For ease of presentation, we introduce the sender's value function V S ,φ h : T h → R to encode sender's expected rewards by using a history-dependent signaling scheme φ := { φ τ } τ ∈T from step h ∈ H onwards, assuming the receiver always follows recommendations. Given a history τ = ( s 1 , a 1 , . . . , s h -1 , a h -1 , s h ) ∈ T h up to step h , such a value function is recursively defined as:

Similarly, we introduce the receiver's action-value function V R ,φ h : A × T h → R to encode the receiver's expected rewards by following sender's action recommendations from step h ∈ H onwards. Formally, given a history τ = ( s 1 , a 1 , . . . , s h -1 , a h -1 , s h ) ∈ T h up to step h , the receiver's expected reward by following the recommendation to play a ∈ A is recursively defined as follows:

Finally, we introduce an additional receiver's value function, denoted by ̂ V R h : S → R , to encode receiver's expected rewards from step h ∈ H onwards after having deviated from recommendations. Formally, for every state s ∈ S , such a value function is recursively defined as follows:

where the maximum operator encodes the fact that the receiver plays so as to maximize future rewards without knowledge of realized sender's private observations after having deviated. 6

By the revelation principle [Kamenica and Gentzkow, 2011], it is well known that in order to find an optimal signaling scheme it is possible to focus w.l.o.g. on (direct) history-dependent signaling schemes under which the receiver is always incentivized to follow sender's action recommendations. These are called persuasive signaling scheme, and they are formally defined as follows:

Definition 1 ( glyph[epsilon1] -persuasiveness) . Let glyph[epsilon1] ≥ 0 . A history-dependent signaling scheme φ := { φ τ } τ ∈T is said to be glyph[epsilon1] -persuasive if, for every step h ∈ H , history τ = ( s 1 , a 1 , . . . , s h -1 , a h -1 , s h ) ∈ T h up to step h , and pair of actions a, a ′ ∈ A , the following holds:

Moreover, we say that the signaling scheme is persuasive if the conditions above hold for glyph[epsilon1] = 0 . We denote the set of all the persuasive signaling schemes by Φ .

Whenever the sender commits to an glyph[epsilon1] -persuasive history-dependent signaling scheme, we assume that the receiver always follows the recommendation issued by the sender. Notably, the recommended action is guaranteed to provide the receiver expected rewards that are at worst glyph[epsilon1] less than those attained by the best possible action.

In conclusion, the goal is to find an optimal signaling scheme for the sender, which is one achieving a sender's expected reward (from step one) V S ,φ greater than or equal to OPT , defined as follows:

3 History-dependent signaling schemes are necessary

Previous works studying Bayesian persuasion in MDPs [Wu et al., 2022, Gan et al., 2022a] focus on Markovian signaling schemes, in which the action recommendation at step h only depends on the current state s h and private observation θ h . Indeed, considering this class of signaling schemes is sufficient to optimize the utility of a sender facing a myopic receiver. Here, we show that this is not the case when the receiver is farsighted. In particular, we show that non-stationary Markovian signaling scheme are suboptimal. To do so, we show that there exists an MDP (see Appendix A) in which the optimal persuasive signaling scheme is history-dependent. 7 Intuitively, an history-dependent signaling scheme can adjust action recommendations depending on the choices available to the receiver in previous steps. Thus, if the receiver had profitable opportunities in the past, the sender must provide a larger expected reward to the receiver in order to be persuasive. Otherwise, the sender can aggressively maximize their expected rewards irrespective of receiver's ones. Formally:

Theorem 1. There exist instances in which a persuasive history-dependent signaling scheme guarantees sender's expected rewards strictly greater than that obtained by an optimal persuasive non-stationary Markovian signaling scheme.

Moreover, we show that optimal non-stationary Markovian signaling schemes are NP -hard to approximate in polynomial time, even when the persuasiveness requirement is relaxed. This further motivates the use of history-dependent signaling schemes when addressing Bayesian persuasion problems in finite-horizon MDPs with a farsighted receiver. Formally, we prove the following:

Theorem 2. There exist two constants α < 1 and glyph[epsilon1] > 0 such that computing an glyph[epsilon1] -persuasive non-stationary Markovian signaling that provides the sender with at least a fraction α of the optimal sender's expected reward OPT is NP -hard .

4 A sufficient subclass of efficiently-representable signaling schemes

Working with history-dependent signaling schemes begets unavoidable computational issues. These are due to the fact that explicitly representing such signaling schemes requires a number of bits growing exponentially in the size of the problem instance, since the number of possible histories is exponential in the time horizon H . In this section, we show how to circumvent such an issue by introducing a convenient subclass of signaling schemes-called promise-form signaling schemeswhich are efficiently representable while being as good as history-dependent ones. In particular, our main result in this section (Theorem 3) shows that there always exists a promise-form signaling scheme which results in sender's expected rewards equal to its optimal value OPT .

4.1 Promise-form signaling schemes

A promise-form signaling scheme is defined by a set σ := { ( I h , ϕ h , g h ) } h ∈H of triplets, where:

I h : S → 2 [0 ,H ] is a function defining, for every state s ∈ S , a finite set I h ( s ) ⊆ [0 , H ] of promises for step h ∈ H . We add the additional requirement that 0 ∈ I 1 ( s ) for all s ∈ S , and, for ease of notation, we set I H +1 ( s ) := { 0 } for every s ∈ S and I := ⋃ h ∈H ⋃ s ∈S I h ( s ) .

ϕ h : S × I × Θ → ∆( A ) is an action-recommendation strategy to be employed at step h ∈ H , where ϕ h ( a | s, ι, θ ) is the probability of recommending action a ∈ A in state s ∈ S when the promise is ι ∈ I h ( s ) and the sender's private observation is θ ∈ Θ .

g h : S × A × I × S → I is a promise function for step h ∈ H such that, whenever h ≤ H and ι ∈ I h ( s ) , g h ( s, a, ι, s ′ ) ∈ I h +1 ( s ′ ) represents the promise for step h +1 if the next state is s ′ ∈ S and, at the current step h , action a ∈ A is recommended in state s ∈ S . 8

Intuitively, the rationale behind promise-form signaling schemes is that, when reaching a state s h ∈ S at step h ∈ H , the sender 'promises' a value ι ∈ I h ( s h ) to the receiver, representing a lower bound on future rewards obtained by following recommendations. Moreover, sender's action recommendations only depend on the current state s h ∈ S , the sender's private observation θ h ∈ Θ , and the current promise ι ∈ I h ( s h ) , through the distribution ϕ h ( s h , ι h , θ h ) . Notice that it is always possible to infer the current promise by looking at the history of past states and action recommendations, by 'composing' the functions g h ′ for the steps h ′ < h . This crucially avoids having to specify an explicit dependency on the full history of past states and action recommendations.

Notice that a promise-form signaling scheme as defined above does not automatically guarantee that the sender honestly keeps their promises. Indeed, in order to ensure that this is the case, we need to enforce additional constraints on the components of the signaling scheme, as we show in Section 4.3.

Let us remark that representing promise-form signaling schemes requires a number of bits polynomial in the size of the problem instance and in |I| , which is the cardinality of the set of promises. While |I| could be arbitrarily large in general, the algorithm that we will present in the following Section 5 guarantees that |I| has 'small' size, by means of a clever choice of the functions I h .

4.2 From promise-form to history-dependent signaling schemes

In the following, we show how the sender can implement promise-form signaling schemes, proving that they represent a subclass of history-dependent ones.

The sender can implement a promise-form signaling scheme σ := { ( I h , ϕ h , g h ) } h ∈H as follows. After committing to σ (Line 1 of Algorithm 1), at each step h ∈ H , in Line 6 of Algorithm 1 they select which action-recommendation strategy to use by reconstructing the current promise on the basis of the history τ h . Such a reconstruction is done by recursively 'composing' functions g h ′ for the preceding steps h ′ < h , by means of the procedure in Algorithm 2 with σ and τ h as inputs. By letting ι ∈ I h ( s h ) be the continuation value obtained by running Algorithm 2, the action recommendation a h in Line 6 is then sampled from ϕ h ( s h , ι, θ h ) . Algorithm 2 clearly runs in time polynomial in the instance size, and, thus, the sender can implement a promise-form signaling scheme efficiently.

In the rest of this section, given a promise-form signaling scheme σ := { ( I h , ϕ h , g h ) } h ∈H , we denote by φ σ := { φ σ τ } τ ∈T the history-dependent signaling scheme induced by σ thorough the implementation procedure described above. Formally, for every history τ = ( s 1 , a 1 , . . . , s h -1 , a h -1 , s h ) ∈ T h up to step h ∈ H , the function φ σ τ : Θ → ∆( A ) is defined so that φ σ τ ( θ ) = ϕ ( s h , ι σ τ , θ ) for every θ ∈ Θ , where ι σ τ ∈ I h ( s h ) denotes the promise value correspond-

Algorithm 2 From histories to promises

ing to history τ , as computed by Algorithm 2 with σ and τ as inputs. As it is easy to see, implementing the promise-form signaling scheme σ as described above is equivalent to using φ σ in Algorithm 1.

Next, we show that the value functions of the sender and the receiver associated with the induced history-dependent signaling scheme φ σ can be efficiently computed by only accessing the components of the promise-form signaling scheme σ . In the following, given any σ := { ( I h , ϕ h , g h ) } h ∈H , for every step h ∈ H we introduce the functions V R ,σ h : A×S×I → R and V R ,σ h : S × I → R , which are jointly recursively defined so that, for every a ∈ A , s ∈ S , and ι ∈ I h ( s ) , it holds:

Then, we can prove the following lemma:

Lemma 1. Given a promise-form signaling scheme σ := { ( I h , ϕ h , g h ) } h ∈H , for every h ∈ H and history τ = ( s 1 , a 1 , . . . , s h -1 , a h -1 , s h ) ∈ T h up to step h , the following holds:

Intuitively, Lemma 1 establishes that the functions V R ,σ h , V R ,σ h , and V S ,σ h 'correctly' encode the value functions of the sender and the receiver for a promise-form signaling scheme σ , when it is implemented according to the procedure described at the beginning of the section.

Finally, given how a promise-form signaling scheme σ is implemented, in the following we say that σ is glyph[epsilon1] -persuasive (for some glyph[epsilon1] ≥ 0 ) if the induced history-dependent signaling scheme φ σ is glyph[epsilon1] -persuasive according to Definition 1. However, using such a definition to check whether σ is glyph[epsilon1] -persuasive is clearly computationally inefficient, since it would require working with exponentially-many histories. In Section 4.3, we introduce an easy way to ensure that a promise-form signaling scheme 'keeps its promises', and we show that this allows to encode glyph[epsilon1] -persuasiveness constraints in an efficient way.

4.3 The power of honesty

Next, we introduce a particular class of promise-form signaling schemes which always guarantee that the sender honestly (approximately) assures promised rewards to the receiver. We call η -honest the promise-form signaling schemes with such a property, which are formally defined as follows:

Definition 2 ( η -honesty) . Let η ≥ 0 . A promise-form signaling scheme σ := { ( I h , ϕ h , g h ) } h ∈H is η -honest if, for every step h ∈ H , state s ∈ S , and promise ι ∈ I h ( s ) , the following holds:

Then, we can prove the following result on η -honest promise-form signaling schemes:

Lemma 2. Let σ := { ( I h , ϕ h , g h ) } h ∈H be a promise-form signaling scheme. If σ is η -honest, then, for every step h ∈ H and state s ∈ S , it holds that V R ,σ h ( s, ι ) ≥ ι -η ( H -h +1) for all ι ∈ I h ( s ) .

Since by Lemma 1 the function V R ,σ h encodes the receiver's value function when σ is implemented by the sender, Lemma 2 intuitively establishes that, at each step h ∈ H and state s ∈ S , the signaling scheme actually 'keeps the promise' of giving at least ι ∈ I h ( s ) future rewards to the receiver, up to an error depending on η . Checking whether a promise-form signaling scheme σ := { ( I h , ϕ h , g h ) } h ∈H is η -honest or not can be done in time polynomial in the instance size and |I| .

Now, we are ready to prove the following crucial lemma:

Lemma 3. Let σ := { ( I h , ϕ h , g h ) } h ∈H be an η -honest promise-form signaling scheme such that, for every h ∈ H , s ∈ S , ι ∈ I h ( s ) , and a, a ′ ∈ A , the following constraint is satisfied:

Then, we can conclude that σ is ( ηH ) -persuasive.

Intuitively, Lemma 3 states that for an η -honest promise-form signaling scheme, the constraint in Equation (2) is equivalent to the one in Definition 1. The crucial advantage of Equation (2) is that it allows to express persuasiveness conditions as 'local' constraints which do not require recursion.

4.4 Promise-form signaling schemes are sufficient

Finally, by exploiting Lemma 3, we can prove the main result of this section: promise-form signaling schemes represent a sufficient subclass of history-dependent ones. Formally:

Theorem 3. There is always a persuasive promise-form signaling scheme σ := { ( I h , ϕ h , g h ) } h ∈H with sender's expected reward equal to OPT . More formally, it holds that V S ,φ σ = OPT for the history-dependent signaling scheme φ σ induced by σ .

5 Approximation scheme

Theorem 3 shows that, in order to find an optimal signaling scheme, one can focus on promise-form signaling schemes that have the nice property of being polynomially representable in the instance size and the cardinality |I| of the set of promises. In this section, we show how to compute an glyph[epsilon1] -persuasive promise-form signaling scheme with sender's expected rewards at least OPT in polynomial time.

We design an algorithm working with sets I h ( s ) defined on a suitable grid, whose size can be properly controlled by a discretization step δ . The algorithm solves a recursively-defined optimization problem for each h ∈ H , s ∈ S , and ι ∈ I h ( s ) , by starting from step H and proceeding in bottom up fashion.

For every step h ∈ H and state s ∈ S , the set I h ( s ) of promises is defined as a suitable subset of a grid D δ := { k δ | k ∈ N ∧ k ≤ glyph[floorleft] H / δ glyph[floorright]} , where δ > 0 is a discretization step to be set depending on the desired relaxation glyph[epsilon1] of the persuasiveness constraints. This also allows us to control the representation size of promise-form signaling schemes, as well as the running time of the algorithm. In particular, it holds |D δ | = O (1 /δ ) and, thus, |I| = O (1 /δ ) since I ⊆ D δ by definition. In the following, for ease of notation, we let glyph[ceilingleft] x glyph[ceilingright] δ := min k ∈ N : kδ ≥ x kδ be the smallest multiple of δ greater than x , while glyph[floorleft] x glyph[floorright] δ := max k ∈ N : kδ ≤ x kδ is the greatest multiple of δ smaller than x .

The algorithm keeps track of recursively-computed values in a set of tables, one for each step. The table at step h ∈ H is encoded by means of a function M δ h : S × D δ → R ∪ {-∞} . Intuitively, for every s ∈ S and ι ∈ D δ , the entry M δ h ( s, ι ) is related to the expected rewards achieved by the sender when 'promising' the receiver expected rewards 'approximately equal' to ι in state s at step h . We also admit the functions M δ h to take value -∞ , which semantically corresponds to the case in which it is impossible to guarantee the promise ι to the receiver in state s at step h . The entry M δ h ( s, ι ) of

the table M δ h at step h is computed recursively by solving a problem P h,s,ι ( M δ h +1 ) that we define in the following, where M δ h +1 is the (previously-computed) table at step h +1 .

By letting M : S × D δ → R ∪{-∞} be a function encoding a generic table over S × D δ , for every step h ∈ H , state s ∈ S , and promise ι ∈ I h ( s ) , we define the value Π h,s,ι ( M ) of the optimization problem P h,s,ι ( M ) as follows:

where problem variables are encoded by the functions κ : Θ → ∆( A ) and q : A×S → D δ , which represent an action-recommendation strategy and a promise function, respectively. 9 The objective function F h,s,M ( κ, q ) of the optimization problem is defined as:

which encodes the sender's expected reward when their values for the next step h + 1 are those specified by the table M . We assume that 0 · ( -∞ ) = -∞ . Moreover, the set Ψ h,s ι is comprised of the functions κ : Θ → ∆( A ) and q : A×S → D δ that satisfy the following constraints:

where Equation (4a) and Equation (4b) play the role of the honesty and the persuasiveness constraints, respectively. Notice that relaxing the honesty constraint yields larger feasible sets. Formally, for any ι ≥ ι ′ ≥ 0 we have that the following holds: Ψ h,s ι ⊆ Ψ h,s ι ′ .

If F h,s,M ( κ, q ) = -∞ for all ( κ, q ) ∈ Ψ h,s ι , we have that Π h,s,ι ( M ) = -∞ . This intuitively comes from the fact that, if Π h,s,ι ( M ) = -∞ , then the value ι promised to the receiver is not realizable.

The optimization problem P h,s,ι ( M ) could be easily cast as a mixed-integer quadratic program, which are too general to be solved efficiently. Thus, we need specifically-tailored procedures to find an approximate solution to it. This discussion is deferred to Section 6. In the following, we assume to have access to an oracle O h,s,ι ( M ) that provides a suitable approximate solution to P h,s,ι ( M ) . In particular, O h,s,ι ( M ) must satisfy the requirements introduced by the following definition:

Definition 3 (Approximate Oracle) . An algorithm O h,s,ι ( M ) is an approximate oracle for P h,s,ι ( M ) if it returns a tuple ( κ, q, v ) such that Π h,s,ι ( M ) ≤ v ≤ F h,s,M ( κ, q ) and ( κ, q ) ∈ Ψ h,s ι -δ .

Intuitively, we ask that an approximate oracle finds a solution ( κ, q ) in a slightly larger set Ψ h,s ι -δ . The oracle also returns a value v to be inserted into M δ h ( s, ι ) , where v is possibly different from the value F h,s,M ( κ, q ) of the objective function. This is needed for technical reasons in order to recover some concavity properties of the functions defining the tables used by the algorithm, which may be lost due to approximations. A complete discussion on this last aspect can be found in Section 6.

Equipped with an approximate oracle as in Definition 3, we are ready to design our approximation scheme that computes an glyph[epsilon1] -persuasive promise-form signaling scheme attaining sender's expected reward at least OPT (Algorithm 3). The algorithm iteratively builds each table M δ h by filling it with the values v returned by the approximate oracle. Moreover, it sets action-recommendation strategies ϕ h and promise functions g h of the signaling scheme σ := { ( I h , ϕ h , g h ) } h ∈H to be equal to the functions κ , q returned by the approximate oracle.

In order to clarify the semantic of the tables M δ h built by Algorithm 3, we prove the following preliminary result, which states that the sender's expected rewards for the signaling scheme returned by Algorithm 3 constitute an upper bound on the entries of the tables M δ h . Formally:

Lemma 4. Let σ := { ( I h , ϕ h , g h ) } h ∈H be returned by Algorithm 3 instantiated with any oracle O h,s,ι as in Definition 3. For every h ∈ H , s ∈ S , ι ∈ I h ( s ) , it holds that V S ,σ h ( s, ι ) ≥ M δ h ( s, ι ) .

Moreover, the entries of the tables M δ h built by Algorithm 3 are upper bounds on the sender's expected rewards provided by an optimal promiseform signaling scheme. Formally:

Lemma 5. Let M δ h , I h (for h ∈ H ) be computed by Algorithm 3 instantiated with any oracle O h,s,ι

Algorithm 3 Approximation scheme

as in Definition 3, and let σ glyph[star] = { ( I glyph[star] h , ϕ glyph[star] h , g glyph[star] h ) } h ∈H be an optimal promise-form signaling scheme. For every h ∈ H , s ∈ S , and ι ∈ I glyph[star] h ( s ) , it holds that glyph[ceilingleft] ι glyph[ceilingright] δ ∈ I h ( s ) and M δ h ( s, glyph[ceilingleft] ι glyph[ceilingright] δ ) ≥ V S ,σ glyph[star] h ( s, ι ) .

By combining Lemma 5 and Lemma 4 we can promptly state and prove the main result of this section.

Theorem 4. For any glyph[epsilon1] > 0 , given an approximate oracle as in Definition 3, Algorithm 3 instantiated with δ = glyph[epsilon1] / 2 H runs in time polynomial in 1 /glyph[epsilon1] and the instance size, while it finds an glyph[epsilon1] -persuasive promise-form signaling scheme that guarantees expected reward at least OPT to the sender.

6 Building a polynomial-time approximate oracle

In the previous section, we described an approximation scheme (Algorithm 3) that works provided it has access to an oracle that approximately solves the problem P h,s,ι ( M ) while satisfying the requirements of Definition 3. In this section, we exploit the specific structure of the problem to design an algorithm implementing such an oracle.

First, we relax the problem P h,s,ι ( M ) into R h,s,ι ( M ) , which optimizes the expected value of the sender's rewards over randomizations of promises ι ∈ D δ . Moreover, we define for all s ′ ∈ S the set D δ ( s ′ , M ) = { ι ∈ D δ : M ( s ′ , ι ) > -∞} as the set of realizable promises , which semantically means

Algorithm 4 Approximate oracle

that those promises can be realized, conditioned on what is stored in the table M . 10 Specifically, the variables of R h,s,ι ( M ) are encoded by a function κ : Θ → ∆( A ) and a randomized function ˜ q : S × A → ∆( D δ ) . The objective function is

which modifies the objective function F h,s,M ( κ, q ) of P h,s,ι ( M ) by introducing the expectation over the possible promises ι ′ ∈ D δ , which are randomized through ˜ q ( ι ′ | a, s ′ ) .

To simplify notation, for any set X , table M , and randomized function λ : X × S → ∆( D δ ) , we denote by λ E : X × S → co ( D δ ) a quantity related to its average, where co denotes the convex hull of a set. Formally, λ E ( x, s ′ ) = ∑ ι ′ ∈D δ ( s ′ ,M ) ι ′ · λ ( ι ′ | x ) for all x ∈ X . Moreover, we denote by glyph[floorleft] λ E glyph[floorright] δ : X → D δ its discrete average such that glyph[floorleft] λ E glyph[floorright] δ ( x, s ′ ) = glyph[floorleft] λ E ( x, s ′ ) glyph[floorright] δ for all x ∈ X and

s ′ ∈ S . Notice that, for ease of notation, we drop the dependence of the operator E from M , as such a dependence is always clear from the context.

By exploiting the notation introduced above, the relaxed optimization problem R h,s,ι ( M ) reads as:

where we relax the functions ˜ q to be distributions over D δ , rather than deterministic. Then, we can prove the following:

Lemma 6. The problem R h,s,ι ( M ) can be solved in time polynomial in 1 /δ and the instance size.

By relying on a solution ( κ, ˜ q ) to R h,s,ι ( M ) , Algorithm 4 is an approximate oracle for P h,s,ι ( M ) . In particular, when the relaxed problem is feasible, the algorithm returns k and the function q obtained by de-randomizing the randomized function ˜ q with its discrete average. Formally:

Theorem 5. For every h ∈ H , s ∈ S , and ι ∈ D δ , if Algorithm 4 is used for all h ′ > h as approximate oracle O h ′ ,s,ι in Algorithm 3, then it implements an approximate oracle as in Definition 3.

Notice that the value v returned by Algorithm 4 is the optimal value of the relaxed problem R h,s,ι ( M ) . It is easy to prove (see the proof of Theorem 5) that the tables built by Algorithm 3 through Algorithm 4 are encoded by concave functions. This is the key feature that allows us to go from a randomized solution to its discrete average without loss in sender's expected rewards, and it is the reason why we need to assume that Algorithm 4 is employed at every step h in Theorem 5.

7 Conclusion

In this paper, we provided a crucial advancement in Bayesian persuasion for sequential decision making, mastering the setting in which a farsighted receiver interacts with an MDP, a problem that was previously thought to be intractable [Wu et al., 2022, Gan et al., 2022a,b].

First, we have shown that the class of Markovian signaling schemes, the standard choice in previous works and most MDP settings, do not constitute the 'right' class of policies, since finding an optimal Markovian signaling scheme is NP -hard. Instead, we demonstrated that history-dependent signaling schemes allow to both circumvent the negative complexity result affecting Markovian signaling schemes and guarantee higher expected rewards to the sender in general. Specifically, we designed an algorithm to find an optimal glyph[epsilon1] -persuasive history-dependent signaling scheme in time polynomial in 1 /glyph[epsilon1] and in the instance size. The crucial component of the algorithm is to restrict the optimization to a convenient subclass of history-dependent signaling schemes, which we call promise-form as they encode the history into a promise of future rewards to the receiver. We showed that promise-form signaling schemes are both efficient to represent and as powerful as the general class of historydependent signaling schemes. An interesting insight of our analysis reveals that being honest , i.e. , keeping promises of future rewards, is the key to persuade farsighted receivers in MDPs.

As a future work, we will investigate the online learning variant of the problem studied in this paper, as done in [Zu et al., 2021, Wu et al., 2022, Bernasconi et al., 2022], in which the transition function is not known. Moreover, it would be interesting to study the problem faced by a sender that needs to persuade a stream of receivers having adversarially-selected types, as done in [Balcan et al., 2015, Castiglioni et al., 2020b, 2021b, Bernasconi et al., 2023].

References

Paola Alimonti and Viggo Kann. Some APX-completeness results for cubic graphs. Theoretical Computer Science , 237:123-134, 04 2000.

Ricardo Alonso and Odilon Câmara. Persuading voters. American Economic Review , 2016.

Yakov Babichenko and Siddharth Barman. Algorithmic aspects of private Bayesian persuasion. In ITCS , 2017.

Maria-Florina Balcan, Avrim Blum, Nika Haghtalab, and Ariel D Procaccia. Commitment without regrets: Online learning in stackelberg security games. In EC , 2015.

Martino Bernasconi, Matteo Castiglioni, Alberto Marchesi, Nicola Gatti, and Francesco Trovò. Sequential information design: Learning to persuade in the dark. In NeurIPS , 2022.

Martino Bernasconi, Matteo Castiglioni, Andrea Celli, Alberto Marchesi, Francesco Trovò, and Nicola Gatti. Optimal rates and efficient algorithms for online bayesian persuasion. In ICML , 2023.

Dimitri Bertsekas. Network optimization: continuous and discrete models , volume 8. Athena Scientific, 1998.

Umang Bhaskar, Yu Cheng, Young Kun Ko, and Chaitanya Swamy. Hardness results for signaling in bayesian zero-sum and network routing games. In EC , 2016.

Peter Bro Miltersen and Or Sheffet. Send mixed signals: earn more, work less. In EC , 2012.

Ozan Candogan. Persuasion in networks: Public signals and k-cores. In EC , 2019.

Matteo Castiglioni and Nicola Gatti. Persuading voters in district-based elections. In AAAI , 2021.

Matteo Castiglioni, Andrea Celli, and Nicola Gatti. Persuading voters: It's easy to whisper, it's hard to speak loud. In AAAI , 2020a.

Matteo Castiglioni, Andrea Celli, Alberto Marchesi, and Nicola Gatti. Online Bayesian persuasion. In NeurIPS , 2020b.

Matteo Castiglioni, Andrea Celli, Alberto Marchesi, and Nicola Gatti. Signaling in Bayesian network congestion games: the subtle power of symmetry. In AAAI , 2021a.

Matteo Castiglioni, Alberto Marchesi, Andrea Celli, and Nicola Gatti. Multi-receiver online Bayesian persuasion. In ICML , pages 1314-1323, 2021b.

Jiarui Gan, Rupak Majumdar, Goran Radanovic, and Adish Singla. Bayesian persuasion in sequential decision-making. In AAAI , 2022a.

Jiarui Gan, Rupak Majumdar, Goran Radanovic, and Adish Singla. Sequential decision making with information asymmetry. In CONCUR , 2022b.

Itay Goldstein and Yaron Leitner. Stress tests and information disclosure. Journal of Economic Theory , 177:34-69, 2018.

Emir Kamenica and Matthew Gentzkow. Bayesian persuasion. American Economic Review , 101(6): 2590-2615, 2011.

Anton Kolotilin. Experimental design to persuade. Games and Economic Behavior , 90:215-226, 2015.

Yishay Mansour, Aleksandrs Slivkins, Vasilis Syrgkanis, and Zhiwei Steven Wu. Bayesian exploration: Incentivizing exploration in Bayesian games. In EC , 2016.

Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons, 2014.

Zinovi Rabinovich, Albert Xin Jiang, Manish Jain, and Haifeng Xu. Information disclosure as a means to security. In AAMAS , 2015.

Jibang Wu, Zixuan Zhang, Zhe Feng, Zhaoran Wang, Zhuoran Yang, Michael I Jordan, and Haifeng Xu. Markov persuasion processes and reinforcement learning. In EC , 2022.

Haifeng Xu, Rupert Freeman, Vincent Conitzer, Shaddin Dughmi, and Milind Tambe. Signaling in Bayesian Stackelberg games. In AAMAS , 2016.

You Zu, Krishnamurthy Iyer, and Haifeng Xu. Learning to persuade on the fly: Robustness against ignorance. In EC , 2021.

A History-dependent signaling schemes are necessary

In this section, we provide an illustrative MDP instance in which an history-dependent signaling scheme is required in order to optimally solve a Bayesian persuasion problem with farsighted receiver. This instance also proves Theorem 1. The instance is depicted in Figure 1. It is easy to check that the receiver can achieve expected rewards

without sender's recommendations. If the sender commits to a non-stationary Markovian signaling scheme φ glyph[star] = { φ h } h ∈H , where φ h : S × Θ → ∆( A ) , they can achieve an expected reward of V S ,φ glyph[star] = V S ,φ glyph[star] 1 ( s 0 ) = 25 while being persuasive. The latter is obtained by recommending actions deterministically as follows: 11

Instead, an history-dependent signaling scheme φ † := { φ τ } τ ∈T can provide the sender with an expected reward of V S ,φ † = V S ,φ † 1 ( s 0 ) = 30 while being persuasive. This can only be obtained by adapting the action recommendation strategy in state s 3 according to whether the receiver passed through state s 1 or state s 2 , as follows:

which makes the recommendation φ = a 0 persuasive as well.

† ( s 0 ,s 2 )

a

0

, a

1

1

2

1

2

a

0

, a

1

a

0

a

1

a

0

, a

1

1

2

1

2

a

0

a

1

a

0

a

1

a

0

, a

1

s

0

s

1

s

2

s

3

s

4

θ

0

θ

1

r

S

=0

r

R

=10

r

S

=0

r

R

=

-

10

r

S

=0

r

R

=20

r

S

=20

r

R

=0

r

S

=100

r

R

=

20

r

S

=20

r

R

=0

-

The latter example proves the existence of an MDP instance where history-dependent signaling schemes are necessary. The reported instance does not require bizarre constructions, and we believe that exploiting history to optimize action recommendations is crucial in practical scenarios as well.

Intuition of Figure 1. In such a scenario, the MDP in Figure 1 can model a ride-sharing platform, where the drivers act as receivers and the platform is the sender. The platform can recommend routes and matching orders with the drivers, which then share with the platform a portion of the profit originating from the trips. In this example, s 3 can be thought as a location with high demand that is costly to reach, such as an airport terminal far from the city center. We can further think of the path s 1 → s 3 as a driver's trip to the airport while serving an order, and s 2 → s 3 as an empty trip instead. The platform can then adapt its order matching strategy according to whether the driver suffered empty costs or got a profit to come to the airport, such as guaranteeing a quick yet cheap match to the former, while waiting a more lucrative trip for the latter. This adaptive strategy can only be executed through an history-dependent signaling scheme. Instead, a Markovian signaling scheme cannot lure a driver into coming to the airport without serving an order, with diminishing profits for the platform.

B Proofs omitted from Section 3

Theorem 2. There exist two constants α < 1 and glyph[epsilon1] > 0 such that computing an glyph[epsilon1] -persuasive non-stationary Markovian signaling that provides the sender with at least a fraction α of the optimal sender's expected reward OPT is NP -hard .

Proof. We reduce from a promise version of vertex cover in cubic graphs. In particular, given a cubic graph ( V, E ) , there exists a γ ∈ (0 , 2] such that it is NP -hard to decide whether there exists a vertex cover of size k or all the vertex covers have size at least (1 + γ ) k [Alimonti and Kann, 2000].

We design an instance such that the set of states S includes s 0 , s 1 , s 2 and s 3 . Moreover, it includes a state s e for each e ∈ E , and a state s v for each v ∈ V . The time horizon is set as H = 3 . All the transition function and reward are independent from the time step h ∈ [ H ] . Hence, we will remove the subscript h from the notation.

The rewards and transition in the different states are as follows:

In state s 0 there is a single state of nature θ 0 and two actions a 0 , 1 and a 0 , 2 . The transition function is such that p ( s 3 | s 0 , a 0 , 1 , θ 0 ) = 1 while p ( s e | s 0 , a 0 , 2 , θ 0 ) = 1 | E | for each e ∈ E . The rewards of the receiver in state s 0 are r R ( a 0 , 1 , s 0 , θ 0 ) = 1 and r R ( a 0 , 2 , s 0 , θ 0 ) = 0 . The rewards of the sender in state s 0 are r S ( a 0 , 1 , s 0 , θ 0 ) = 0 and r S ( a 0 , 2 , s 0 , θ 0 ) = 1 .

In state s 1 , there is a single state of nature θ 1 and a single actions a 1 . The transition function is p ( s 2 | s 1 , a 1 , θ 1 ) = 1 , while the rewards of the sender and the receiver in s 1 are always 0 .

In state s 2 , there is a single state of nature θ 2 and a single actions a 2 . The transition function is p ( s v | s 2 , a 2 , θ 2 ) = 1 | V | for each v ∈ V . The sender's and receiver's rewards in s 2 are 0 .

In state s 3 , there is a single state of nature θ 3 and a single actions a 3 . The transition function is such that p ( s 3 | s 3 , a 3 , θ 3 ) = 1 . The rewards of the sender and the receiver in s 3 are always 0 .

For each e = ( v, u ) ∈ E , in the state s e , there is a single state of nature θ e , and two actions a e,v and a e,u . The transition function is such that p ( s v | s e , a e,v , θ e ) = 1 and p ( s u | s e , a e,u , θ e ) = 1 . The rewards of the sender and the receiver in state s e are 0 .

For each v ∈ V in the state s v , there are two states of nature θ v, 1 and θ v, 2 . There are three actions a v, 1 , a v, 2 , and a v, 3 . The transition function is such that p ( s 3 | s v , a, θ ) = 1 for each a and θ . The rewards of the receiver in s v are r R ( s v , a v, 1 , θ v, 1 ) = 1 , r R ( s v , a v, 2 , θ v, 2 ) = 1 , r R ( s v , a v, 3 , θ v, 1 ) = r R ( s v , a v, 3 , θ v, 2 ) = 1 2 and 0 otherwise. The rewards of the sender in s v is 1 2 if the receiver plays a v, 3 , i.e. , r S ( s v , a v, 3 , θ v, 1 ) = r S ( s v , a v, 3 , θ v, 2 ) = 1 2 , and 0 otherwise.

All nature's states are drawn uniformly on the nature's outcome at each node.

Finally, the initial distribution over the states is β such that β ( s 0 ) = β ( s 1 ) = 1 2 .

e

1

e

2

v

1

v

2

v

3

In Figure 2 we have a simple instance of a cubic graph (undirected graph with degree bounded by 3 ), and in Figure 3 we have the instance the persuasion MDP built from the cubic graph in Figure 2.

We recall that a non-stationary Markovian signaling scheme is defined by a set of functions { φ h } h ∈H , where φ h : S × Θ → ∆( A ) . However, we use both states s 1 and s 2 so that it takes two steps to arrive in states s v both when starting from s 0 and when starting from s 1 . Hence, each state s (excluding the sink state s 3 ) can be reached only with a specific time step h . This allows us to work only with stationary signaling scheme, i.e. , such that the signals do not depend on the time step. Hence, we can remove the subscript h from the signaling scheme notation.

a

1

a

2

1

|

V

|

1

|

V

|

1

|

V

|

1

|

E

|

1

|

E

|

a

0

,

2

a

e

1

,v

1

a

e

1

,v

2

a

e

2

,v

2

a

e

2

,v

3

a

0

,

1

a

3

s

1

s

2

s

v

1

s

v

2

s

v

3

s

e

1

s

e

2

s

0

s

3

G

v

3

G

v

1

G

v

2

r

S

=1

r

R

=0

r

S

=0

r

R

=1

(a) Instance of Bayesian Persuasion in MDP

r

S

=1

/

2

r

R

=1

/

2

r

S

=0

r

R

=0

r

S

=0

r

R

=1

r

S

=1

/

2

r

R

=1

/

2

r

S

=0

r

R

=1

r

S

=0

r

R

=0

1

2

1

2

a

v,

1

a

v,

2

a

v,

3

a

v,

1

a

v,

2

a

v,

3

s

v

θ

1

θ

2

s

3

G

v

(b) Gadget

G

v

Figure 3: Instance of Bayesian persuasion in MDP constructed from the cubic graph ( V, E ) of Figure 2. Sender's and receiver's rewards are reported only when different from 0 . Solid lines represents actions (that are reported on the corresponding edges) while dashed lines represents stochastic outcomes (either of the transition model or of the prior). For each v ∈ V the gadget G v is reported Figure 3b.

We will show that if there exists a vertex cover of size k then there exists a persuasive signaling scheme such that the sender's expected rewards is at least 3 4 -k 4 | V | , while if all the vertex covers have size at least (1 + γ ) k then all the glyph[epsilon1] -persuasive signaling schemes have sender's expected rewards strictly less than α ( 3 4 -k 4 | V | ) , where glyph[epsilon1] = 1 3 · 10 -4 γ 2 and α = 1 -γ 100 . This will conclude the

proof as we could use an algorithm for our problem with approximation factor less or equal to α to distinguish between the two cases of the original problem.

Completeness. Suppose there exists a vertex cover V ∗ of size k .

Consider the signaling scheme φ ∗ such that φ ∗ ( a 0 , 2 | s 0 , θ 0 ) = 1 , φ ∗ ( a 1 | s 1 , θ 1 ) = 1 , φ ∗ ( a 2 | s 2 , θ 2 ) = 1 , and φ ∗ ( a 3 | s 3 , θ 3 ) = 1 . Moreover, by the definition of vertex cover , for each e ∈ E there exists a vertex v e ∈ V ∗ such that v e ∈ e = ( v e , u ) . We set φ ∗ ( a e,v e | s e , θ e ) = 1 . Finally, for each v ∈ V ∗ , we set φ ∗ ( a v, 1 | s v , θ v, 1 ) = 1 , φ ∗ ( a v, 2 | s v , θ v, 2 ) = 1 , and φ ∗ ( a v, 3 | s v , θ v, 1 ) = φ ∗ ( a v, 3 | s v , θ v, 2 ) = 1 for all v / ∈ V ∗ .

First, we show that the signaling scheme is persuasive. Consider the state s 0 . The receiver following the recommendations will always reach a state v ∈ V ∗ and in this state will get utility 1 . It is easy to see that playing action a 0 , 1 the utility is 1 . Hence, in state s 0 the receiver will not deviate. In states s 1 , s 2 , and s 3 , there is only one action available, hence the signaling scheme is trivially persuasive in these states. In a state s e , e ∈ E , the receiver has no incentive to deviate. Indeed, following the recommendation they will transition to a state s v , v ∈ V ∗ , and will get 1 , while not following the recommendation they can get at most 1 . In all the states s v , for v ∈ V ∗ , the receiver gets 1 , while deviating can get at most 1 . Finally, in a state s v , v / ∈ V ∗ , the receiver gets 1 2 , while deviating can get at most 1 2 . Hence, the signaling scheme is persuasive.

Moreover, the sender's expected rewards for playing according to φ ∗ is 1 if the initial state is s 0 , as in this state the signaling scheme recommends a 0 , 2 . On the other hand, if the initial state is s 1 , then a state s v with v ∈ V ∗ is reached with probability k/ | V | and a state s v with v / ∈ V ∗ is reached with probability 1 -k/ | V | . Then, it is easy to see that the sender's expected rewards in reaching a state s v with v ∈ V ∗ is 0 and 1 2 otherwise. Thus the total sender's expected rewards of the signaling scheme φ ∗ is:

Soundness. Consider a signaling scheme φ , and suppose that all the vertex covers have size at least (1 + γ ) k . We show that the sender's expected rewards is strictly less than α ( 3 4 -k 4 | V | ) . First, notice that if the sender's expected rewards is at least α ( 3 4 -k 4 | V | ) it has to recommend a 0 , 2 with probability at least glyph[lscript] := 3 / 10 , i.e. , φ ( a 0 , 2 | s 0 , θ 0 ) ≥ 3 10 . Indeed, if this is not the case the sender's expected rewards is at most

as it gives at most 1 + 1 2 from s 0 when a 0 , 2 is played (which happens with probability less then 3 / 10 ) and 1 2 from s 1 . Thus we can only consider the cases in which φ ( a 0 , 2 | s 0 , θ 0 ) ≥ 3 10 .

Then, suppose that for n states s e , e ∈ E , the receiver's expected rewards following the recommendations from there on is at most 1 -δ , where δ = γ/ 100 . Denote with ̂ E ⊆ E be this set, where | ̂ E | = n , and with ˜ E := E \ ̂ E . The glyph[epsilon1] -persuasiveness constraint in state s 0 when the recommended action is a 0 , 2 implies:

as the left hand side is an upper bound on the receiver's expected rewards starting from s 0 . The equation above implies that,

and by rearranging it:

Let ˜ V be the set of states s v , v ∈ V , such that the receiver's expected rewards from there on is at least 1 -δ . Notice that for each state s e , with e ∈ ˜ E , there exists a state s v , with v ∈ ˜ V , such that v ∈ e , i.e. , v covers e . This implies that ˜ V covers ˜ E .

By contradiction, we show that | ˜ V | ≥ (1 + γ ) k -n . Suppose otherwise. Then, the set ˜ V is such that | ˜ V | < (1 + γ ) k -n and covers at least | ˜ E | edges. Hence, there exists a super-set of ˜ V of size strictly smaller than (1 + γ ) k that covers all the edges, reaching a contradiction. This holds as adding a vertex v for each e = ( v, u ) ∈ ̂ E to ˜ V , we obtain a vertex cover of E .

Then, we show that in all the states s v , v ∈ ˜ V the sender's expected rewards from there on is at most δ . It is easy to see that action a 3 is recommended with probability at most 2 /δ in each state s v , v ∈ ˜ V and the sender collects 1 2 . If this is not the case, the receiver's expected rewards is strictly smaller than 2 δ 1 2 +(1 -2 δ )1 = 1 -δ . This implies that the sender's expected rewards from a state s v , v ∈ ˜ V is at most δ . A similar argument shows that the sender's expected rewards from a state s e , e ∈ ˜ E , is at most δ .

We conclude the proof by showing an upper bound on the sender's expected rewards. In the following, we will exploit that the graph is cubic and | V | ≥ k ≥ | E | 3 . Moreover, we assume w.l.o.g. that there are no unconnected vertices, which implies that | E | ≥ | V | 2 and thus that k ≥ | V | 6 . Hence, the sender's expected rewards when the initial state is s 0 is at most:

Moreover, the sender's expected rewards when the initial state is s 1 is at most:

where we used that | ˜ V | ≥ (1 + γ ) k -n , n ≤ γ | E | / 90 and that | E | / | V | ≤ 3 .

Thus, since we start from s 0 or s 1 with probability 1 / 2 , the sender's expected rewards is at most:

where we used that k ≥ | V | / 6 and δ := γ/ 100 . This concludes the proof.

C Proofs omitted from Section 4

Lemma 1. Given a promise-form signaling scheme σ := { ( I h , ϕ h , g h ) } h ∈H , for every h ∈ H and history τ = ( s 1 , a 1 , . . . , s h -1 , a h -1 , s h ) ∈ T h up to step h , the following holds:

Proof. The statement can be easily proved by induction on the steps H .

The base case of the induction is h = H . By definition of V R ,φ σ H ( a, τ ) , for every action a ∈ A and history τ = ( s 1 , a 1 , . . . , s H -1 , a H -1 , s H ) ∈ T H , the following holds:

Since by construction φ σ τ ( θ ) = ϕ H ( s H , ι σ τ , θ ) for every θ ∈ Θ , we have that:

Now, take h < H and assume that the statement holds for h +1 . By definition of V R ,φ σ h ( a, τ ) , for every a ∈ A and τ = ( s 1 , a 1 , . . . , s h -1 , a h -1 , s h ) ∈ T h , it holds:

Moreover, for every s ′ ∈ S , the relation V R ,φ σ h +1 ( τ ⊕ ( a, s ′ )) = V R ,σ h +1 ( s ′ , ι σ τ ⊕ ( a,s ′ ) ) holds by induction since τ ⊕ ( a, s ′ ) belongs to T h +1 . Given how Algorithm 2 is designed, it holds:

which, together with φ σ τ ( θ ) = ϕ h ( s h , ι σ τ , θ ) for every θ ∈ Θ , gives V R ,φ h ( a, τ ) = V R ,σ h ( a, s h , ι σ τ ) . Similar inductive arguments show that V S ,φ σ h ( τ ) = V S ,σ h ( s h , ι σ τ ) for every step h ∈ H and history τ = ( s 1 , a 1 , . . . , s h -1 , a h -1 , s h ) ∈ T h up to h , concluding the proof.

σ

Lemma 2. Let σ := { ( I h , ϕ h , g h ) } h ∈H be a promise-form signaling scheme. If σ is η -honest, then, for every step h ∈ H and state s ∈ S , it holds that V R ,σ h ( s, ι ) ≥ ι -η ( H -h +1) for all ι ∈ I h ( s ) .

Proof. We prove the statement by induction.

The base case of the induction is h = H . Since σ is η -honest, for every s ∈ S and ι ∈ I H ( s ) :

Now, take h < H and assume that the statement holds for h +1 . For every s ∈ S and ι ∈ I h ( s ) :

where the first inequality holds by induction, the second one by η -honesty, while the last equality holds since ∑ θ ∈ Θ µ h ( θ | s ) ϕ h ( a | s, ι, θ ) ∑ s ′ ∈S p h ( s ′ | s, a, θ ) = 1 . This concludes the proof.

Lemma 3. Let σ := { ( I h , ϕ h , g h ) } h ∈H be an η -honest promise-form signaling scheme such that, for every h ∈ H , s ∈ S , ι ∈ I h ( s ) , and a, a ′ ∈ A , the following constraint is satisfied:

Then, we can conclude that σ is ( ηH ) -persuasive.

Proof. We recall that, for glyph[epsilon1] ≥ 0 , the promise-form signaling scheme σ is glyph[epsilon1] -persuasive if the historydependent signaling scheme φ σ := { φ σ τ } τ ∈T induced by σ is glyph[epsilon1] -persuasive according to Definition 1.

As a consequence, in order to prove the statement, we have to show that, for every step h ∈ H , history τ = ( s 1 , a 1 , . . . , s h -1 , a h -1 , s h ) ∈ T h up to step h , and pair of actions a, a ′ ∈ A , it holds:

By Lemma 1, we have that V R ,φ σ h ( a, τ ) = V R ,σ h ( a, s h , ι σ τ ) . Moreover, the following holds:

where the first inequality holds by Lemma 2, while the second one holds thanks to Equation 2. Finally, the statement is readily proved by noticing that η ( H -h ) ≤ ηH .

Theorem 3. There is always a persuasive promise-form signaling scheme σ := { ( I h , ϕ h , g h ) } h ∈H with sender's expected reward equal to OPT . More formally, it holds that V S ,φ σ = OPT for the history-dependent signaling scheme φ σ induced by σ .

Proof. The proof works by showing that, given any persuasive history-dependent signaling scheme, one can always build an η -honest and persuasive promise-form signaling scheme whose sender's expected reward is at least as good. This, together with Lemma 1 clearly proves the statement.

Given any persuasive history-dependent signaling scheme φ = { φ τ } τ ∈T , we build a promise-form signaling scheme σ = { ( I h , ϕ h , g h ) } h ∈H as follows:

I h ( s ) := { V R ,φ h ( τ ) | τ = ( s 1 , a 1 , . . . , s h -1 , a h -1 , s h ) ∈ T h ∧ s h = s } for all h > 1 , s ∈ S .

ϕ h ( s, ι, θ ) = φ τ glyph[star] h,s,ι ( θ ) for all h ∈ H , s ∈ S , ι ∈ I h ( s ) , and θ ∈ Θ , where:

which is guaranteed to exist given how I h ( s ) is defined.

g h ( s, a, ι, s ′ ) = V R ,φ h +1 ( τ glyph[star] h,s,ι ⊕ ( a, s ′ )) for all h ∈ H , s ∈ S , a ∈ A , ι ∈ I h ( s ) , and s ′ ∈ S .

As a first step, we prove that, if φ is persuasive, then the promise-form signaling scheme σ that we have just built is persuasive as well. This can be easily proved by exploiting Lemma 3.

First, we prove that σ is an η -honest for η = 0 . Formally, for every h ∈ H , s ∈ S , and ι ∈ I h ( s ) :

where Equation (6) holds by definition of τ glyph[star] h,s,ι , Equation (7) holds by definition of V R ,φ h ( τ glyph[star] h,s,ι ) , Equation (8) holds by definition of g h ( s, a, ι, s ′ ) , while Equation (9) holds by definition of ϕ h ( a | s, ι, θ ) . This proves that σ is an η -honest promise-form signaling scheme, for η = 0 .

In order to apply Lemma 3, we also need to to prove that σ satisfies the conditions in Equation (2). By applying definitions, it is easy to check that, for every h ∈ H , s ∈ S , ι ∈ I h ( s ) , and a ∈ A :

Moreover, since φ is persuasive we have that, for every h ∈ H , s ∈ S , ι ∈ I h ( s ) , and a, a ′ ∈ A :

where the last equality holds by definition of ϕ h ( a | s, ι, θ ) . Then, by combining Equation (10) and Equation (11), we get that, for every h ∈ H , s ∈ S , ι ∈ I h ( s ) , and a, a ′ ∈ A :

which means that σ satisfies Equation (2). Thus, by Lemma 3 we can conclude that σ is persuasive. In order to conclude the proof, it remains to show that σ achieves a sender's expected reward at least as good as that obtained by φ . Formally, we prove that V S ,φ σ h ( τ ) ≥ V S ,φ h ( τ ) for every step h ∈ H and history τ ∈ T h up to h . We will prove such a result by induction.

The base case of the induction is h = H . For every τ = ( s 1 , a 1 , . . . , s H -1 , a H -1 , s H ) ∈ T H :

where the last equality holds by Lemma 1.

Now, let us take h < H and assume that the statement that we want to prove holds for h +1 . Then, for every τ = ( s 1 , a 1 , . . . , s h -1 , a h -1 , s h ) ∈ T h , it holds:

where Equation (12) holds by definition of τ glyph[star] h,s h ,ι σ τ for the promise ι σ τ ∈ I h ( s h ) , Equation (13) holds by definition of ϕ h ( s h , ι σ τ , θ ) , Equation (14) holds by definition of τ glyph[star] h +1 ,s ′ ,g h ( s h ,a,ι σ τ ,s ′ ) together with the fact that g h ( s h , a, ι σ τ , s ′ ) = V R ,φ h +1 ( τ glyph[star] h,s h ,ι σ τ ⊕ ( a, s ′ )) , Equation (15) holds by induction, while Lemma 1 proves Equation (16) and Equation (17).

D Proofs omitted from Section 5

Lemma 4. Let σ := { ( I h , ϕ h , g h ) } h ∈H be returned by Algorithm 3 instantiated with any oracle O h,s,ι as in Definition 3. For every h ∈ H , s ∈ S , ι ∈ I h ( s ) , it holds that V S ,σ h ( s, ι ) ≥ M δ h ( s, ι ) .

Proof. Let σ = { ( I h , ϕ h , g h ) } h ∈H be the promise-form signaling scheme returned by Algorithm 3 instantiated with an approximate oracle O h,s,ι as in Definition 3, and let ( κ, q, v ) ←O h,s,ι -δ ( M δ h +1 ) . We prove the statement by induction. The base case for h = H of the induction holds trivially. Then, for every h < H , s ∈ S , and ι ∈ I h ( s ) , assuming the statement holds for h +1 we have that:

where the first inequality holds by the the inductive assumption, while the second inequality holds thanks to Definition 3.

Lemma 5. Let M δ h , I h (for h ∈ H ) be computed by Algorithm 3 instantiated with any oracle O h,s,ι as in Definition 3, and let σ glyph[star] = { ( I glyph[star] h , ϕ glyph[star] h , g glyph[star] h ) } h ∈H be an optimal promise-form signaling scheme. For every h ∈ H , s ∈ S , and ι ∈ I glyph[star] h ( s ) , it holds that glyph[ceilingleft] ι glyph[ceilingright] δ ∈ I h ( s ) and M δ h ( s, glyph[ceilingleft] ι glyph[ceilingright] δ ) ≥ V S ,σ glyph[star] h ( s, ι ) .

glyph[star]

Proof. We prove by induction that M δ h ( s, glyph[ceilingleft] ι glyph[ceilingright] δ ) ≥ V σ h ( s, ι ) for all h ∈ H , s ∈ S ,and ι ∈ I glyph[star] h ( s ) . As a base for the induction, we consider h = H . Notice that M H ( s, glyph[ceilingleft] ι glyph[ceilingright] δ ) is an upper bound on the solution of P H,s, glyph[floorleft] ι glyph[floorright] δ ( M δ H +1 ) (this holds by Definition 3). Moreover, ˆ κ ( a | θ ) = ϕ glyph[star] H ( a | s, ι, θ ) and ˆ q ( a, s ′ ) = 0 is a feasible solution to P H,s, glyph[floorleft] ι glyph[floorright] δ ( M δ H +1 ) and so (ˆ κ, ˆ q ) ∈ Ψ h,s glyph[floorleft] ι glyph[floorright] δ ⊆ Ψ h,s ι -δ . This readily implies that M δ H ( s, glyph[ceilingleft] ι glyph[ceilingright] δ ) ≥ V S ,σ glyph[star] H ( s, ι ) , since:

As for the inductive step, consider any h < H . First, we show that ˆ κ ( a | θ ) = ϕ glyph[star] h ( a | s, ι, θ ) and ˆ q ( a, s ′ ) = glyph[ceilingleft] g glyph[star] h ( s, a, ι, s ′ ) glyph[ceilingright] δ is a feasible solution to P h,s, glyph[floorleft] ι glyph[floorright] δ ( M δ h +1 ) . Then, we show that it gets expected rewards larger than V S ,σ glyph[star] h ( s, ι ) .

First we prove feasibility of (ˆ κ, ˆ q ) . Consider first the constraint of Equation (4a).

which proves that (ˆ κ, ˆ q ) satisfies the constraint of Equation (4a).

Now, let us turn our attention to the constraint of Equation (4b):

which proves (ˆ κ, ˆ q ) satisfies the constraint of Equation (4b) and thus P h,s, glyph[floorleft] ι glyph[floorright] δ ( M δ h +1 ) is feasible, i.e. , (ˆ κ, ˆ q ) ∈ Ψ glyph[floorleft] ι glyph[floorright] δ ⊂ Ψ ι -δ .

Now, we prove that it gets expected rewards larger than σ glyph[star] .

where we used the induction assumption in the second inequality.

In conclusion, for every ι ∈ I glyph[star] h ( s ) , it holds V σ glyph[star] h ( s, ι ) ≥ 0 , and, thus, M δ h ( s, glyph[ceilingleft] ι glyph[ceilingright] δ ) ≥ 0 , implying that P h,s,ι -δ is feasible and thus ι ∈ I h ( s )

Theorem 4. For any glyph[epsilon1] > 0 , given an approximate oracle as in Definition 3, Algorithm 3 instantiated with δ = glyph[epsilon1] / 2 H runs in time polynomial in 1 /glyph[epsilon1] and the instance size, while it finds an glyph[epsilon1] -persuasive promise-form signaling scheme that guarantees expected reward at least OPT to the sender.

Proof. First of all we need to prove that the σ returned by Algorithm 3 is indeed a promise-form signaling scheme. All the properties are trivially satisfied except for i) g h ( s, a, ι, s ′ ) ∈ I h +1 ( s ′ ) for each s ∈ S , a ∈ A , s ′ ∈ S , and ι ∈ I h ( s ) , and ii) 0 ∈ I 1 ( s ) for each s .

We start proving the first property. Let s ∈ S , a ∈ A , s ′ ∈ S , and ι ∈ I h ( s ) . Since ι ∈ I h ( s ) , by the definition of Algorithm 3 it holds that v > -∞ , where ( κ, q, v ) ←O h,s,ι -δ ( M δ h +1 ) is the solution returned by the call to the oracle. It is easy to see that this implies that M δ h +1 ( s ′ , q ( a, s ′ )) > -∞ . We consider two cases.

Then:

If h = H , then we have by construction that M δ h +1 ( s ′ , q ( a, s ′ )) > -∞ if and only if q ( a, s ′ ) = 0 . Thus, g h ( s, a, ι, s ′ ) = q ( a, s ′ ) = 0 ∈ I H +1 ( s ′ ) .

If h < H , then we M δ h +1 ( s ′ , q ( a, s ′ )) > -∞ implying that v ′ > -∞ , where ( κ ′ , q ′ , v ′ ) ← O h +1 ,s ′ ,q ( a,s ′ ) -δ ( M δ h +2 ) is the solution returned by the call to the oracle at step h +1 . By the definition of Algorithm 3, this implies that q ( a, s ′ ) ∈ I h +1 ( s ′ ) . Thus, g h ( s, a, ι, s ′ ) = q ( a, s ′ ) ∈ I H +1 ( s ′ ) .

Now, we prove the second property. We only need to prove that for all s ∈ S we have 0 ∈ I 1 ( s ) but this is easily proved by Lemma 5 instantiated for h = 1 and ι = 0 . Indeed we have that M δ 1 ( s, 0) ≥ V S ,σ glyph[star] h ( s, 0) , where we recall that M δ 1 ( s, 0) = v and ( κ, q, v ) ←O h,s, -δ ( M δ 2 ) . By the definition of Algorithm 3 this implies that 0 ∈ I 1 ( s ) .

Then, we prove the persuasiveness. By Lemma 3 we only need to prove that σ := { ( I h , ϕ h , g h ) } h ∈H satisfies the constraint of Equation 2 and that are η -honest for some η . This holds trivially for η = 2 δ as for all h ∈ H , s ∈ S and ι ∈ I h ( s ) we have by construction that for all h ∈ H , s ∈ S and ι ∈ I h ( s ) , and ( κ, q, v ) ←O h,s,ι -δ ( M δ h +1 ) we have

and by construction ( κ, q ) ∈ Ψ h,s ι -2 δ . This implies that the constraints of Equation (4a) and Equation (4b) are verified and we can apply Lemma 3 which let us conclude that σ is (2 δH ) -persuasive.

We now prove the optimality of the signaling scheme σ returned by Algorithm 3. By combining Lemma 4 and Lemma 5 we have that for all s ∈ S , h ∈ H and ι ∈ I glyph[star] h ( s )

which proves our statement.

E Proofs omitted from Section 6

In this section we prove that the optimization problem R h,s,ι ( M ) admits a polynomial time algorithm that finds an exact solution. We rewrite here for clarity the problem R h,s,ι ( M ) :

Even if the optimization problem is defined over functions, clearly we can represent the functions κ and q with finite number of variables with linear constraints (to assure that the the function's outputs are distributions). In order to handle the quadratic terms of R h,s,ι ( M ) we define a 'product' variable z a,s ′ ,ι for every a ∈ A , s ′ ∈ S , and ι ∈ D δ ( s ′ , M ) , which is used in place of

in Program 18a and has to satisfy the constraint for all a ∈ A and s ′ ∈ S :

For the linear variable we can introduce the non-negative variables ξ a,θ = κ ( a | θ ) for each θ ∈ Θ and a ∈ A , that need to satisfy the simplex constraint for all θ ∈ Θ , i.e. , ∑ a ∈A ξ a,θ = 1 . By using these variables we can write the problem as an LP h,s,ι ( M ) :

Since LP h,s,ι ( M ) has polynomial number of variable and constraints one can find a solution in polynomial time.

The next two lemmas show that one can use LP h,s,ι ( M ) . The first shows that any solution to R h,s,ι ( M ) can be used to find a solution to LP h,s,ι ( M ) .

Lemma 7. Let ( κ, q ) ∈ Ψ h,s ι be a feasible solution to R h,s,ι ( M ) , then there exists a feasible solution to LP h,s,ι ( M ) with the same value.

Proof. Let ( κ, ˜ q ) be a feasible solution to R h,s,ι ( M ) . Then define:

and

Then one can show by direct calculation that all the constraints of LP h,s,ι ( M ) are satisfied and that the objective value of LP h,s,ι ( M ) is Ω h,s,ι ( M ) (which is the value of program R h,s,ι ( M ) ).

The next lemma show a result which is 'complementary' to the one above.

Lemma 8. Given a feasible solution of LP h,s,ι ( M ) one can find a solution to R h,s,ι ( M ) with at least the same value.

Proof. Given a feasible solution z and ξ to LP h,s,ι ( M ) . Construct a solution ( κ, ˜ q ) to R h,s,ι ( M ) as follow for each ι ′ ∈ D δ ( s ′ , M ) , a ∈ A and s ∈ S :

and for all a ∈ A and θ ∈ Θ :

Notice that if for a specific h ∈ H , s ∈ S , s ′ ∈ S and a ∈ A , the constraint of Equation (19d) impose that if ∑ θ ∈ Θ µ h ( θ, s ) ξ a,θ p h ( s ′ | s, a, θ ) = 0 then z a,s ′ ,ι ′ = 0 for all ι ′ . This means that in any case the following condition holds:

We now fix any h ∈ H , s ∈ S and ι ∈ D δ show that the constraints of R h,s,ι ( M ) are satisfied, by using Equation (20). Let us start with the constraint of Equation (18b):

where the last inequality hold as ( z, ξ ) is a feasible solution to LP h,s,ι ( M ) . This proves that the constraint of Equation (18b) is satisfied.

For Equation (18c) similarly:

where in the last inequality we used that ( z, ξ ) is a feasible solution to LP h,s,ι ( M ) . Thus the constraint of Equation (18c) is verified. This proves that ( κ, ˜ q ) is feasible for R h,s,ι ( M ) . Then, by plugging Equation (20) into the objective of R h,s,ι ( M ) one directly prove that the values of the two problems is the same.

Now we are ready to prove the main result of this section:

Lemma 6. The problem R h,s,ι ( M ) can be solved in time polynomial in 1 /δ and the instance size.

Proof. Take a solution ( z, ξ ) to LP h,s,ι ( M ) . This can be done in polynomial time as LP h,s,ι ( M ) is a linear program with polynomial many variables and constraints. Then apply to the solution ( z, ξ ) the trasformation used in Lemma 8 to obtain a feasible solution to R h,s,ι ( M ) . This gives an optimal solution to R h,s,ι ( M ) . To prove the last result, assume that there would exists a feasible solution ( κ ′ , ˜ q ′ ) to R h,s,ι ( M ) such that ˜ F h,s,M ( κ ′ , ˜ q ′ ) > ˜ F h,s,M ( κ, ˜ q ) , and apply the trasformation defined in Lemma 7. This would find a solution ( z ′ , ξ ′ ) strictly better then ( z, ξ ) which contradicts the optimality of ( z, ξ ) for LP h,s,ι ( M ) .

Theorem 5. For every h ∈ H , s ∈ S , and ι ∈ D δ , if Algorithm 4 is used for all h ′ > h as approximate oracle O h ′ ,s,ι in Algorithm 3, then it implements an approximate oracle as in Definition 3.

Proof. Fix any ¯ h ∈ H , ¯ s ∈ S and ¯ ι ∈ D δ . We show that for h = ¯ h +1 , s ∈ S and for any distribution γ ∈ ∆( D δ ) , it holds that:

where the table M δ h is the one built by Algorithm 3 with oracle of Algorithm 4.

Define ι E := ∑ ι ∈D δ ( s,M δ h ) ι · γ ( ι ) . Then for every ι, ι ′ ∈ D δ and ι ′ ≤ ι we have that

as ι it only appears in the RHS of the constraint of the problem R h,s,ι ( M δ h ) .

By construction of Algorithm 3 we have that:

and that by Lemma 7 and Lemma 8, Ω h,s,ι ( M δ h +1 ) is equal to the value of LP h,s,ι ( M ) described by Equations (19), for each ι . This means that the function ι ↦→ Ω h,s,ι ( M δ h +1 ) is concave [Bertsekas, 1998, Theorem 5.1].

Thus for every distribution γ ∈ ∆( D δ ) we have:

Observe that we cannot apply directly the above concavity property as, for every s ∈ S , in Equation (21) we are only selecting the components of γ such that M δ h > -∞ .

To solve this problem we can, for any distribution γ ∈ ∆( D δ ) , table M and s ∈ S . define the new distribution ˜ γ on D δ ( s, M ) that puts all the mass on the -∞ components of M on 0 . Formally ˜ γ ( ι ) = γ ( ι ) for all ι ∈ D δ ( s,M ) , and ˜ γ (0) = γ (0) + ∑ ι ∈D δ \D δ ( s,M ) γ ( ι ) . Note that with this definition we have ι E = ∑ ι ∈D δ ( s,M ) ι · ˜ γ ( ι ) and ∑ ι ∈D δ ( s,M ) γ ( ι )Ω h,s,ι ( M ) ≤ ∑ ι ∈D δ ( s,M ) ˜ γ ( ι )Ω h,s,ι ( M ) . Combining these inequalities we can conclude that:

where the last inequality follows since it is easy to see that Ω h,s, 0 ( M δ h +1 ) ≥ 0 and the last equality follows from Lemma 7. This proves Equation (21).

Now assume that ( κ, ˜ q ) is a the optimal solution to R ¯ h, ¯ s, ¯ ι ( M δ ¯ h +1 ) . Clearly if ( κ, ˜ q E ) ∈ Ψ ¯ h, ¯ s ¯ ι then ( κ, q ) ∈ Ψ ¯ h, ¯ s ¯ ι -δ , where q = glyph[floorleft] ˜ q E glyph[floorright] δ .

Using the inequality of Equation (21) we can readily perform the following inequalities:

and thus Algorithm 4 returns a tuple that satisfies the conditions required by Definition 3.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studied a Bayesian persuasion problem where the sender and receiver act sequentially. \n\nBelow are two main changes in the new problem setting of this work:\n(1) The authors assume that the sender stops providing recommendations to the receiver if the receiver does not follow the recommendation. \n(2) Under this assumption, the authors also consider farsighted receivers as opposed to myopic receivers in previous works. \n\nIn the new setting, the authors showed that the Markovian signaling schemes are not optimal (additionally, finding the optimal Markovian signaling scheme in the previous problem settings in NP-hard), and they instead introduced a new class of promise-form signaling schemes for the new problem setting. The authors also show that the promise-form signaling schemes can be found in polynomial time while guaranteeing the schemes satisfy \\epsilon-persuasive property.\n\nSTRENGTHS:\n1. Clearly listed all key theoretical findings under the assumption that the sender will\n2. A novel class of promise-form signaling schemes is given with approximation algorithms that are relatively practical and can be completed in polynomial time\n\nWEAKNESSES:\n1. Overall a difficult-to-read paper (more difficult than the popular papers the authors cited) due to inadequate description and lack of concrete examples of the problem. Why sequential move in Bayesian persuasion is an important research topic is not highlighted. For a general audience not familiar with Bayesian persuasion, it is hard to tell why the new scenario is important.\n2. Lack of justifications for the critical assumption that the sender will stop providing recommendations. Further justifications are needed to show that it is rational for the sender to prefer to stop recommending over other strategies commonly used like tit-for-tat, etc. If this is the typical case in real-world applications, the authors should also point out that to improve the soundness of this assumption. Since all findings in this paper are based on this assumption, I encourage the authors to put more emphasis on this.\n3. Lack of discussion on the possible limitations\n\nQUESTIONS:\n1. Can you please provide application scenarios that fit well the new case\n2. Can you please add a discussion on the potential limitations\n3. In all problems that involve strategic manipulations, the potential fairness issues\n4. Why are numerical experiments included in some of the related works, e.g., \"Bayesian Persuasion in Sequential Decision-Making\" but not here in this paper? What are some of the major differences that result in the decision of skipping the numerical experiment part?\n\nLIMITATIONS:\nNot that I can find due to my limited background in this topic.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses a history-dependent signaling scheme in persuading a farsighted receiver. It first show that it is necessary for the sender to adopt a non-stationary and non-Markovian signaling scheme. Specifically, for every step and state reached at that step, this scheme defines a randomized mapping from the sender's private observations to action recommendations for the receiver, based on the whole history of states and receiver's actions observed up to that step. While such signaling policy could be intractable to describe, the paper provides a crucial simplification, promised-form signaling schemes, that allows the sender to only design finite size signaling scheme with optimal performance. Finally, the paper proposes a PTAS algorithm to determine the optimal promised-form signaling schemes.\n\nSTRENGTHS:\n1. The paper is very well written! The authors clearly explain the motivation, the model, the results, and the proofs with vivid intuitions, making the technical concepts very easy to follow.\n2. The paper extends the previous work on signaling schemes in MDPs to a more general setting, where the receiver is farsighted. The paper made several important technical and conceptual contributions to this problem, including the necessity of non-stationary and non-Markovian signaling schemes, the simplification to the promised-form signaling schemes, and the PTAS algorithm to determine the optimal promised-form signaling schemes.\n\nWEAKNESSES:\n1. The method is related to the literature of dynamic stackelberg equilibrium. The authors should discuss the relationship between their methods. \n2. I expect the authors to provide some real world applications of their model and methods, e.g., expand on the ride-sharing example in Appendix A.\n\nQUESTIONS:\nHow does the author think of the learning problem under this farsighted setup? Is it also possible to design a no-regret learning algorithm for the sender to learn the optimal promised-form signaling scheme?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers a specific model of information design, where the receiver takes actions on a sequential decision process under a global, unknown natural state $\\theta$. To make the model simple, the work assumes that the sequential decision process to be an MDP with a known model, plus the ability of the receiver to exactly optimize the cumulative reward once a (belief of) $\\theta$ is given. In this case, the signaling scheme represents a (more general) mapping from the natural state and the trajectory up to the current step to a distribution of actions as information revelation. Further, when making decisions the receiver is not allowed to use the posterior distribution/belief of the natural state obtained from previous steps (which means only $\\hat\\theta_h$ is used).\n\nGiven the above assumptions in the model, the work provides a polynomial-time algorithm to obtain an $\\epsilon$-persuasive signaling scheme. This disagrees with the NP-hard-like claims given in previous works. Such disagreement stems from the use of trajectory information and thereafter its simplified version of promise form. The algorithm is natural but quite creative.\n\nSTRENGTHS:\n1. The work provides new models of information design in sequential decision problems. The new model no longer possesses theoretical hardness.\n2. The work proposes a new polynomial-time algorithm to find an $\\epsilon$-persuasive signaling scheme.\n\nWEAKNESSES:\nSeveral limitations persist: 1) MDPs are assumed to be exactly solved. 2) MDP models are known 3) Receiver can't aggregate historical information of the natural state 4) No experiments.\n\nQUESTIONS:\nThis conclusion indeed applies to the model-based scenario (meaning that the receiver knows the state set $S$, the observation set $\\Theta$, the state transition distribution $p$, and the observation distribution $\\mu$). However, does this conclusion also apply to the model-free case? This question might not be within the scope of the work, but the conclusion drawn is a bit too broad if such estimation of model is now involved. If this situation is not discussed, the authors should emphasize this important assumption in the abstract and introduction. Specifically, it should be clarified which information the receiver is assumed to know and base their decisions on.\n\nThis assumption represents a strong capability for the receiver. If it does not possess this knowledge, the sender's manipulation could be more powerful, and Markovian signaling schemes might be viable. For instance, if the receiver is unaware of the state and can only observe the sender's signals, and it needs to estimate the state and its transitions based on those signals, does the sender have the opportunity to confuse the receiver's judgments and achieve stronger persuasion? I find the claim \"We consider the most general setting\" in line 85 a bit too strong.\n\nAdditionally, can the revelation principle argument still be applied in a sequential setting? Does recommending only one action for each state achieve the goal of persuading a receiver in an MDP? I could not find any relevant discussion on this. Has the author considered \"future-dependent\" signaling schemes: recommending a set of future actions for each state instead of just one action? Or sending a signal $m$ to encode a set of future actions they wish to recommend? Moreover, in the aforementioned scenario, can the sender confuse multiple states by sending signal $m$? If the revelation principle is abandoned, is a history-dependent signaling scheme still necessary? If there is no discussion on the validity of the revelation principle, this assumption should be prominently emphasized.\n\nLIMITATIONS:\nSee weakness", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers a (finite-horizon) dynamic persuasion problem\nbetween a sender and a receiver, both of whom are long-lived. At each\ntime $t$, there is a publicly-observable state $s_t$ and a\npayoff-relevant quantity $\\theta_t$, which is observed only by the\nsender, and whose distribution depends on the current state (and\nis independent of other quantities). Based on the observation of\n$\\theta_t$ the sender recommends an action to the receiver (which may\nor may not be followed). The publicly-observable state then updates to\n$s_{t+1}$ according to a transition kernel that depends on the current\nstate $s_t$, the quantity $\\theta_t$ and the action chosen by the\nreceiver. Both the sender and the receiver seek to maximize their\ntotal expected payoffs.\n\nPrevious work on this topic, with few exceptions, has focused on\nmyopic receivers, motivated by settings in which the receiver is\nshort-lived. The difference here then is the focus on long-lived,\nfar-sighted receiver. In this setting, the paper first shows (via an\nexample) that the class of Markovian signaling schemes (whose\nrecommendations only depend on the current state $s_t$) is\ninsufficient for optimal persuasion, and the sender can do better\nusing a signaling scheme that takes into account the history of the\nprocess. Due to the computational difficulties in working with general\nhistory-dependent schemes, the paper then considers promise-form\nsignaling schemes, which make recommendations based not only on the\ncurrent state, but also on a (history-dependent) \"promise\", which is a\nguarantee on the receiver's continuation payoffs. Essentially, the\npromise succinctly summarizes the history, thereby reducing the\ncomputational complexity to be polynomial in the size of set of\npromises. The authors show that, upon imposing a honesty condition on\nthe promises across time, the class of promise-form signaling schemes\nsuffice for optimal persuasion. The authors also propose an\napproximation scheme for computing approximately-persuasive\npromise-form signaling schemes with good payoff guarantees, that is\npolynomial in the approximation factor.\n\nSTRENGTHS:\n+ The paper considers an interesting variation of the sequential\n  persuasion problem, allowing for far-sighted receivers. This makes\n  the problem substantially more complex. Nevertheless, the paper\n  identifies a class of relatively simple and approximately persuasive\n  signaling schemes that nevertheless achieve optimal payoffs for the\n  sender, and are furthermore computationally tractable.\n\n+ The paper illustrates well the insufficiency of the class of\n  Markovian signaling schemes, and furthermore (adapting existing\n  results) shows that finding a constant-factor approximation within\n  the class of Markovian signaling schemes is NP-hard.\n\n+ The class of promise-form signaling schemes is fairly simple and\n  easy to implement; furthermore, it seems approximately-optimal such\n  schemes can be computed via solving an LP (repeatedly).\n\nWEAKNESSES:\n+ While the class of promise-form signaling schemes is interesting,\n  there is a significant line of work in economics that studies the\n  use of promises in repeated games with incomplete information. The\n  paper does not cite those papers, nor does it place its\n  contributions within that context. A particularly relevant paper in\n  this line is Abreu, Pierce and Stacchetti (Econometrica, 1990),\n  whose results imply the sufficiency of the class of \"promise-form\"\n  strategies for discounted repeated games with imperfect monitoring.\n\n+ Similarly, the paper would benefit from connecting with general\n  literature on repeated games (with or without incomplete\n  information). For instance, the insufficiency of Markov signaling\n  schemes is very much in the same vein as the inefficiency of Markov\n  perfect equilibria in, say, repeated prisoner's dilemma to sustain\n  cooperation. With far-sighted receivers, it is not surprising that\n  Markov signaling schemes are not optimal for the sender. \n\n+ $\\epsilon$-persuasiveness: In the analysis of history-dependent (or\n  promise-form) signaling schemes, the authors relax the\n  persuasiveness requirement to $\\epsilon$-persuasiveness. There is a\n  subtle issue in interpreting this relaxation. To explain, a natural\n  relaxation would be that the receiver's expected continuation payoff\n  from following the recommendation is at most $\\epsilon$ worse than\n  choosing any other action, *after* receiving the recommendation.\n  Specifically, the expectation taken here would be with respect to\n  the posterior belief after receiving the recommendation. However,\n  the condition in Definition 1 requires something different; it\n  states that the receiver's expected payoff from following a\n  recommendation, *multiplied* by the probability of receiving that\n  recommendation, should be at most $\\epsilon$ worse. In particular,\n  there is an extra factor equaling the probability of recommending a\n  particular action.\n\n  While this may seem like a minor technical issue, this has\n  substantial implication on the assumption that the receiver would\n  adopt such a recommendation. For instance, this suggests that as\n  long as the probability of recommending an action is small, the\n  sender can recommend an action that can yield substantially lower\n  continuation payoff for the receiver, and still expect the receiver\n  to accept the recommendation. This seems to be a very strong\n  assumption on the receiver's behavior, that does not align with the\n  assumption that the receivers are (approximately) Bayesian.\n  Moreover, with such a strong assumption, it is no longer clear if\n  $OPT$ is the right benchmark for comparison.\n\n  A potential fix to this issue would be to impose the relaxation on the\n  conditional expectation, i.e., to replace the $\\epsilon$ term in the\n  definition with $\\epsilon \\sum_{\\theta}\n  \\mu_h(\\theta|s_h)\\phi_\\tau(a|\\theta)$. However, it is not clear if\n  the later approximation results continue to apply with this change.\n\n+ Finally, while the paper makes sound and rigorous technical\n  contribution, there is not enough discussion motivating the specific\n  model being studied. For instance, there is no discussion of the\n  motivation behind far-sightedness assumption; the myopic behavior of\n  the receivers in previous work is frequently motivated by assuming a\n  series of short-lived receivers. In particular, are there any\n  specific applications where a single sender and a single receiver\n  interact in the manner studied? (I think this is especially useful\n  given the somewhat complicated form of the signaling scheme\n  proposed.) Some discussion here would benefit the paper by grounding\n  the theoretical results.\n\nQUESTIONS:\n+ Do the approximation results continue to hold if the relaxation of\n  the persuasiveness constraint is imposed on the conditional\n  expectation?\n\n+ With the current definition of $\\epsilon$-persuasiveness, it may be\n  possible to design mechanisms that achieve payoffs substantially\n  better than $OPT$. Are there any guarantees on how small (or large)\n  this difference can be?\n\nLIMITATIONS:\nThe assumptions are stated clearly. Some discussion of the limitations\ninduced by relaxing the persuasiveness/honesty requirements would be\nhelpful.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-19"}